"We're trying to avoid Ansible exiting when there's an error, leaving the node in a half-deployed state. "
Hmm. Not sure of your production environment, but seems something about the package is very wrong if upgrades sometimes fail that I would want to get sorted out. Package installs should be meant to be reliable. Consider moving your database schema upgrade invocation outside of the package? As for leaving nodes in broken states, this is a good case for a rolling update with load balancers. I don't know your infrastructure enough to say if that's possible, but ansible is designed to stop on a failure on purpose -- which would leave failed nodes out of rotation. You could also consider a seperate rollback playbook to use if you so decided to apply it, keeping it from maybe applying it self unexpectedly. On Fri, Jun 13, 2014 at 2:35 PM, Derrick Bryant <[email protected]> wrote: > We're trying to avoid Ansible exiting when there's an error, leaving the > node in a half-deployed state. We want Ansible to attempt a rollback if the > deployment of new software fails. That is, if any of the tasks fail, we > want Ansible to stop executing the existing play and run some recover > playbook instead. > > I'm not sure if we are going down the right track with the current > playbook. We're definitely open to other suggestions of how to accomplish > our goal. > > Thank you, > Derrick Bryant > > > On Fri, Jun 13, 2014 at 2:05 PM, Michael DeHaan <[email protected]> > wrote: > >> I'm not sure why you are ignoring failures on the install RPM step. That >> seems not good. >> >> And ignoring errors on the service part, also not good! >> >> What's the reason for all of this? >> >> >> >> >> >> On Thu, Jun 12, 2014 at 10:51 AM, Derrick Bryant <[email protected]> >> wrote: >> >>> Thanks for the reply Michael. >>> >>> I came across the group_by module when Googling around but I didn't >>> understand how to use it until I started typing this reply. Below is an >>> edited version of my previously posted playbook snippet. The only question >>> I have is how should I skip the task "Stopping nginx" if "Installing new >>> RPM" fails? Should I add "when: result|success" to all of my tasks so that >>> later tasks will not be applied if one fails? Once one task fails, no other >>> deployment tasks should be attempted and the rollback playlists should be >>> called. >>> >>> >>> >>> --- >>> - hosts: all >>> vars: >>> RPM_NAME: 'company_app_pacakge_name' >>> remote_user: ansible >>> sudo: yes >>> #accelerate: true >>> tasks: >>> - name: Installing new RPM >>> yum: "name={{ RPM_NAME }} state=present" >>> ignore_errors: true >>> >>> register: result >>> tags: >>> - testing >>> - group_by: key=failed_hosts >>> when: result|failed >>> >>> - name: Stopping nginx >>> service: name=nginx state=stopped >>> ignore_errors: true >>> register: result >>> tags: >>> - testing >>> - group_by: key=failed_hosts >>> when: result|failed >>> >>> << other tasks >> >>> >>> >>> - hosts: failed_hosts >>> name: "Rolling back deployment on {{ ansible_fqdn }}" >>> include: rollback/exit.yml >>> >>> >>> On Wednesday, June 11, 2014 6:43:21 PM UTC-4, Michael DeHaan wrote: >>> >>>> Includes with conditionals are implemented by including all the steps >>>> and applying the conditional to each included task. >>>> >>>> You will get skipped messages. >>>> >>>> Often a better solution is using the group_by module to create a >>>> dynamic group, and then applying that config to just the hosts that match. >>>> >>>> This almost always produces cleaner output and is used often to apply >>>> different configs based on OS. >>>> >>>> http://docs.ansible.com/group_by_module.html >>>> >>>> >>>> On Wed, Jun 11, 2014 at 12:52 PM, Derrick Bryant <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm running in to an issue with conditional includes. I'm trying to >>>>> create an automatic rollback procedure for one of my playbooks which >>>>> attempts to deploy new versions of our company software. I'm using >>>>> includes >>>>> to abstract the common roll back tasks to separate YAML files in a >>>>> directory called "rollback". I've included a small sample of the playbook >>>>> below which includes "rollback/exit.yml" when the result of the previous >>>>> task failed. What seems to happen is that ansible includes the tasks from >>>>> the rollback/exit.yml file and applies the "when: result|failed" condition >>>>> to every task in the included yaml file. This produces the desired result >>>>> but does print out a bunch of "skipped" messages as shown below. Is there >>>>> a >>>>> way to tell ansible to only include the tasks if the previous task failed >>>>> or to not show the skipped tasks? Is there a better way to accomplish what >>>>> I'm trying to do? >>>>> >>>>> >>>>> I'm using a version of the command below to run the playbook (filename >>>>> of playbook changed): >>>>> >>>>> ansible-playbook playbooks/company_app.yml -i playbooks/hosts --tags >>>>>> "testing" >>>>>> >>>>>> >>>>> playbooks/company_app.yml >>>>> >>>>> --- >>>>>> - hosts: all >>>>>> vars: >>>>>> RPM_NAME: 'company_app_pacakge_name' >>>>>> remote_user: ansible >>>>>> sudo: yes >>>>>> #accelerate: true >>>>>> tasks: >>>>>> - name: Installing new RPM >>>>>> yum: "name={{ RPM_NAME }} state=present" >>>>>> ignore_errors: true >>>>>> tags: >>>>>> - testing >>>>>> register: result >>>>>> - include: rollback/exit.yml >>>>>> when: result|failed >>>>>> <<snip>> >>>>>> >>>>>> >>>>> playbooks/rollback/exit.yml >>>>> --- >>>>> - fail: msg="The company_app deployment failed." >>>>> tags: >>>>> - testing >>>>> >>>>> Ansible version: >>>>> $ ansible-playbook --version >>>>> ansible-playbook 1.6.3 >>>>> >>>>> Skipped message: >>>>> >>>>> >>>>> >>>>> [ansible@master01 ~]$ ansible-playbook playbooks/company_app.yml -i >>>>> playbooks/hosts --tags "testing" >>>>> >>>>> >>>>> >>>>> >>>>> PLAY [all] >>>>> ******************************************************************** >>>>> >>>>> >>>>> >>>>> >>>>> GATHERING FACTS >>>>> *************************************************************** >>>>> ok: [ansible_node] >>>>> >>>>> >>>>> >>>>> >>>>> TASK: [Installing new RPM] >>>>> **************************************************** >>>>> >>>>> >>>>> >>>>> ok: [ansible_node] >>>>> >>>>> >>>>> >>>>> >>>>> TASK: [fail msg="The company_app deployment failed."] >>>>> **************************** >>>>> >>>>> >>>>> >>>>> skipping: [ansible_node] >>>>> >>>>> >>>>> >>>>> >>>>> Thanks for the help, >>>>> Derrick >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Ansible Project" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/ansible-project/7f43cbe3-58fc-4384-bc3f- >>>>> 64dd86e14714%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/ansible-project/7f43cbe3-58fc-4384-bc3f-64dd86e14714%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Ansible Project" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/ansible-project/eded9641-db4c-4b23-b832-5ba081734b85%40googlegroups.com >>> <https://groups.google.com/d/msgid/ansible-project/eded9641-db4c-4b23-b832-5ba081734b85%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Ansible Project" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/ansible-project/NBnq9vUZSFQ/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwue9kT66zL%3DcKt05DOBrkmgK1wNWMW0vpw_2agzOdFaw%40mail.gmail.com >> <https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwue9kT66zL%3DcKt05DOBrkmgK1wNWMW0vpw_2agzOdFaw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/CAP8kV6wV8xrgR0bKhwnj6Zb53qyN51yLL6NbfGM7XcD6WyJBQw%40mail.gmail.com > <https://groups.google.com/d/msgid/ansible-project/CAP8kV6wV8xrgR0bKhwnj6Zb53qyN51yLL6NbfGM7XcD6WyJBQw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyve5WTM-oOJMGfK%3DFqV2SqyKvRx3gFSTJrSqoNBdFrSQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
