[Bug 1124384] Re: reload-configuration can confuse upstart
I believe the problem here is that -- in the case where a JobClass has no running instances -- a reload is causing the original JobClass to be destroyed. That eventually leads to Event-blockers to reach zero (as a result of the JobClasses EventOperators being freed) causing the event to be destroyed. That behaviour is correct inasmuch as events are being destroyed when they should. What is not correct is that the reload results in all this happening *before* the newly-loaded JobClass gets a chance to increment Event-blockers to avoid the event being freed. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
16:54 cjwatson jodh: I *think* I may see the general structure of what's going on here 16:54 cjwatson jodh: This debugging should make it about as clear to you as it is to me so far, I think 16:54 cjwatson 7init: conf_load_path_with_override: Loading configuration file /etc/init/rc-sysinit.conf 16:54 cjwatson ... 16:54 cjwatson 7init: conf_file_destroy: Destroyed unused job rc-sysinit 16:54 cjwatson 7init: event_unblock: name: 'filesystem', new blockers: 3 16:55 cjwatson jodh: When we tear down the old job, we end up unreferencing and destroying the event operators it refers to, at least enough to cause them to decrement various event-blockers 16:56 cjwatson jodh: So the 'filesystem' event ends up entering the finished state far too early because it's been wrongly unblocked 17:08 jodh cjwatson: gotcha - that is indeed subtle. Could be an interesting one to fix too ;) 17:10 cjwatson jodh: My feeling is that the nih_free in conf_file_destroy needs to be something more careful involving keeping references 17:10 cjwatson jodh: But that also perhaps job_class_reconsider shouldn't replace a job that has events that are blocking some other event? I'm not quite sure, I don't know the structures quite well enough 17:10 cjwatson jodh: Is that enough to get you going for now though? 17:11 jodh cjwatson: sure is - thanks!! -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
I've verified this is still an issue with the following user-data on upstart at
'1.8-0ubuntu1'.
This is re-creatable via the most recent daily image
(ubuntu-raring-daily-amd64-server-20130403). I used ami-88a4c1e1.
But you can download and recreate as above. The user-data provided to
recreate is below. The 'createmark' is necessary because we've disabled
the 'initctl reload' call to lessen the issue. touching that file turns
the call back on.
#cloud-config-archive
- filename: createmark
content: |
#cloud-boothook
#!/bin/sh
touch /run/cloud-init-upstart-reload
- content: |
#!/bin/sh
echo $(date -R): user-script run === | tee /run/user-script.log
- filename: myjob.conf
content: |
#upstart-job
description a test upstart job
start on stopped rc RUNLEVEL=[2345]
console output
task
script
echo $(date -R): upstart job run === | tee /run/upstart-job.log
end script
- content: |
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to cloud-init in Ubuntu.
https://bugs.launchpad.net/bugs/1124384
Title:
reload-configuration can confuse upstart
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
Ok, another recreate that is possibly simpler to debug. Below, we create 2 jobs, one invokes reload-configuration, the other just writes something to /run. We add those jobs to a pristine raring container, start it, and expect them to run. Only the first will run. I've explicitly ran 'start console' from the first so you can log in, otherwise you wont get a getty, so you can't log in to look around. cat EOF | tee job1.conf description invoke reload for fun start on filesystem console output task script read x y /proc/uptime cmd=initctl reload-configuration ( echo up $x: running $cmd === initctl reload-configuration echo got $? ) 21 | tee /run/job1.log sh -c 'sleep 5; start console' end script EOF cat EOF | tee job2.conf description do something around rc.local start on stopped rc RUNLEVEL=[2345] console output task script read x y /proc/uptime echo up $x: touching /run/did-something === | tee /run/job2.log echo $x /run/did-something end script EOF # precise base sudo apt-get install lxc sname=raring-amd64-source cname=raring-amd64 sudo lxc-create -n raring-amd64-source -t ubuntu -- --arch=amd64 raring sudo lxc-clone -o $sname -n $cname sudo cp job1.conf job2.conf /var/lib/lxc/$cname/rootfs/etc/init/ sudo lxc-start -n $cname ## now log in as 'ubuntu'. You *should* see a file /run/did-something. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Description changed: Under bug 1080841 we made cloud-init invoke 'initctl reload- configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | +#boothook #!/bin/sh -echo $(date -R): user-script run === | tee /run/user-script.log +touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: | -#upstart-job -description a test upstart job -start on stopped rc RUNLEVEL=[2345] -console output -task -script -echo $(date -R): upstart job run === | tee /run/upstart-job.log -end script + #!/bin/sh + echo $(date -R): user-script run === | tee /run/user-script.log + - content: | + #upstart-job + description a test upstart job + start on stopped rc RUNLEVEL=[2345] + console output + task + script + echo $(date -R): upstart job run === | tee /run/upstart-job.log + end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs: - * bug 1080841: should reload configuration if an upstart job is added - * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image + * bug 1080841: should reload configuration if an upstart job is added + * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image ** Description changed: Under bug 1080841 we made cloud-init invoke 'initctl reload- configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | -#boothook -#!/bin/sh -touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 + #cloud-boothook + #!/bin/sh + touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: | #!/bin/sh echo $(date -R): user-script run === | tee /run/user-script.log - content: | #upstart-job description a test upstart job start on stopped rc RUNLEVEL=[2345] console output task script echo $(date -R): upstart job run === | tee /run/upstart-job.log end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs: * bug 1080841: should reload configuration if an upstart job is added * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Summary changed: - cloud-init parses yaml incorrectly + reload-configuration can confuse upstart -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Description changed: - Example from failure: https://jenkins.qa.ubuntu.com/job/raring-server- - ec2/ARCH=amd64,REGION=eu-west-1,STORAGE=ebs,TEST=multi-part-ud,label - =ubuntu-server-ec2-testing/17/console + Under bug 1080841 we made cloud-init invoke 'initctl reload- + configuration' after it wrote a upstart job. This was necessary because + inotify is not supported on all filesystems (overlayfs being the one of + most current interst). - Cloud-init is parsing YAML incorrectly - runcmd: - ... - [ sh, -c, echo = $(date) ': hello world!' = ] - ... - [ sh, -c, 'echo $(date) /root/runcmd.date.txt' ] - ... - [ wget, http://slashdot.org;, -O, /tmp/index.html ] - ... - touch /tmp/done + This seems to be causing upstart some pain, and resulting in cloud-final + (and 'rc') not being run. - Fails, while changing - touch /tmp/done to - [ touch /tmp/done ] - works. + Easy user-data to reproduce the problem is: + + #cloud-config-archive + - content: | +#!/bin/sh +echo $(date -R): user-script run === | tee /run/user-script.log + - content: | +#upstart-job +description a test upstart job +start on stopped rc RUNLEVEL=[2345] +console output +task +script +echo $(date -R): upstart job run === | tee /run/upstart-job.log +end script + + You should (and do on quantal) end up with 2 files written to /run. + + I've verified that the same behavior is true on quantal. If you change + cloud-init to notify upstart about a job immediately after it writes it, + then quantal's upstart gets confused also. + + Related bugs: + * bug 1080841: should reload configuration if an upstart job is added + * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image ** Also affects: upstart (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Attachment added: dmesg when 'initctl reload-configuration' is *not* used https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1124384/+attachment/3528559/+files/dmesg.no-reload.txt ** Changed in: cloud-init (Ubuntu) Status: New = Confirmed ** Changed in: upstart (Ubuntu) Status: New = Confirmed ** Changed in: upstart (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Attachment added: dmesg when 'initctl reload-configuration' is used https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1124384/+attachment/3528558/+files/dmesg.reload.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Attachment added: dmesg with --debug when 'initctl reload-configuration' is used https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+attachment/3528635/+files/dmesg-debug-reload.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
** Attachment added: dmesg with --debug when 'initctl reload-configuration' is *not* used https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+attachment/3528636/+files/dmesg-debug-no-reload.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
This bug has been reported on the Ubuntu ISO testing tracker. A list of all reports related to this bug can be found here: http://iso.qa.ubuntu.com/qatracker/reports/bugs/1124384 ** Tags added: iso-testing -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1124384] Re: reload-configuration can confuse upstart
I mistakingly posted recreate instructions at https://bugs.launchpad.net/ubuntu/+source/cloud- init/+bug/1103881/comments/6 . they're not comletely irrelevant there, but I did intend for that to go here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/1124384 Title: reload-configuration can confuse upstart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
