[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
I first reproduced the problem on the training cloud, then installed the PPA packages. The problem went away, the cloud is responsive within the first two minutes of restarting the frontend. -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
** Also affects: eucalyptus (Ubuntu Karmic) Importance: Undecided Status: New ** Changed in: eucalyptus (Ubuntu Karmic) Status: New = Triaged ** Changed in: eucalyptus (Ubuntu Karmic) Importance: Undecided = High ** Changed in: eucalyptus (Ubuntu Karmic) Assignee: (unassigned) = Dustin Kirkland (kirkland) ** Changed in: eucalyptus (Ubuntu Karmic) Milestone: None = karmic-updates -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
I have nominated this for Karmic, in the hopes of getting an SRU out soon. We think we have tracked this problem to a very simple fix...ensuring that the iptables module gets loaded in initial eucalyptus upstart job. I have uploaded eucalyptus_1.6~bzr931-0ubuntu7.5~ppa1 to the ubuntu-virt ppa: * https://launchpad.net/~ubuntu-virt/+archive/ppa I don't have a current Karmic test environment, since I'm totally focused on Lucid development right now. If I could get one person to confirm this fix on their Karmic UEC, I will upload to karmic-proposed and begin the SRU process (where the fix will again need to be confirmed). Nick/Boris/Torsten- can any of you guys help verify the fix in the PPA? Thanks, :-Dustin ** Changed in: eucalyptus (Ubuntu Karmic) Status: Triaged = In Progress -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
Manually testing the change worked for me. I'll try the built package shortly -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
This bug was fixed in the package eucalyptus - 1.6.2~bzr1136-0ubuntu3 --- eucalyptus (1.6.2~bzr1136-0ubuntu3) lucid; urgency=low * eucalyptus-cc.eucalyptus-cc-publication.upstart, eucalyptus-common.eucalyptus.upstart, eucalyptus-network.upstart, eucalyptus-sc.eucalyptus-sc-publication.upstart, eucalyptus-walrus.eucalyptus-walrus-publication.upstart, rules: rework our eucalyptus starting condition to depend on a new upstart emitted signal, eucalyptus-network-is-ready, which is only fired once the network interface providing the default route is in fact up, and listening on a real ip address, LP: #503180 * debian/eucalyptus-common.eucalyptus.upstart: - don't respawn eucalyptus-cloud - ensure that the iptables module gets loaded soon enough, otherwise much bad behavior happens in various nasty ways, most notably, a wedged database which renders the CLC non responsive on restart/reboot, LP: #503180 and dupes, LP: #444352, #444946, #467521, #478573, #480048 -- Dustin Kirkland kirkl...@ubuntu.com Tue, 02 Feb 2010 17:13:52 -0800 ** Branch linked: lp:~ubuntu-core-dev/eucalyptus/ubuntu ** Changed in: eucalyptus (Ubuntu Lucid) Status: Confirmed = Fix Released -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
Okay, I talked to Scott about similar upstart situations last week, and I think I have his blessing on what I thought would be hacky. Basically, we need to create an additional, upstart (psuedo) job, that would start on started networking, and basically loop until some conditional is settled (in this case, ip_address_obtained), at which point this psuedo job would emit a signal. The Eucalyptus upstart jobs would simply wait on this signal. I'll implement this at the distro sprint. ** Summary changed: - eucalyptus-cloud doesn't reply to requests + eucalyptus-cloud doesn't reply to requests (due to upstart networking issue) -- eucalyptus-cloud doesn't reply to requests (due to upstart networking issue) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
** Summary changed: - eucalyptus-cloud doesn't reply to requests (due to upstart networking issue) + eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) -- eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior) https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
It looks like starting eucalyptus on net-device-up IFACE=eth0 is not sufficient. It definitely looks sensitive to startup order / network up though. I'll let Dustin investigate this one deeper. -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
Thank Thierry, We've finally been able to (we believe) reproduce this type of condition on our Lucid machines, and have figured out the reason why it is being triggered on lucid UEC installations. The Eucalyptus front-end components (cloud, walrus, sc) require that, on startup, the network interface that was used to register components is up and configured (i.e. has an IP address that was used at component registration time). Our init process ensures this by running after the 'network' init script completes, but in the upstart case it looks like the eucalyptus components can start after the network process has started, but before it completes. In sum, if eucalyptus starts, then the interface becomes configured, this condition will be triggered and the front-end will need to be restarted so that it can go through the initialization process again. Note that this is a startup time requirement only (i.e., once the system is up, and for some reason the interface goes down and comes up again), the service will stay active; it is just during the startup that eucalyptus requires that the interface is alive and configured. Is there a way to have the eucalyptus upstart scripts wait until the network interface is fully alive before starting the front-end components? Thanks! -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
That makes sense. I will test with an upstart script that clearly waits for eth0 up before starting eucalyptus and let you know. -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
Greetings, We're having trouble reproducing this problem on our lucid systems against eucalyptus-cloud-1.6.2~bzr1124-0ubuntu3. We think that the issue is possibly related to some process startup behavior, but it would be great to get some more system information or a process by which the issue can often be triggered - some questions follow, the answers to which will help us try to reproduce: 1.) does this condition appear on a fresh boot (i.e. there are definitely no prior eucalyptus-cloud processes running when a new eucalyptus-cloud process is started)? 2.) does this condition appear when there is certainly the primary network interface up and configured? 3.) does this condition on 'start eucalyptus-cloud', 'restart eucalyptus-cloud' or both? 4.) do you see the condition when the cloud/walrus/sc are all on the same system or is this a stand-alone cloud service (or some other topology)? 5.) does this happen after a cloud has been configured/working or is it always during the initial setup (during registration of other components)? If we can trap the problem a bit further, we'll surely be able to find the solution! regards -dan -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
I hit it systematically on the following test: Alpha2 ISO, UEC installer, select CLC+Walrus+CC+SC Just after reboot (so, fresh boot, clearly), the right ports come up... but... sudo euca_conf --get-credentials mycreds.zip stalls sudo euca_conf --list-walruses answers You need to be on the CLC... Running sudo stop eucalyptus leaves a eucalyptus-cloud process running that only SIGKILL will stop. After killing the process, sudo start eucalyptus... everything runs fine. I haven't tried so much other conditions. Looking at the upstart job, since it now starts on runlevel [2345] I think lo is certainly up, but nothing is ensured about eth0. Could that explain it ? -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
** Tags added: iso-testing -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
** Changed in: eucalyptus (Ubuntu) Milestone: None = lucid-alpha-3 ** Changed in: eucalyptus (Ubuntu) Assignee: (unassigned) = Dustin Kirkland (kirkland) ** Also affects: eucalyptus (Ubuntu Lucid) Importance: High Assignee: Dustin Kirkland (kirkland) Status: Confirmed -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
I've seen similar problems in the past, but waiting for some extended time (15 minutes) solved it. In that time the Web UI was also not working, the admin user was reported to not exist. -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
I also have to kill -9 the remaining eucalyptus-cloud process after the stop. I don't have the same exception in cloud-error.log. In cloud- output.log everything seems normal, then nothing gets added anymore. euca_conf is run, nothing shows in the log. Then after some time you start getting tons of: NioServerHandler.java:145 org.jboss.netty.handler.timeout.ReadTimeoutException NioServerHandler.java:148 org.jboss.netty.handler.timeout.WriteTimeoutException NioServerHandler.java:154 Internal Error. That seems to point to some lockup in the eucalyptus-cloud process. Note that euca_conf still returns 0 in that case, which is confusing. -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 503180] Re: eucalyptus-cloud doesn't reply to requests
I ran into this one as well. I had to stop eucalyptus / start eucalyptus to get a running system. ** Changed in: eucalyptus (Ubuntu) Status: New = Confirmed -- eucalyptus-cloud doesn't reply to requests https://bugs.launchpad.net/bugs/503180 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
