*** This bug is a duplicate of bug 503180 ***
https://bugs.launchpad.net/bugs/503180
This bug was fixed in the package eucalyptus - 1.6.2~bzr1136-0ubuntu3
---
eucalyptus (1.6.2~bzr1136-0ubuntu3) lucid; urgency=low
* eucalyptus-cc.eucalyptus-cc-publication.upstart,
*** This bug is a duplicate of bug 503180 ***
https://bugs.launchpad.net/bugs/503180
The DB deadlock in Eucalyptus itself has been fixed, but there is a
remaining upstart/init issue, Bug #503180. I'm marking this bug a
duplicate of that one, at this point.
Ultimately, eucalyptus-cloud
** Changed in: eucalyptus (Ubuntu Karmic)
Status: Triaged = Won't Fix
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is
** Changed in: eucalyptus
Status: New = Invalid
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to
** Changed in: eucalyptus
Assignee: (unassigned) = chris grzegorczyk (chris-grze)
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team,
** Tags added: iso-testing
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
** Also affects: eucalyptus
Importance: Undecided
Status: New
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is
** Changed in: eucalyptus (Ubuntu Karmic)
Assignee: Thierry Carrez (ttx) = Dustin Kirkland (kirkland)
--
DB deadlock on reboot prevents UEC from working, temporarily - 403 Forbidden
errors
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of
I'm seeing this quite a bit now in my UEC. I'm suggesting that we
should probably identify a fix and roll it out in an SRU.
Thierry, if you disagree, feel free to close it won't-fix again.
:-Dustin
** Changed in: eucalyptus (Ubuntu Karmic)
Milestone: None = karmic-updates
** Changed in:
** Changed in: ubuntu-release-notes
Assignee: (unassigned) = Dustin Kirkland (kirkland)
** Changed in: ubuntu-release-notes
Assignee: Dustin Kirkland (kirkland) = Thierry Carrez (ttx)
--
DB deadlock on reboot prevents UEC from working, temporarily
** Changed in: ubuntu-release-notes
Assignee: Thierry Carrez (ttx) = Mathias Gug (mathiaz)
** Changed in: ubuntu-release-notes
Status: New = In Progress
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
You received this bug
** Description changed:
Installed 20091006, works alright.
Rebooted the cloud/cluster controller at 09:46
Started an instance (i-45410877) at 09:48:06
The instance stayed at pending (without an IP address assigned), then
terminated.
cloud-output.log shows an apparent DB deadlock
** Description changed:
Installed 20091006, works alright.
Rebooted the cloud/cluster controller at 09:46
Started an instance (i-45410877) at 09:48:06
The instance stayed at pending (without an IP address assigned), then
terminated.
cloud-output.log shows an apparent DB deadlock
Documented at
https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#UEC%20may%20refuse%20to%20serve%20the%20first%20requests%20received%20after%20startup:
On startup a UEC system may not process requests correctly, for instance
returning 403 Forbidden errors in response to some client requests.
This
Proposed Title:
Upon reboot, UEC might refuse to serve the first requests
Text:
On specific UEC systems we observed a database deadlock condition on reboot
that prevents a UEC cloud to serve the first 1 to 6 requests it receives, until
the deadlock is automatically cleared (444352).
--
DB
Maybe that's clearer:
Proposed Title:
UEC might refuse to serve the first requests received after boot
Text:
On specific UEC systems we observed a database deadlock condition after boot
that prevents a UEC cloud to answer correctly to the first requests it
receives, usually triggering 503
Hmm, I'm now experiencing this problem on my cluster, and it
particularly sucks. :-(
:-Dustin
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is
After waiting ~5 minutes, it seems to have become unwedged, and working
again. Looks like I need to reboot, and walk away for ~10 minutes, get
some coffee, and come back.
:-Dustin
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
You
Minor change also required to powernap, to test for the writability of
/var/cache/powerwake/ethers, before attempting to update the cache.
Committed upstream.
Now that I've patched both of these, Cloud Power Management once again
works as expected!
:-Dustin
--
DB deadlock on reboot prevents
Please disregard that last comment... Posted to the wrong bug. Sorry.
:-Dustin
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to
I ran into the same problem rebooting after and apt-get upgrade. Had to
restart the services manually to be able to log in again... Enclosing
relevant part of the log.
** Attachment added: cloud-output.log
http://launchpadlibrarian.net/33956025/cloud-output.log
--
DB deadlock on reboot
Too late to nail and fix for release, especially that deep in the code.
** Changed in: eucalyptus (Ubuntu Karmic)
Status: Triaged = Won't Fix
** Changed in: eucalyptus (Ubuntu Karmic)
Assignee: (unassigned) = Thierry Carrez (ttx)
--
DB deadlock on reboot prevents UEC from working,
Further testing seems to invalidate the eucalyptus started before
network theory.
I added a loop to wait for the external IP address to show up before
starting up eucalyptus. It does *not* prevent the DEADLOCK issues.
I added a loop to delay eucalyptus startup by 60 seconds arbitrarily,
checked
More testing:
Trying with static networking setup does *not* prevent DEADLOCK issues.
It's probably not linked to the stop procedure either, since I stop
eucalyptus cleanly before rebooting.
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
Finally I tested disabling eucalyptus start at boot:
(From a running system)
Stopping eucalyptus.
Clearing logs.
Starting eucalyptus manually
Testing: PASS. no deadlock.
Stopping eucalyptus.
Rebooting, waiting for system to be fully up.
Starting eucalyptus manually
Testing: FAIL. deadlock issue.
So to summarize:
- To trigger the bug, reboot your cluster and try running
euca-describe-availability-zones verbose once eucalyptus is ready. It fails,
and makes DEADLOCK errors appear in the logs.
- After a few tries, eucalyptus does a hard reset on connections and
autofixes itself.
- This
I think we need some confirmation here.
Anyone with a UEC setup, please do:
$ sudo stop eucalyptus
$ sudo rm /var/log/eucalyptus/*
$ sudo reboot
...
# wait for reboot complete and port 8443 to show up on netstat
$ . eucarc
$ euca-describe-availability-zones verbose
$ grep Reset
Thierry,
I ran all of these.
After reboot, I got a few Reset messages in cloud-output.log, like this:
11:21:56 WARN DatabaseUtil | - Resetting:
z8kfsx84g4e6c2if50wl|6e905272
But after clearing the logs, and warm restart of eucalyptus, port 8443
came up, my availability zones were
Thierry asked me to lower the priority, since I'm not reproducing this
problem now.
:-Dustin
** Changed in: eucalyptus (Ubuntu Karmic)
Importance: Medium = Low
--
DB deadlock on reboot prevents UEC from working, temporarily
https://bugs.launchpad.net/bugs/444352
You received this bug
Can't reproduce it on a stop/start eucalyptus. My hint is that we wedge
something when we reboot that we don't wedge on regular stop.
I can reproduce it every time on reboot. It takes 5 DB connections
before the system resets hard. Sometimes the system by itself will do
enough connections so that
From my limited upstart knowledge, what we do on regular stop that we appear
not doing on reboot is... stopping eucalyptus.
However adding stop on runlevel [!23] doesn't seem to be sufficient, so maybe
it's a red herring.
--
DB deadlock on reboot prevents UEC from working, temporarily
Lowering priority since its transient.
Probably related to networking not being up before Eucalyptus starts, since
apparently it only hits DHCP-based configs
** Changed in: eucalyptus (Ubuntu Karmic)
Importance: High = Medium
** Changed in: eucalyptus (Ubuntu Karmic)
Assignee: Dustin
32 matches
Mail list logo
