[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus/1.6.2 Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in Ubuntu. https://bugs.launchpad.net/bugs/590929 Title: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) To manage notifications about this bug go to: https://bugs.launchpad.net/eucalyptus/+bug/590929/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus/1.6.2 Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/590929 Title: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) To manage notifications about this bug go to: https://bugs.launchpad.net/eucalyptus/+bug/590929/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in Ubuntu. https://bugs.launchpad.net/bugs/590929 Title: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) To manage notifications about this bug go to: https://bugs.launchpad.net/eucalyptus/+bug/590929/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/590929 Title: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) To manage notifications about this bug go to: https://bugs.launchpad.net/eucalyptus/+bug/590929/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus (Ubuntu) Status: Confirmed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu30.4 --- eucalyptus (1.6.2-0ubuntu30.4) lucid-proposed; urgency=low * debian/patches/590929.diff: cherry pick upstream bzr r1235..1236 to fix LP: #590929; volume creation/deletion issues; add newly used LVM commands to wrappers.conf -- Dustin Kirkland kirkl...@ubuntu.com Wed, 01 Sep 2010 17:33:09 -0600 ** Changed in: eucalyptus (Ubuntu Lucid) Status: Fix Committed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Changed in: eucalyptus (Ubuntu) Status: Confirmed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu30.4 --- eucalyptus (1.6.2-0ubuntu30.4) lucid-proposed; urgency=low * debian/patches/590929.diff: cherry pick upstream bzr r1235..1236 to fix LP: #590929; volume creation/deletion issues; add newly used LVM commands to wrappers.conf -- Dustin Kirkland kirkl...@ubuntu.com Wed, 01 Sep 2010 17:33:09 -0600 ** Changed in: eucalyptus (Ubuntu Lucid) Status: Fix Committed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've performed a full re-installation of the frontend as requested. Before any tests or registration, I've updated all eucalyptus packages to 1.6.2-0ubuntu30.4. I've also adjusted cluster1.storage.maxtotalvolumesizeingb in the web front end (set to 250). I did 10 test loops (as usual creating 40 volumes). + No Internal Errors were reported. + Space available and # of volumes correctly reported in sc-stats.log + Deletion worked perfectly But I found one (just one) failure reported by euca-describe-volumes, 10 seconds after the creation loop : VOLUME vol-526005D5 1 cluster1failed 2010-09-15T17:00:32.7Z A few seconds later, euca-describe-volumes didn't report anything about this failure, weird. But I had only 39 volumes created. This is what was reported in cloud-error.log: 12:00:34 [SystemUtil:pool-10-thread-4] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap vgcreate vg-lp4W3A.. /dev/loop18 error: /var/lock/lvm/P_orphans: flock failed: Resource temporarily unavailable Can't get lock for orphan PVs com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap vgcreate vg-lp4W3A.. /dev/loop18 error: /var/lock/lvm/P_orphans: flock failed: Resource temporarily unavailable Can't get lock for orphan PVs at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91) at edu.ucsb.eucalyptus.storage.LVM2Manager.createVolumeGroup(LVM2Manager.java:170) at edu.ucsb.eucalyptus.storage.LVM2Manager.createLogicalVolume(LVM2Manager.java:430) at edu.ucsb.eucalyptus.storage.LVM2Manager.createVolume(LVM2Manager.java:476) at edu.ucsb.eucalyptus.cloud.ws.BlockStorage$VolumeCreator.run(BlockStorage.java:798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 12:00:34 [BlockStorage:pool-10-thread-4] ERROR com.eucalyptus.util.EucalyptusCloudException: Unable to create volume group vg-lp4W3A.. for /dev/loop18 -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've performed a full re-installation of the frontend as requested. Before any tests or registration, I've updated all eucalyptus packages to 1.6.2-0ubuntu30.4. I've also adjusted cluster1.storage.maxtotalvolumesizeingb in the web front end (set to 250). I did 10 test loops (as usual creating 40 volumes). + No Internal Errors were reported. + Space available and # of volumes correctly reported in sc-stats.log + Deletion worked perfectly But I found one (just one) failure reported by euca-describe-volumes, 10 seconds after the creation loop : VOLUME vol-526005D5 1 cluster1failed 2010-09-15T17:00:32.7Z A few seconds later, euca-describe-volumes didn't report anything about this failure, weird. But I had only 39 volumes created. This is what was reported in cloud-error.log: 12:00:34 [SystemUtil:pool-10-thread-4] ERROR com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap vgcreate vg-lp4W3A.. /dev/loop18 error: /var/lock/lvm/P_orphans: flock failed: Resource temporarily unavailable Can't get lock for orphan PVs com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap vgcreate vg-lp4W3A.. /dev/loop18 error: /var/lock/lvm/P_orphans: flock failed: Resource temporarily unavailable Can't get lock for orphan PVs at edu.ucsb.eucalyptus.util.SystemUtil.run(SystemUtil.java:91) at edu.ucsb.eucalyptus.storage.LVM2Manager.createVolumeGroup(LVM2Manager.java:170) at edu.ucsb.eucalyptus.storage.LVM2Manager.createLogicalVolume(LVM2Manager.java:430) at edu.ucsb.eucalyptus.storage.LVM2Manager.createVolume(LVM2Manager.java:476) at edu.ucsb.eucalyptus.cloud.ws.BlockStorage$VolumeCreator.run(BlockStorage.java:798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 12:00:34 [BlockStorage:pool-10-thread-4] ERROR com.eucalyptus.util.EucalyptusCloudException: Unable to create volume group vg-lp4W3A.. for /dev/loop18 -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've upgraded the system to euca* 1.6.2-0ubuntu30.4 (much better that only the SC). I hope that rev 1235/1236 are in this release but according to the delivery dates it should be. So the status is better but I still see some errors while creating volumes with a loop (40) : Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. With the creation of 40 volumes, this error is reported only one time (max 3 times). The deletion loop does not report any errors and a vgscan shows a clean system, great. If I run the creation loop several times I see in sc-stats.log that the amount of space used and the number of managed volumes never decrease. Above the limit (Disk space reserved for volumes) the previous error msg is returned for each creation request. The *broken* system has been upgraded with lucid-proposed packages, rebooted before executing the test. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Hi Sylvain, Can you please attach the /var/log/eucalyptus of all the machines showing the error? Also, exact sequence of commands and output would help. Thank you. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Neil wanted the testing to be done on a perfectly clean, newly installed system. Neil, what do you make of Sylvain's test procedure? -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Proper error messages are getting reported on upgraded systems without the residual artefacts that were reported in the bug: verified, -done. ** Changed in: jaguarundi Status: Confirmed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Tags added: verification-done ** Tags removed: verification-needed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've upgraded the system to euca* 1.6.2-0ubuntu30.4 (much better that only the SC). I hope that rev 1235/1236 are in this release but according to the delivery dates it should be. So the status is better but I still see some errors while creating volumes with a loop (40) : Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. With the creation of 40 volumes, this error is reported only one time (max 3 times). The deletion loop does not report any errors and a vgscan shows a clean system, great. If I run the creation loop several times I see in sc-stats.log that the amount of space used and the number of managed volumes never decrease. Above the limit (Disk space reserved for volumes) the previous error msg is returned for each creation request. The *broken* system has been upgraded with lucid-proposed packages, rebooted before executing the test. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Hi Sylvain, Can you please attach the /var/log/eucalyptus of all the machines showing the error? Also, exact sequence of commands and output would help. Thank you. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Neil wanted the testing to be done on a perfectly clean, newly installed system. Neil, what do you make of Sylvain's test procedure? -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Make sure you increase the disk space reserved for volumes (default is 50G) if you expect volume creation beyond that limit to succeed :) Also, not sure I read the previous message correctly. There are fixes to the CLC as well as the SC so you will need to upgrade and restart both. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Proper error messages are getting reported on upgraded systems without the residual artefacts that were reported in the bug: verified, -done. ** Changed in: jaguarundi Status: Confirmed = Fix Released -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Tags added: verification-done ** Tags removed: verification-needed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Sylvain, Did you install from scratch, or upgrade the broken system? :-Dustin -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Sylvain, Did you install from scratch, or upgrade the broken system? :-Dustin -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've applied the fix from http://archive.ubuntu.com/ubuntu/pool/main/e/eucalyptus/eucalyptus- sc_1.6.2-0ubuntu30.4_amd64.deb and unfortunately the creation loop still reports some errors : Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. Deletion loop iterates well but volumes are still present in /var/lib/eucalyptus/volumes and a quick 'sudo losetup -a' shows them all. ** Changed in: eucalyptus (Ubuntu) Status: Fix Released = Confirmed ** Changed in: jaguarundi Status: In Progress = Confirmed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Sylvain, are you positive that you are running the right version? Did you start from a clean state? Are there any other error messages on the storage controller? Is lvdisplay, vgdisplay and pvdisplay added to the white list of commands? We have tested this several times on our end and I'm puzzled that you'd still see this, but we'd be happy to take a look. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I've applied the fix from http://archive.ubuntu.com/ubuntu/pool/main/e/eucalyptus/eucalyptus- sc_1.6.2-0ubuntu30.4_amd64.deb and unfortunately the creation loop still reports some errors : Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. Deletion loop iterates well but volumes are still present in /var/lib/eucalyptus/volumes and a quick 'sudo losetup -a' shows them all. ** Changed in: eucalyptus (Ubuntu) Status: Fix Released = Confirmed ** Changed in: jaguarundi Status: In Progress = Confirmed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Sylvain, are you positive that you are running the right version? Did you start from a clean state? Are there any other error messages on the storage controller? Is lvdisplay, vgdisplay and pvdisplay added to the white list of commands? We have tested this several times on our end and I'm puzzled that you'd still see this, but we'd be happy to take a look. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Accepted eucalyptus into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: eucalyptus (Ubuntu Lucid) Status: In Progress = Fix Committed ** Tags added: verification-needed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Branch linked: lp:ubuntu/lucid-proposed/eucalyptus -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
As far as volume-creation/deletion goes, it seems to be running like a charm. The only small thing is when deleting volumes cloud-debug.log gets at least one error logged for each volume: 17:29:44 ERROR [SystemUtil:pool-10-thread-4] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address Not sure this _is_ a problem, but here it is. Even with these errors being reported, creation/deletion succeeded for 1,586 volumes (and yes, there were 1,586 lines as above printed out in the cloud-debug log). I am not yet tagging 'verification-done' because I still want to run an instance test. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Oh, forgot, sorry: logs saved as usual: Using saved push location: bzr+ssh://bazaar.launchpad.net/~hggdh2/%2Bjunk/uec-qa/ Pushed up to revision 52. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
17:29:44 ERROR [SystemUtil:pool-10-thread-4] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address Carlos that error is actually not a real error, it is unfortunately tagged as such when logging. The Storage Controller is trying to make sure that the loop device exists before trying to use it or remove it, basically the equivalent of a stat. We can fix the way this is being reported, but it is not critical. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Cool. I could not see it as an error anyway, but it is good to have your confirmation. And I agree, not critical at all. Thank you, Neil. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I consider this 'verification-done', but will not change the tag until OEM QA run their own tests. Instance run tests results saved: Using saved push location: bzr+ssh://bazaar.launchpad.net/~hggdh2/%2Bjunk/uec-qa/ Pushed up to revision 53. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to eucalyptus in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Accepted eucalyptus into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: eucalyptus (Ubuntu Lucid) Status: In Progress = Fix Committed ** Tags added: verification-needed -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Branch linked: lp:ubuntu/lucid-proposed/eucalyptus -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
As far as volume-creation/deletion goes, it seems to be running like a charm. The only small thing is when deleting volumes cloud-debug.log gets at least one error logged for each volume: 17:29:44 ERROR [SystemUtil:pool-10-thread-4] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address Not sure this _is_ a problem, but here it is. Even with these errors being reported, creation/deletion succeeded for 1,586 volumes (and yes, there were 1,586 lines as above printed out in the cloud-debug log). I am not yet tagging 'verification-done' because I still want to run an instance test. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Oh, forgot, sorry: logs saved as usual: Using saved push location: bzr+ssh://bazaar.launchpad.net/~hggdh2/%2Bjunk/uec-qa/ Pushed up to revision 52. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
17:29:44 ERROR [SystemUtil:pool-10-thread-4] com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address com.eucalyptus.util.ExecutionException: ///usr/lib/eucalyptus/euca_rootwrap losetup /dev/loop151 error: loop: can't get info on device /dev/loop151: No such device or address Carlos that error is actually not a real error, it is unfortunately tagged as such when logging. The Storage Controller is trying to make sure that the loop device exists before trying to use it or remove it, basically the equivalent of a stat. We can fix the way this is being reported, but it is not critical. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
Cool. I could not see it as an error anyway, but it is good to have your confirmation. And I agree, not critical at all. Thank you, Neil. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
I consider this 'verification-done', but will not change the tag until OEM QA run their own tests. Instance run tests results saved: Using saved push location: bzr+ssh://bazaar.launchpad.net/~hggdh2/%2Bjunk/uec-qa/ Pushed up to revision 53. -- [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs) https://bugs.launchpad.net/bugs/590929 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Description changed:
When testing out bug 586134 I found another issue and although similar
to bug 517086 appears not to be the same issue.
If you create for example 200 1GB volumes like this:
for i in `seq 1 200` ; do euca-create-volume -z cluster1 -s 1 ; done
You will end up with quite a few failed. If you look in the logs it is
usually caused by vgcreate failing for unknown reasons as the output of
the command is not logged. If you run the same command on the command
line that failed it will then work.
Also if you then delete all the volumes like this:
for i in `euca-describe-volumes | awk '{print $2}'` ; do euca-delete-
volume $i ; done
You will end up with a huge number that do not get removed. If you look
in the logs it is usually caused by lvremove failing for unknown reasons
like with vgcreate. Again if you run the same command on the command
line that failed it will then work.
Also despite the failure to properly delete the volume euca-describe-
volumes will have invalid information. For example if you delete the
volume it will no longer show up in the list but it will still be there
as seen by losetup and vgscan, and will still be set back up by
eucalyptus when rebooted but it will not show up in the euca-describe-
volume list.
Chris
+
+ ===
+ SRU INFORMATION
+
+ IMPACT:
+ This bug affects users of UEC, trying to create EBS volumes using the
euca-create-volume command. In the current packages, this command will fail
from time to time, for no apparent or reported reason, and Eucalyptus does not
recover gracefully and retry the volume creation. This is a problem that heavy
UEC users will probably encounter at some point in time.
+
+ HOW ADDRESSED:
+ This bug has been addressed by the upstream Eucalyptus maintainer of this
code. The Eucalyptus 2.0 branch (in Maverick) does not suffer from this bug,
as much of the storage related code has been refactored in the interest of
scalability. Neil Soman has graciously backported the minimal subset of those
changes necessary to address this bug. Still the patch is a few hundred lines.
But it's all in the interest of solving this bug.
+
+ PATCH:
+ See bzr revisions r1235 and r1236 of lp:eucalyptus (upstream Eucalyptus
branch).
+
+ REPRODUCE:
+ To reproduce this bug, simply:
+ for i in $(seq 1 40); do euca-create-volume -z cluster1 -s 1; done
+ sleep 10
+ euca-describe-volumes | grep fail
+ euca-describe-volumes | awk '{print $2}' | xargs -i euca-delete-volume {}
+ I see failures trying to create a few as 40 volumes. (Note that I chose 40,
as I have 50GB of disk free). We would expect being able to do the above ad
nauseum, without unhandled errors.
+
+ REGRESSION POTENTIAL:
+ There is perhaps some chance of regression in the storage related
functionality in Eucalyptus. For this reason, this particular SRU package will
be heavily tested by Canonical OEM Server QA as well as Ubuntu Platform QA and
Ubuntu Platform Server team members.
+
+ ===
** Changed in: eucalyptus (Ubuntu Lucid)
Milestone: None = ubuntu-10.04.2
--
[SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC
+ ENT configs)
https://bugs.launchpad.net/bugs/590929
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 590929] Re: [SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC + ENT configs)
** Description changed:
When testing out bug 586134 I found another issue and although similar
to bug 517086 appears not to be the same issue.
If you create for example 200 1GB volumes like this:
for i in `seq 1 200` ; do euca-create-volume -z cluster1 -s 1 ; done
You will end up with quite a few failed. If you look in the logs it is
usually caused by vgcreate failing for unknown reasons as the output of
the command is not logged. If you run the same command on the command
line that failed it will then work.
Also if you then delete all the volumes like this:
for i in `euca-describe-volumes | awk '{print $2}'` ; do euca-delete-
volume $i ; done
You will end up with a huge number that do not get removed. If you look
in the logs it is usually caused by lvremove failing for unknown reasons
like with vgcreate. Again if you run the same command on the command
line that failed it will then work.
Also despite the failure to properly delete the volume euca-describe-
volumes will have invalid information. For example if you delete the
volume it will no longer show up in the list but it will still be there
as seen by losetup and vgscan, and will still be set back up by
eucalyptus when rebooted but it will not show up in the euca-describe-
volume list.
Chris
+
+ ===
+ SRU INFORMATION
+
+ IMPACT:
+ This bug affects users of UEC, trying to create EBS volumes using the
euca-create-volume command. In the current packages, this command will fail
from time to time, for no apparent or reported reason, and Eucalyptus does not
recover gracefully and retry the volume creation. This is a problem that heavy
UEC users will probably encounter at some point in time.
+
+ HOW ADDRESSED:
+ This bug has been addressed by the upstream Eucalyptus maintainer of this
code. The Eucalyptus 2.0 branch (in Maverick) does not suffer from this bug,
as much of the storage related code has been refactored in the interest of
scalability. Neil Soman has graciously backported the minimal subset of those
changes necessary to address this bug. Still the patch is a few hundred lines.
But it's all in the interest of solving this bug.
+
+ PATCH:
+ See bzr revisions r1235 and r1236 of lp:eucalyptus (upstream Eucalyptus
branch).
+
+ REPRODUCE:
+ To reproduce this bug, simply:
+ for i in $(seq 1 40); do euca-create-volume -z cluster1 -s 1; done
+ sleep 10
+ euca-describe-volumes | grep fail
+ euca-describe-volumes | awk '{print $2}' | xargs -i euca-delete-volume {}
+ I see failures trying to create a few as 40 volumes. (Note that I chose 40,
as I have 50GB of disk free). We would expect being able to do the above ad
nauseum, without unhandled errors.
+
+ REGRESSION POTENTIAL:
+ There is perhaps some chance of regression in the storage related
functionality in Eucalyptus. For this reason, this particular SRU package will
be heavily tested by Canonical OEM Server QA as well as Ubuntu Platform QA and
Ubuntu Platform Server team members.
+
+ ===
** Changed in: eucalyptus (Ubuntu Lucid)
Milestone: None = ubuntu-10.04.2
--
[SRU] eucalyptus create and delete volumes sometimes fail on lvm commands (POC
+ ENT configs)
https://bugs.launchpad.net/bugs/590929
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
