[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix". ** Changed in: cloud-init (Ubuntu Hardy) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Branch linked: lp:~ubuntu-branches/ubuntu/natty/cloud-init/natty- proposed ** Branch linked: lp:ubuntu/oneiric-proposed/cloud-init -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Branch linked: lp:ubuntu/lucid-updates/cloud-init -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Branch linked: lp:ubuntu/lucid-proposed/cloud-init ** Branch linked: lp:~ubuntu-branches/ubuntu/maverick/cloud-init /maverick-proposed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
apt (Ubuntu Precise) running Ubuntu in VM-ware through a proxy and a Secure-Web-Gateway. To disable the http pipeline when doing a dist-upgrade did the trick for me. (temporary) sudo apt-get -o Acquire::http::Pipeline-Depth="0" -y dist-upgrade (permanent) echo 'Acquire::http::Pipeline-Depth "0";' | sudo tee /etc/apt/apt.conf.d/99-no-pipelining -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
this has been mitigated by applying a fix to the EC2 images. ** Changed in: apt (Ubuntu Precise) Status: Confirmed => Won't Fix ** Changed in: apt (Ubuntu) Importance: High => Low ** Tags removed: rls-mgr-p-tracking ** Tags added: rls-p-nottracking -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
This bug was fixed in the package cloud-init - 0.5.15-0ubuntu3.1 --- cloud-init (0.5.15-0ubuntu3.1) maverick-proposed; urgency=low * add ability to configure Acquire::http::Pipeline-Depth via cloud-config setting 'apt_pipelining' (LP: #948461) * debian/cloud-init.postinst: address population of apt_pipeline setting on installation. -- Scott MoserFri, 16 Mar 2012 14:36:07 -0400 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
This bug was fixed in the package cloud-init - 0.6.1-0ubuntu22.1 --- cloud-init (0.6.1-0ubuntu22.1) oneiric-proposed; urgency=low * add ability to configure Acquire::http::Pipeline-Depth via cloud-config setting 'apt_pipelining' (LP: #948461) * debian/cloud-init.postinst: address population of apt_pipeline setting on installation. -- Ben HowardFri, 16 Mar 2012 15:44:50 -0600 ** Changed in: cloud-init (Ubuntu Maverick) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
This bug was fixed in the package cloud-init - 0.6.1-0ubuntu8.1 --- cloud-init (0.6.1-0ubuntu8.1) natty-proposed; urgency=low * add ability to configure Acquire::http::Pipeline-Depth via cloud-config setting 'apt_pipelining' (LP: #948461) * debian/cloud-init.postinst: address population of apt_pipeline setting on installation. -- Ben HowardFri, 16 Mar 2012 11:05:48 -0600 ** Changed in: cloud-init (Ubuntu Oneiric) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
This bug was fixed in the package cloud-init - 0.5.10-0ubuntu1.7 --- cloud-init (0.5.10-0ubuntu1.7) lucid-proposed; urgency=low * add ability to configure Acquire::http::Pipeline-Depth via cloud-config setting 'apt_pipelining' (LP: #948461) * debian/cloud-init.postinst: address population of apt_pipeline setting on installation. -- Scott MoserFri, 16 Mar 2012 14:32:50 -0400 ** Changed in: cloud-init (Ubuntu Lucid) Status: Fix Committed => Fix Released ** Changed in: cloud-init (Ubuntu Natty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Re #29: I plan on spinning a final Maverick on its EOF for historical reasons. The other reason why I would like to see Maverick get this to updates is for people who upgrade from Maverick to something newer. Re #27: I've confirmed the fixes in proposed work. We're good to go here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Changed in: apt (Ubuntu Oneiric) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Changed in: apt (Ubuntu Natty) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Tags added: rls-mgr-p-tracking -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Changed in: apt (Ubuntu Hardy) Status: New => Won't Fix ** Changed in: apt (Ubuntu Lucid) Status: New => Won't Fix ** Changed in: apt (Ubuntu Maverick) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
FYI, I accepted the maverick one, even though it is EOL very soon, thinking that it may be useful for generating one last AMI which people will use to migrate to natty. If no new cloud image will be produced, I don't see any reason to actually move that to maverick-updates. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Changed in: cloud-init (Ubuntu Lucid) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Hello Ben, or anyone else affected, Accepted cloud-init into maverick-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: cloud-init (Ubuntu Maverick) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Hello Ben, or anyone else affected, Accepted cloud-init into natty-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: cloud-init (Ubuntu Natty) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Hello Ben, or anyone else affected, Accepted cloud-init into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: cloud-init (Ubuntu Oneiric) Status: Triaged => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
I notice this bug is reported against both apt and cloud-init, but the patches are only for cloud-init. Are there any changes required of apt? If not, perhaps those tasks can all be closed out? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Description changed: - SRU Justification + [Problem] + When using S3 mirrors, the apt client requests files A,B,C,D in the same request, but S3 responds with files B,C,D,A or some other arbitrary ordering. Apt then saves the files under the wrong file names. + + This is due to a bug in Amazon AWS S3, where http pipelining does not + work in violation of RFC 2068. apt uses http pipelining currently, so + this prevents apt from using mirrors hosted on S3 cloud instances. + [Impact] - EC2 Instances cannot reliably use the new S3 Mirrors. - - Currently Canonical IS runs mirrors on instances in each EC2 - Availability zone. Ben Howard has put infrastructure in place to run - these mirrors entirely out of Amazon's S3. This will give better - performance and reliability for Ubuntu users in EC2. - - There is a bug in the S3 HTTP implementation that is exposed when using - apt with the pipelining turned on. Therefore, we cannot put these S3 - mirrors in place until after this fix is released. + Blocks use of S3 mirrors for providing ubuntu updates. Affects Hardy, Lucid, Maverick, Oneiric, Natty, and Precise. + + Use of S3 mirrors is considered as a way of enhancing our mirror + services. Currently mirrors have moments of instability, so having the + option of switching to S3 mirrors is very important for update + reliability. [Development Fix] - On installation or upgrade, cloud-init will populate /etc/apt/apt.conf.d/90cloud-init-pipelining with: - //Written by cloud-init per 'apt_pipelining - Acquire::http::Pipeline-Depth 0 - - After that, this will happen only once "per instance". The user can disable this function if they do not want it by modifying that file, or by providing cloud-init with userdata like: - #cloud-config - apt_pipelining: 0 - - This was first included in cloud-init upstream at revision 536, and into - Ubuntu precise in 0.6.3~bzr539-0ubuntu1 . - - [1] http://bazaar.launchpad.net/~cloud-init-dev/cloud- - init/trunk/revision/536 + Fixed in Precise's cloud-init as of these comments, which adds an apt_pipelining option, turned off by default but configurable. + http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/536 + http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/537 [Stable Fix] - Identical logic and behavior to the upstream fix has been ported back to each of 10.04, 10.10, 11.04, 11.10. + TODO [Text Case] 1) Launch an instance in EC2 - $ ubuntu-ec2-run $SUITE - 2) Select the S3 mirrors: -sed -i.dist "s,ec2.archive.ubuntu.com,ec2.archive.ubuntu.com.s3.amazonaws.com,g" /etc/apt/sources.list + 2) Run: sed -i "s,ec2.archive.ubuntu.com,ec2.archive.ubuntu.com.s3.amazonaws.com,g" /etc/apt/sources.list + 3.) Set your apt debug levels to whatever you want 4) Run "apt-get -y update" - 5) Run "apt-get --download-only -y install firefox xorg libreoffice" - - Alternatively for 4 and 5 you can just use the attachment 'apt-get - download loop' - - Broken Behavior: - Apt will fail eventually with 'Hash sum Mismatch' or some other download error. - - Fixed Behavior: - Run apt-get download forever with no errors. + 5) Run "apt-get -y install firefox xorg libreoffice > /tmp/install.log 2>&1" + Broken Behavior: Cached files are being misnamed + Fixed Behavior: Cached files are properly named + + The S3 EC2 mirrors seem to reproduce it most reliably. The downloaded + files are whole and match the proper checksums, but they have the wrong + content. [Regression Potential] - The potential is present for performance regression due to disabling pipelining. - However, enabling pipelining in apt was argued against by Robert Collins, an upstream squid developer. He has stated it is a "micro enhancement", and causes significant issues for many clients. - SRU Justification + http pipelining is an optimization technique in apt which allows multiple requests to be made at the same time. When it works, it provides a small performance boost, but when it doesn't the failures are arbitrary and not adequately explained. + + Thus, the expected regression is slightly slower apt behavior for non-S3 + cloud users. However, testing shows the impact is less than a few + seconds so will likely be unnoticed. In the off chance that it does + inadvertently cause non-trivial performance impact, the change is + configurable on a per-host instance. [Original Report] [SRU: Cloud-Init] r536 and r537 contains a fix that: 1.) Disables APT pipelining on first boot for new Cloud Images instances 2.) Disables APT pipelining on installation for existing cloud-images. The rational for this SRU is due to the Mirror situation for EC2 Cloud images. Currently, the mirrors are proving to have moments of instability that is causing both end-user and development pain. In order to address this, we built out Amazon AWS S3 mirrors. Unfortantly, due to a bug in S3, apt http pipeling doe
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Description changed: + SRU Justification [Impact] - + EC2 Instances cannot reliably use the new S3 Mirrors. + + Currently Canonical IS runs mirrors on instances in each EC2 + Availability zone. Ben Howard has put infrastructure in place to run + these mirrors entirely out of Amazon's S3. This will give better + performance and reliability for Ubuntu users in EC2. + + There is a bug in the S3 HTTP implementation that is exposed when using + apt with the pipelining turned on. Therefore, we cannot put these S3 + mirrors in place until after this fix is released. [Development Fix] - + On installation or upgrade, cloud-init will populate /etc/apt/apt.conf.d/90cloud-init-pipelining with: + //Written by cloud-init per 'apt_pipelining + Acquire::http::Pipeline-Depth 0 + + After that, this will happen only once "per instance". The user can disable this function if they do not want it by modifying that file, or by providing cloud-init with userdata like: + #cloud-config + apt_pipelining: 0 + + This was first included in cloud-init upstream at revision 536, and into + Ubuntu precise in 0.6.3~bzr539-0ubuntu1 . + + [1] http://bazaar.launchpad.net/~cloud-init-dev/cloud- + init/trunk/revision/536 [Stable Fix] - + Identical logic and behavior to the upstream fix has been ported back to each of 10.04, 10.10, 11.04, 11.10. [Text Case] - - 1. - 2. - 3. - Broken Behavior: - Fixed Behavior: + 1) Launch an instance in EC2 + $ ubuntu-ec2-run $SUITE + 2) Select the S3 mirrors: +sed -i.dist "s,ec2.archive.ubuntu.com,ec2.archive.ubuntu.com.s3.amazonaws.com,g" /etc/apt/sources.list + 4) Run "apt-get -y update" + 5) Run "apt-get --download-only -y install firefox xorg libreoffice" + + Alternatively for 4 and 5 you can just use the attachment 'apt-get + download loop' + + Broken Behavior: + Apt will fail eventually with 'Hash sum Mismatch' or some other download error. + + Fixed Behavior: + Run apt-get download forever with no errors. [Regression Potential] - http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/536 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/537 ___ apt-get is appears to be mangling the local caching file names, which is appearing as a hashsum or size mismatch. Evidence below: ___ Output of "apt-get -y install firefox": Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/p/python-defer/python-defer_1.0.2+bzr481-1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/u/ubufox/xul-ext-ubufox_2.0-0ubuntu1_all.deb Size mismatch ___ apt-get -y install firefox with debug for HTTP turned on: Get:63 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-defer all 1.0.2+bzr481-1 [10.9 kB] HTTP/1.1 200 OK^M x-amz-id-2: EXccd6GVMfE6ly4SYdwy313VK42d/gI3ncyxmaotuVIMbBBi6FJkuDYWrzLw7vXE^M x-amz-request-id: 7572BAB5E6FFFAC0^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 03 Feb 2012 08:54:27 GMT^M ETag: "6c07f8db615cc32657b1cc57450a1608"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 15142^M Server: AmazonS3^M ^M Get:64 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-aptdaemon all 0.43+bzr769-0ubuntu1 [80.3 kB] HTTP/1.1 200 OK^M x-amz-id-2: IcNECY1OtzHOXnTnTW3fqoJGyfdODq+qRXXGYTDEOJxfw0CP9UH8EUSllVR3Gf3F^M x-amz-request-id: 5D629E3BA012^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 02 Mar 2012 20:55:40 GMT^M ETag: "501373631a241847cf76d13aea9264b3"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 56824^M Server: AmazonS3^M ^M Get:65 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main aptdaemon all 0.43+bzr769-0ubuntu1 [15.1 kB] HTTP/1.1 200 OK^M x-amz-id-2: HHNSOZKH74D4I1XCaOiwDOzIWxIVJ88ibfKxvIp/4NpkczzPmsOhMqxrTj2T02qO^M x-amz-request-id: A6FD3143E6A0BA1A^M Date: Tue, 06 Mar 2012 20:06:34 GMT^M Last-Modified: Sat, 21 Jan 2012 00:48:55 GMT^M ETag: "3653165af1f20a437a14632ce0a2e6c2"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 10902^M Server: AmazonS3^M __
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
These are all uploaded and waiting: * lucid: https://launchpad.net/ubuntu/lucid/+queue?queue_state=1&queue_text=cloud-init * maverick: https://launchpad.net/ubuntu/maverick/+queue?queue_state=1&queue_text=cloud-init * natty: https://launchpad.net/ubuntu/natty/+queue?queue_state=1&queue_text=cloud-init * oneiric: https://launchpad.net/ubuntu/oneiric/+queue?queue_state=1&queue_text=cloud-init -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Description changed: + [Impact] + + + [Development Fix] + + + [Stable Fix] + + + [Text Case] + + 1. + 2. + 3. + Broken Behavior: + Fixed Behavior: + + [Regression Potential] + http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/536 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/537 - ___ apt-get is appears to be mangling the local caching file names, which is appearing as a hashsum or size mismatch. Evidence below: ___ Output of "apt-get -y install firefox": Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/p/python-defer/python-defer_1.0.2+bzr481-1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/u/ubufox/xul-ext-ubufox_2.0-0ubuntu1_all.deb Size mismatch ___ apt-get -y install firefox with debug for HTTP turned on: Get:63 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-defer all 1.0.2+bzr481-1 [10.9 kB] HTTP/1.1 200 OK^M x-amz-id-2: EXccd6GVMfE6ly4SYdwy313VK42d/gI3ncyxmaotuVIMbBBi6FJkuDYWrzLw7vXE^M x-amz-request-id: 7572BAB5E6FFFAC0^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 03 Feb 2012 08:54:27 GMT^M ETag: "6c07f8db615cc32657b1cc57450a1608"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 15142^M Server: AmazonS3^M ^M Get:64 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-aptdaemon all 0.43+bzr769-0ubuntu1 [80.3 kB] HTTP/1.1 200 OK^M x-amz-id-2: IcNECY1OtzHOXnTnTW3fqoJGyfdODq+qRXXGYTDEOJxfw0CP9UH8EUSllVR3Gf3F^M x-amz-request-id: 5D629E3BA012^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 02 Mar 2012 20:55:40 GMT^M ETag: "501373631a241847cf76d13aea9264b3"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 56824^M Server: AmazonS3^M ^M Get:65 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main aptdaemon all 0.43+bzr769-0ubuntu1 [15.1 kB] HTTP/1.1 200 OK^M x-amz-id-2: HHNSOZKH74D4I1XCaOiwDOzIWxIVJ88ibfKxvIp/4NpkczzPmsOhMqxrTj2T02qO^M x-amz-request-id: A6FD3143E6A0BA1A^M Date: Tue, 06 Mar 2012 20:06:34 GMT^M Last-Modified: Sat, 21 Jan 2012 00:48:55 GMT^M ETag: "3653165af1f20a437a14632ce0a2e6c2"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 10902^M Server: AmazonS3^M ___ Evaluation of the MD5sums of downloaded failed items: root@ip-10-6-85-206:/var/cache/apt/archives/partial# md5sum * 501373631a241847cf76d13aea9264b3 aptdaemon_0.43+bzr769-0ubuntu1_all.deb 6c07f8db615cc32657b1cc57450a1608 python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb 566f7c92a9fcd0c5c40d08a977176291 python-defer_1.0.2+bzr481-1_all.deb 3653165af1f20a437a14632ce0a2e6c2 xul-ext-ubufox_2.0-0ubuntu1_all.deb ___ Snipets from Packages.bz2 From Meta-data: Package: aptdaemon MD5sum: 6c07f8db615cc32657b1cc57450a1608 Package: xul-ext-ubufox MD5sum: 501373631a241847cf76d13aea9264b3 Package: python-aptdaemon MD5sum: 566f7c92a9fcd0c5c40d08a977176291 Package: python-defer MD5sum: 3653165af1f20a437a14632ce0a2e6c2 ___ "dpkg -i" on pacakges downloaded root@ip-10-6-85-206:/var/cache/apt/archives/partial# dpkg -I aptdaemon_0.43+bzr769-0ubuntu1_all.deb new debian package, version 2.0. size 56824 bytes: control archive= 1520 bytes. 23 bytes, 1 lines conffiles 1023 bytes,29 lines control 1804 bytes,21 lines md5sums Package: xul-ext-ubufox Source: ubufox Version: 2.0-0ubuntu1 Architecture: all Maintainer: Ubuntu Mozilla Team Installed-Size: 383 Depends: aptdaemon, libglib2.0-0 (>= 2.26) Recommends: firefox (>= 4.0~b6) Enhances: firefox Breaks: ubufox (<< 0.9~rc2-0ubuntu3)a Replaces: ubufox (<< 0.9~rc2-0ubuntu3) Provides: firefox-ubufox, ubufox Section: web Priority: optional Homepage: https://launchpad.net/ubufox Description: Ubuntu-specific configurati
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
Ben and I did backports, and I've uploaded to lucid, maverick, natty, oneiric. We're waiting on approval for opulation into -proposed. The plan from there is to get testing and population into -updates. Then, * release new images including this modification * give some time for existing instances to populate * email to different lists announcing need for the update/change * switch S3 mirrors into production. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Description changed: - apt-get is appears to be mangling the local caching file names, which is - appearing as a hashsum or size mismatch. + [SRU: Cloud-Init] + r536 and r537 contains a fix that: + 1.) Disables APT pipelining on first boot for new Cloud Images instances + 2.) Disables APT pipelining on installation for existing cloud-images. + + The rational for this SRU is due to the Mirror situation for EC2 Cloud + images. Currently, the mirrors are proving to have moments of + instability that is causing both end-user and development pain. In order + to address this, we built out Amazon AWS S3 mirrors. Unfortantly, due to + a bug in S3, apt http pipeling does not work with S3. As a result, in + order to provide enhanced availability for the Cloud Image repositories + in EC2, apt http pipelining needs to be disabled ahead of flipping the + new S3 mirrors as the active mirrors. + + The performance impact of this change is unnoticable and in testing, has + proven to provide no ill effect. http pipeplining allows for multiple + requests to be made at the same time. The apt change log states that + http pipelining is a "micro-enhancement" and within the EC2 environment + shaves a few seconds of many requests. When apt http pipeling works, at + best, it saves a few seconds, when it fails against a buggy HTTP1.1 + server or behind a proxy that does not support it, the failures are + arbitrary and not adiquetly explained. This change will enhance the + Cloud Images by making apt request packages sequentially instead of in + batches and prevent arbitrary failures caused by non-compliant web + servers. Cloud image users, both on EC2 and outside, will benefit from + this fix. + + http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/536 + http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/537 + + + ___ + apt-get is appears to be mangling the local caching file names, which is appearing as a hashsum or size mismatch. Evidence below: ___ Output of "apt-get -y install firefox": Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/p/python-defer/python-defer_1.0.2+bzr481-1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/u/ubufox/xul-ext-ubufox_2.0-0ubuntu1_all.deb Size mismatch ___ apt-get -y install firefox with debug for HTTP turned on: Get:63 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-defer all 1.0.2+bzr481-1 [10.9 kB] HTTP/1.1 200 OK^M x-amz-id-2: EXccd6GVMfE6ly4SYdwy313VK42d/gI3ncyxmaotuVIMbBBi6FJkuDYWrzLw7vXE^M x-amz-request-id: 7572BAB5E6FFFAC0^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 03 Feb 2012 08:54:27 GMT^M ETag: "6c07f8db615cc32657b1cc57450a1608"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 15142^M Server: AmazonS3^M ^M Get:64 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-aptdaemon all 0.43+bzr769-0ubuntu1 [80.3 kB] HTTP/1.1 200 OK^M x-amz-id-2: IcNECY1OtzHOXnTnTW3fqoJGyfdODq+qRXXGYTDEOJxfw0CP9UH8EUSllVR3Gf3F^M x-amz-request-id: 5D629E3BA012^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 02 Mar 2012 20:55:40 GMT^M ETag: "501373631a241847cf76d13aea9264b3"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 56824^M Server: AmazonS3^M ^M Get:65 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main aptdaemon all 0.43+bzr769-0ubuntu1 [15.1 kB] HTTP/1.1 200 OK^M x-amz-id-2: HHNSOZKH74D4I1XCaOiwDOzIWxIVJ88ibfKxvIp/4NpkczzPmsOhMqxrTj2T02qO^M x-amz-request-id: A6FD3143E6A0BA1A^M Date: Tue, 06 Mar 2012 20:06:34 GMT^M Last-Modified: Sat, 21 Jan 2012 00:48:55 GMT^M ETag: "3653165af1f20a437a14632ce0a2e6c2"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 10902^M Server: AmazonS3^M ___ Evaluation of the MD5sums of downloaded failed items: root@ip-10-6-85-206:/var/cache/apt/archives/partial# md5sum * 501373631a241847cf76d13aea9264b3 aptdaemon_0.43+bzr769-0ubuntu1_all.deb 6c07f8db615cc32657b1cc57450a1608 python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb 5
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Branch linked: lp:~smoser/ubuntu/lucid/cloud-init/lp942961-apt- pipeline ** Branch linked: lp:~smoser/ubuntu/maverick/cloud-init/lp942961-apt- pipeline -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 948461] Re: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly
** Also affects: cloud-init (Ubuntu) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu Precise) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Precise) Status: New => Fix Released ** Changed in: cloud-init (Ubuntu Oneiric) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Oneiric) Status: New => Triaged ** Changed in: cloud-init (Ubuntu Natty) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Natty) Status: New => Triaged ** Changed in: cloud-init (Ubuntu Maverick) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Maverick) Status: New => Triaged ** Changed in: cloud-init (Ubuntu Lucid) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Lucid) Status: New => Triaged ** Changed in: cloud-init (Ubuntu Hardy) Importance: Undecided => High ** Changed in: cloud-init (Ubuntu Hardy) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to cloud-init in Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
