** Description changed: + ==== SRU Justification ==== [Impact] - <fill me in with explanation of severity and frequency of bug on users and justification for backporting the fix to the stable release> + EC2 Instances cannot reliably use the new S3 Mirrors. + + Currently Canonical IS runs mirrors on instances in each EC2 + Availability zone. Ben Howard has put infrastructure in place to run + these mirrors entirely out of Amazon's S3. This will give better + performance and reliability for Ubuntu users in EC2. + + There is a bug in the S3 HTTP implementation that is exposed when using + apt with the pipelining turned on. Therefore, we cannot put these S3 + mirrors in place until after this fix is released. [Development Fix] - <fill me in with an explanation of how the bug has been addressed in the development branch, including the relevant version numbers of packages modified in order to implement the fix. > + On installation or upgrade, cloud-init will populate /etc/apt/apt.conf.d/90cloud-init-pipelining with: + //Written by cloud-init per 'apt_pipelining + Acquire::http::Pipeline-Depth 0 + + After that, this will happen only once "per instance". The user can disable this function if they do not want it by modifying that file, or by providing cloud-init with userdata like: + #cloud-config + apt_pipelining: 0 + + This was first included in cloud-init upstream at revision 536, and into + Ubuntu precise in 0.6.3~bzr539-0ubuntu1 . + + [1] http://bazaar.launchpad.net/~cloud-init-dev/cloud- + init/trunk/revision/536 [Stable Fix] - <fill me in by pointing out a minimal patch applicable to the stable version of the package.> + Identical logic and behavior to the upstream fix has been ported back to each of 10.04, 10.10, 11.04, 11.10. [Text Case] - <fill me in with detailed *instructions* on how to reproduce the bug. This will be used by people later on to verify the updated package fixes the problem.> - 1. - 2. - 3. - Broken Behavior: - Fixed Behavior: + 1) Launch an instance in EC2 + $ ubuntu-ec2-run $SUITE + 2) Select the S3 mirrors: + sed -i.dist "s,ec2.archive.ubuntu.com,ec2.archive.ubuntu.com.s3.amazonaws.com,g" /etc/apt/sources.list + 4) Run "apt-get -y update" + 5) Run "apt-get --download-only -y install firefox xorg libreoffice" + + Alternatively for 4 and 5 you can just use the attachment 'apt-get + download loop' + + Broken Behavior: + Apt will fail eventually with 'Hash sum Mismatch' or some other download error. + + Fixed Behavior: + Run apt-get download forever with no errors. [Regression Potential] - <fill me in with a discussion of likelihood and potential severity of regressions and how users could get inadvertently affected. + The potential is present for performance regression due to disabling pipelining. + However, enabling pipelining in apt was argued against by Robert Collins, an upstream squid developer. He has stated it is a "micro enhancement", and causes significant issues for many clients. + ==== SRU Justification ==== [Original Report] [SRU: Cloud-Init] r536 and r537 contains a fix that: 1.) Disables APT pipelining on first boot for new Cloud Images instances 2.) Disables APT pipelining on installation for existing cloud-images. The rational for this SRU is due to the Mirror situation for EC2 Cloud images. Currently, the mirrors are proving to have moments of instability that is causing both end-user and development pain. In order to address this, we built out Amazon AWS S3 mirrors. Unfortantly, due to a bug in S3, apt http pipeling does not work with S3. As a result, in order to provide enhanced availability for the Cloud Image repositories in EC2, apt http pipelining needs to be disabled ahead of flipping the new S3 mirrors as the active mirrors. The performance impact of this change is unnoticable and in testing, has proven to provide no ill effect. http pipeplining allows for multiple requests to be made at the same time. The apt change log states that http pipelining is a "micro-enhancement" and within the EC2 environment shaves a few seconds of many requests. When apt http pipeling works, at best, it saves a few seconds, when it fails against a buggy HTTP1.1 server or behind a proxy that does not support it, the failures are arbitrary and not adiquetly explained. This change will enhance the Cloud Images by making apt request packages sequentially instead of in batches and prevent arbitrary failures caused by non-compliant web servers. Cloud image users, both on EC2 and outside, will benefit from this fix. http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/536 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/537 _______________________________________________________________________________________ apt-get is appears to be mangling the local caching file names, which is appearing as a hashsum or size mismatch. Evidence below: _______________________________________________________________________________________ Output of "apt-get -y install firefox": Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/p/python-defer/python-defer_1.0.2+bzr481-1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/a/aptdaemon/aptdaemon_0.43+bzr769-0ubuntu1_all.deb Size mismatch Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/u/ubufox/xul-ext-ubufox_2.0-0ubuntu1_all.deb Size mismatch _______________________________________________________________________________________ apt-get -y install firefox with debug for HTTP turned on: Get:63 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-defer all 1.0.2+bzr481-1 [10.9 kB] HTTP/1.1 200 OK^M x-amz-id-2: EXccd6GVMfE6ly4SYdwy313VK42d/gI3ncyxmaotuVIMbBBi6FJkuDYWrzLw7vXE^M x-amz-request-id: 7572BAB5E6FFFAC0^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 03 Feb 2012 08:54:27 GMT^M ETag: "6c07f8db615cc32657b1cc57450a1608"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 15142^M Server: AmazonS3^M ^M Get:64 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main python-aptdaemon all 0.43+bzr769-0ubuntu1 [80.3 kB] HTTP/1.1 200 OK^M x-amz-id-2: IcNECY1OtzHOXnTnTW3fqoJGyfdODq+qRXXGYTDEOJxfw0CP9UH8EUSllVR3Gf3F^M x-amz-request-id: 5DEEEE629E3BA012^M Date: Tue, 06 Mar 2012 20:06:13 GMT^M Last-Modified: Fri, 02 Mar 2012 20:55:40 GMT^M ETag: "501373631a241847cf76d13aea9264b3"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 56824^M Server: AmazonS3^M ^M Get:65 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/ precise/main aptdaemon all 0.43+bzr769-0ubuntu1 [15.1 kB] HTTP/1.1 200 OK^M x-amz-id-2: HHNSOZKH74D4I1XCaOiwDOzIWxIVJ88ibfKxvIp/4NpkczzPmsOhMqxrTj2T02qO^M x-amz-request-id: A6FD3143E6A0BA1A^M Date: Tue, 06 Mar 2012 20:06:34 GMT^M Last-Modified: Sat, 21 Jan 2012 00:48:55 GMT^M ETag: "3653165af1f20a437a14632ce0a2e6c2"^M Accept-Ranges: bytes^M Content-Type: application/x-debian-package^M Content-Length: 10902^M Server: AmazonS3^M _______________________________________________________________________________________ Evaluation of the MD5sums of downloaded failed items: root@ip-10-6-85-206:/var/cache/apt/archives/partial# md5sum * 501373631a241847cf76d13aea9264b3 aptdaemon_0.43+bzr769-0ubuntu1_all.deb 6c07f8db615cc32657b1cc57450a1608 python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb 566f7c92a9fcd0c5c40d08a977176291 python-defer_1.0.2+bzr481-1_all.deb 3653165af1f20a437a14632ce0a2e6c2 xul-ext-ubufox_2.0-0ubuntu1_all.deb _______________________________________________________________________________________ Snipets from Packages.bz2 From Meta-data: Package: aptdaemon MD5sum: 6c07f8db615cc32657b1cc57450a1608 Package: xul-ext-ubufox MD5sum: 501373631a241847cf76d13aea9264b3 Package: python-aptdaemon MD5sum: 566f7c92a9fcd0c5c40d08a977176291 Package: python-defer MD5sum: 3653165af1f20a437a14632ce0a2e6c2 _______________________________________________________________________________________ "dpkg -i" on pacakges downloaded root@ip-10-6-85-206:/var/cache/apt/archives/partial# dpkg -I aptdaemon_0.43+bzr769-0ubuntu1_all.deb new debian package, version 2.0. size 56824 bytes: control archive= 1520 bytes. 23 bytes, 1 lines conffiles 1023 bytes, 29 lines control 1804 bytes, 21 lines md5sums Package: xul-ext-ubufox Source: ubufox Version: 2.0-0ubuntu1 Architecture: all Maintainer: Ubuntu Mozilla Team <[email protected]> Installed-Size: 383 Depends: aptdaemon, libglib2.0-0 (>= 2.26) Recommends: firefox (>= 4.0~b6) Enhances: firefox Breaks: ubufox (<< 0.9~rc2-0ubuntu3)a Replaces: ubufox (<< 0.9~rc2-0ubuntu3) Provides: firefox-ubufox, ubufox Section: web Priority: optional Homepage: https://launchpad.net/ubufox Description: Ubuntu-specific configuration defaults and apt support for Firefox Adds Ubuntu-specific modifications to Firefox. . Integrates the browser with Ubuntu to: * Enable searching for missing plugins from Ubuntu software catalog * Add the following options to the Help menu - Get help on-line - Help translating Firefox - Ubuntu Release Notes * Set homepage to Ubuntu Start Page * Display a restart notification after upgrading Firefox * Add ask.com to the search engines. . You can uninstall this if you prefer to use a pristine Firefox install. root@ip-10-6-85-206:/var/cache/apt/archives/partial# dpkg -I python-aptdaemon_0.43+bzr769-0ubuntu1_all.deb new debian package, version 2.0. size 15142 bytes: control archive= 1508 bytes. 68 bytes, 2 lines conffiles 1390 bytes, 30 lines control 1088 bytes, 15 lines md5sums Package: aptdaemon Version: 0.43+bzr769-0ubuntu1 Architecture: all Maintainer: Ubuntu Developers <[email protected]> Installed-Size: 188 Depends: python, python-aptdaemon (= 0.43+bzr769-0ubuntu1), python-gi, gir1.2-glib-2.0 Breaks: software-center (<< 1.1.21) Section: admin Priority: extra Homepage: https://launchpad.net/aptdaemon Description: transaction based package management service Aptdaemon allows normal users to perform package management tasks, e.g. refreshing the cache, upgrading the system, installing or removing software packages. . Currently it comes with the following main features: . - Programming language independent D-Bus interface, which allows one to write clients in several languages - Runs only if required (D-Bus activation) - Fine grained privilege management using PolicyKit, e.g. allowing all desktop user to query for updates without entering a password - Support for media changes during installation from DVD/CDROM - Support for debconf (Debian's package configuration system) - Support for attaching a terminal to the underlying dpkg call . This package contains the aptd script and all the data files required to run the daemon. Moreover it contains the aptdcon script, which is a command line client for aptdaemon. The API is not stable yet. Original-Maintainer: Julian Andres Klode <[email protected]> ProblemType: Bug DistroRelease: Ubuntu 12.04 Package: apt 0.8.16~exp12ubuntu4 ProcVersionSignature: Ubuntu 3.2.0-18.28-virtual 3.2.9 Uname: Linux 3.2.0-18-virtual x86_64 ApportVersion: 1.94-0ubuntu1 Architecture: amd64 Date: Tue Mar 6 21:21:17 2012 Ec2AMI: ami-0400dd6d Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1c Ec2InstanceType: m1.large Ec2Kernel: aki-825ea7eb Ec2Ramdisk: unavailable SourcePackage: apt UpgradeStatus: No upgrade log present (probably fresh install)
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/948461 Title: apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
