[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Antonio Rosales: Work items changed: Work items: [racb] debootstrap production code: POSTPONED [racb] apt production code: POSTPONED - [utlemming] merge s3aptmirror production code: TODO + [utlemming] merge s3aptmirror production code: POSTPONED apt-ftparchive by-hash generator command: POSTPONED apt-ftparchive modifications to automatically generate by-hash: POSTPONED ubumirror patch: POSTPONED debmirror patch: POSTPONED Fix squid-deb-proxy cache settings as appropriate: POSTPONED - [smoser] Pin RR DNS in squid-deb-proxy: TODO + [smoser] Pin RR DNS in squid-deb-proxy: POSTPONED Testing: POSTPONED Implement InRelease in archive.ubuntu.com mirrors: POSTPONED Add by-hash generation to archive.ubuntu.com: POSTPONED Add by-hash generation to Launchpad PPAs: POSTPONED [racb] apt proof of concept: DONE [racb] debootstrap proof of concept: DONE [racb] s3aptmirror patch: DONE [racb] Live S3 mirror: DONE -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Antonio Rosales: Whiteboard changed: Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements Proposed spec: https://wiki.ubuntu.com/AptByHash Discussion: https://lists.ubuntu.com/archives/ubuntu-devel/2012-July/035487.html = User Stories = * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. = Assumptions = None = Test Plans = Test will be conducted in 2 ways: a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] [1] https://code.launchpad.net/~smoser/+junk/check-archive/ = Release Note = Ubuntu's public mirrors now contain an improved archive format. Coupled with updates to apt transient apt-update errors are a thing of the past. Note: feature freeze is upon us, and I think this work is too invasive for an FFe. My priorities mean that I can't work on this for the next few weeks, so it looks like this will have to slip to R. --racb + + [21-Sep-2012] Postponing remaining work items until R cycle per Robie's + bove note. -a.rosales -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robie Basak: Work items changed: Work items: - [racb] debootstrap production code: TODO - [racb] apt production code: TODO - [racb] merge s3aptmirror production code: TODO - apt-ftparchive by-hash generator command: TODO - apt-ftparchive modifications to automatically generate by-hash: TODO - ubumirror patch: TODO - debmirror patch: TODO - Fix squid-deb-proxy cache settings as appropriate: TODO + [racb] debootstrap production code: POSTPONED + [racb] apt production code: POSTPONED + [utlemming] merge s3aptmirror production code: TODO + apt-ftparchive by-hash generator command: POSTPONED + apt-ftparchive modifications to automatically generate by-hash: POSTPONED + ubumirror patch: POSTPONED + debmirror patch: POSTPONED + Fix squid-deb-proxy cache settings as appropriate: POSTPONED [smoser] Pin RR DNS in squid-deb-proxy: TODO - Testing: TODO - Implement InRelease in archive.ubuntu.com mirrors: TODO - Add by-hash generation to archive.ubuntu.com: TODO - Add by-hash generation to Launchpad PPAs: TODO + Testing: POSTPONED + Implement InRelease in archive.ubuntu.com mirrors: POSTPONED + Add by-hash generation to archive.ubuntu.com: POSTPONED + Add by-hash generation to Launchpad PPAs: POSTPONED [racb] apt proof of concept: DONE [racb] debootstrap proof of concept: DONE [racb] s3aptmirror patch: DONE [racb] Live S3 mirror: DONE -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robie Basak: Whiteboard changed: Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements Proposed spec: https://wiki.ubuntu.com/AptByHash Discussion: https://lists.ubuntu.com/archives/ubuntu-devel/2012-July/035487.html = User Stories = * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. = Assumptions = None = Test Plans = Test will be conducted in 2 ways: a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] [1] https://code.launchpad.net/~smoser/+junk/check-archive/ = Release Note = Ubuntu's public mirrors now contain an improved archive format. Coupled with updates to apt transient apt-update errors are a thing of the past. + + Note: feature freeze is upon us, and I think this work is too invasive + for an FFe. My priorities mean that I can't work on this for the next + few weeks, so it looks like this will have to slip to R. -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robie Basak: Whiteboard changed: Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements Proposed spec: https://wiki.ubuntu.com/AptByHash Discussion: https://lists.ubuntu.com/archives/ubuntu-devel/2012-July/035487.html = User Stories = * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. = Assumptions = None = Test Plans = Test will be conducted in 2 ways: a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] [1] https://code.launchpad.net/~smoser/+junk/check-archive/ = Release Note = Ubuntu's public mirrors now contain an improved archive format. Coupled with updates to apt transient apt-update errors are a thing of the past. Note: feature freeze is upon us, and I think this work is too invasive for an FFe. My priorities mean that I can't work on this for the next - few weeks, so it looks like this will have to slip to R. + few weeks, so it looks like this will have to slip to R. --racb -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robie Basak: Work items changed: Work items: - support InRelease file in archive.ubuntu.com mirrors: TODO - follow debian thread for upstream apt solution: TODO - apt client changes: TODO - debootstrap changes: TODO - archive.ubuntu.com change to new repository format: TODO - update squid-deb-proxy to remove special case handling of Packages: TODO + [racb] debootstrap production code: TODO + [racb] apt production code: TODO + [racb] merge s3aptmirror production code: TODO + apt-ftparchive by-hash generator command: TODO + apt-ftparchive modifications to automatically generate by-hash: TODO + ubumirror patch: TODO + debmirror patch: TODO + Fix squid-deb-proxy cache settings as appropriate: TODO + [smoser] Pin RR DNS in squid-deb-proxy: TODO + Testing: TODO + Implement InRelease in archive.ubuntu.com mirrors: TODO + Add by-hash generation to archive.ubuntu.com: TODO + Add by-hash generation to Launchpad PPAs: TODO + [racb] apt proof of concept: DONE + [racb] debootstrap proof of concept: DONE + [racb] s3aptmirror patch: DONE + [racb] Live S3 mirror: DONE -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robie Basak: Whiteboard changed: Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements + Proposed spec: https://wiki.ubuntu.com/AptByHash + Discussion: https://lists.ubuntu.com/archives/ubuntu-devel/2012-July/035487.html - = User Stories = - * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. - * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. - * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. + = User Stories = + * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. + * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. + * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. = Assumptions = None = Test Plans = Test will be conducted in 2 ways: - a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. - b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] -[1] https://code.launchpad.net/~smoser/+junk/check-archive/ + a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. + b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] + [1] https://code.launchpad.net/~smoser/+junk/check-archive/ = Release Note = Ubuntu's public mirrors now contain an improved archive format. Coupled with updates to apt transient apt-update errors are a thing of the past. -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robbie Williamson: Priority: Undefined = Medium -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Dave Walker: Definition Status: Review = Approved -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Scott Moser: Definition Status: Discussion = Review -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Scott Moser: Whiteboard changed: - === Please Use Etherpad for Editing === - Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements - Are there any work items for this? -- Daviey + = User Stories = + * Jim is using JuJu, and adds a node. He does not want to see apt fail due to a race condition. + * Marley is using the Ubuntu installer for network based installs against a local mirror that is updated from a remote mirror. He does not want installs to fail due to the mirror being updated. + * Sally wants to use a http proxy to cache deb downloads and does not want to be vulnerable to caching mismatched files that would cause apt failures. + + = Assumptions = + None + + = Test Plans = + Test will be conducted in 2 ways: + a.) implied test: much of our automated infrastructure relies on apt and previously failed quite commonly due to apt. These errors should be erased. + b.) explicit test: we should ideally have some canary that would ensure our public mirrors do not fail. One example starting point is [1] +[1] https://code.launchpad.net/~smoser/+junk/check-archive/ + + = Release Note = + Ubuntu's public mirrors now contain an improved archive format. Coupled with updates to apt transient apt-update errors are a thing of the past. -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Scott Moser: Work items set to: Work items: support InRelease file in archive.ubuntu.com mirrors: TODO follow debian thread for upstream apt solution: TODO apt client changes: TODO debootstrap changes: TODO archive.ubuntu.com change to new repository format: TODO update squid-deb-proxy to remove special case handling of Packages: TODO -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Dave Walker: Whiteboard changed: === Please Use Etherpad for Editing === Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements + + Are there any work items for this? -- Daviey -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Clint Byrum: Whiteboard changed: + Problem: Archive during development is often inconsistent due to Hash + sum mismatch. This is because Packages.gz is updated in a non-atomic + manner with the 'Release' file. + + Current mitigation(s): + * Expires header set to exact same time so caches will not cache the archive in a broken state. This reduces the window for problems to just the time between writing Packages.gz and Release. + + Options to discuss: + * pdiff for smaller windows of inconsitency (still updates in a non-atomic way unfortunately) + * Versioned Packages.gz (Packages.1.gz, Packages.2.gz, or maybe Packages.201205021431.gz) + * A/B style atomic whole-directory archive refreshes. (A is live, write new refresh to B, make B live with re-writing a symlink, then repeat with A/B swapped). [clint- I don't think this is feasible w/o CoW filesystem snapshots on the archive servers] + Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Scott Moser: Whiteboard changed: - Problem: Archive during development is often inconsistent due to Hash - sum mismatch. This is because Packages.gz is updated in a non-atomic - manner with the 'Release' file. - - Current mitigation(s): - * Expires header set to exact same time so caches will not cache the archive in a broken state. This reduces the window for problems to just the time between writing Packages.gz and Release. - - Options to discuss: - * pdiff for smaller windows of inconsitency (still updates in a non-atomic way unfortunately) - * Versioned Packages.gz (Packages.1.gz, Packages.2.gz, or maybe Packages.201205021431.gz) - * A/B style atomic whole-directory archive refreshes. (A is live, write new refresh to B, make B live with re-writing a symlink, then repeat with A/B swapped). [clint- I don't think this is feasible w/o CoW filesystem snapshots on the archive servers] + === Please Use Etherpad for Editing === Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Scott Moser: Whiteboard changed: - - http://pad.ubuntu.com/uds-q-servercloud-apt-improvements + Etherpad: http://pad.ubuntu.com/uds-q-servercloud-q-apt-improvements + Blueprint: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robbie Williamson: Drafter: Scott Moser = Ubuntu Server Team -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robbie Williamson: Definition Status: New = Discussion -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-q-apt-improvements] Apt Improvements
Blueprint changed by Robbie Williamson: Assignee: (none) = Scott Moser -- Apt Improvements https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-apt-improvements -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
