Re: deb822 sources by default for bookworm
Julian Andres Klode wrote: > docker images probably can move earlier as they don't have > as much interactive users that use tools that might be broken > (e.g. software-properties). I didn't see any open disagreement in this thread (apologies if I missed it elsewhere), and I think this makes sense, so I've opened [1]. [1]: https://github.com/debuerreotype/debuerreotype/pull/128 There's still an open discussion point on whether it should be "debian.sources" or something like "00debian.sources" and I think the timeline is still a little fluid, especially depending on whether / how many folks complain and how heavy the breakage is (ie, "my sed one-liner needs to change" vs "this really-useful-in-containers package fundamentally can't handle this correctly"). I guess I'm also wondering (in the spirit of #877012 - hopefully dkg will forgive me for the explicit CC) whether "Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg" is sufficient or whether folks think that it needs to somehow be more specific? (please keep me CC'd in replies or comment on that PR directly) ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4
Re: deb822 sources by default for bookworm
On Fri, Nov 05, 2021 at 06:16:10PM +0100, Julian Andres Klode wrote: > On Thu, Nov 04, 2021 at 12:13:48AM +0800, Shengjing Zhu wrote: > > On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode wrote: > > > > > > Hi all, > > > > > > I'd like us to move from > > > > > > /etc/apt/sources.list > > > > > > to > > > /etc/apt/sources.list.d/debian.sources > > > > > > > While it's really a nice feature for the third-party repository, I > > don't see the benefits to change the default one, especially the path. > > I had to admit that I have countless scripts which run `sed > > /etc/apt/souces.list`, to change the default mirror, as well as in the > > Dockerfile. (fwiw you could use a 'mirror+file' entry (see man apt-transport-mirror) specifying your preferred mirror of the day. As a bonus that will reuse the files of the old mirror instead of discarding them blindly) > There's a technical limitation in that we get the format from the file > extension. It's a bit annoying. It was shortly the case that .list files could contain both formats, but I opted to change that as of course countless things poking into those files broke left and right. cups used to detect if it ran on a Debian-like system by checking the sources.list file for deb entries… I doubt it does nowadays as there are countless better options now, I just mention it as trivial example of the type of unexpected breakage which is to be expected… Best regards David Kalnischkies signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
On Thu, Nov 04, 2021 at 12:13:48AM +0800, Shengjing Zhu wrote: > On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode wrote: > > > > Hi all, > > > > I'd like us to move from > > > > /etc/apt/sources.list > > > > to > > /etc/apt/sources.list.d/debian.sources > > > > While it's really a nice feature for the third-party repository, I > don't see the benefits to change the default one, especially the path. > I had to admit that I have countless scripts which run `sed > /etc/apt/souces.list`, to change the default mirror, as well as in the > Dockerfile. There's a technical limitation in that we get the format from the file extension. It's a bit annoying. Also this makes multi-origin downstream distros cleaner, so you can have /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/downstream.sources instead of worrying about either creating a combined sources.list or dropping just your downstream bits into sources.list.d We'll have to see how much trouble that causes, and it might work to make sources.list a symlink to sources.list.d/debian.sources, and have apt inspect readlink() to get the "real" file extension for sources.list -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Re: deb822 sources by default for bookworm
On Fri, Nov 05, 2021 at 05:31:49PM +0100, David Kalnischkies wrote:
> On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote:
> > On 03-11-2021 16:45, Julian Andres Klode wrote:
> > > There is some software "parsing" sources.list on its own, most of that
> > > is better served by `apt-get indextargets` (and for downloading stuff
> > > based on the urls, `apt-helper download-file`, such that it respects
> > > proxies and supports all transports users may use in sources.list)
> >
> > Like autopkgtest. When I was working on it to support Debian's migration
> > testing, I looked at python-apt and because that didn't support it,
> > stopped thinking. With indextargets and download-file I guess we could
> > work on it again. When were those introduced? Ubuntu needs it on old
> > releases so before autopkgtest can change it, we'd need support for a while.
>
> `apt-get indextargets` is from 2015 and a part of the acquire-additional-
> files feature used mainly by apt-file and appstream to have apt download
> files it isn't using itself, so those tools don't have to implement it.
>
> The job of indextargets is it mostly to give access to metadata (and
> crucially filenames) for those previously configured and hopefully now
> downloaded files. apt-file e.g. asks for the Contents files in this way
> to avoid exposing file naming logic and location to other tools.
>
> So, for the filenames of all (downloaded) Packages files:
> apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages'
> (the default output is deb822 stanzas you could grep with more powerful
> tools than the simple inbuilt line-based filter)
>
> Note that you either have to implement opening compressed files yourself
> or use `/usr/lib/apt/apt-helper cat-file`.
>
>
> That was historically the most common reason to fiddle with sources.list
> parsing, hence Julian referring to it, but this seems not what autpkgtest
> is aiming for. On a casual look (well, grep) I see only:
>
> * lib/adt_testbed.py apt-pocket codepath seems to want to construct new
> sources.list entries based on existing ones. That should be possible
> with some indextargets busywork in general, but I am not completely
> sure what is going on here and the gymnastics should be similar to…
>
> * setup-commands/setup-testbed tries to find the mirror (and release)
> used for target distribution based on your current system. I am
> a bit surprised that works actually…
>
> Anyway, the later could perhaps be implement with:
> apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \
>grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2
>
> (That is one line only for posterity – as you see, I am trying to fix
> the too general search by checking against Origin as defined by the
> Release file of a repository, but that would need work still to
> eliminate same-origin-but-different-repo cases)
>
>
> Parsing of the sources files is not really indextargets job through, so
> it might not always work for that task: It e.g. doesn't work if the data
> files are not on disk which might or might not be okay for you (there is
> 'guess' mode, but that of course has no metadata extracted from the
> Release files – the Origin I was using above).
>
> The apt family doesn't really have a publicly exposed way of reasoning
> about sources.list (or .sources) files and I am not quite sure it really
> can as subtil differences between repositories make it hard to give them
> all a common interface which makes sense. (I will probably be proven
> wrong by Julian though.)
>
>
> Like, for example, if stable is in the sources, make sure there
> is also updates and security there and/or add them. What for Debian are
> three distinct repositories might for others very well be components.
>
> Assuming you even know which line refers to Debian: I was using Origin
> above for this task as we can't really guess based on the URI. And even
> then… that logic above finds the tor+mirror+file source I am using,
> that won't work for autopkgtest, but I am special and this just
> a default fallback, so I might be thinking way too much about it…
We could render what we passed as a deb822 format in apt{,-get}
sources list (remind me to add subcommand support to the parser).
Possibly with some normalization in there. Presumably if you have
a paragraph "Suites: foo bar Components: x y" you end up with 2
paragraphs in the output, as IIRC we group our sources entries
by Release file.
I think that provides a reasonable top-level interface. You could
of course then also normalize this further such that all fields only
have one value, and give them all IDs and then you can apt source
remove or stuff, idk.
Will have to see how I implement the deb822 parsing and editing
on the python-apt side.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
signature.asc
Description: PGP signature
Re: deb822 sources by default for bookworm
On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote: > On 03-11-2021 16:45, Julian Andres Klode wrote: > > There is some software "parsing" sources.list on its own, most of that > > is better served by `apt-get indextargets` (and for downloading stuff > > based on the urls, `apt-helper download-file`, such that it respects > > proxies and supports all transports users may use in sources.list) > > Like autopkgtest. When I was working on it to support Debian's migration > testing, I looked at python-apt and because that didn't support it, > stopped thinking. With indextargets and download-file I guess we could > work on it again. When were those introduced? Ubuntu needs it on old > releases so before autopkgtest can change it, we'd need support for a while. `apt-get indextargets` is from 2015 and a part of the acquire-additional- files feature used mainly by apt-file and appstream to have apt download files it isn't using itself, so those tools don't have to implement it. The job of indextargets is it mostly to give access to metadata (and crucially filenames) for those previously configured and hopefully now downloaded files. apt-file e.g. asks for the Contents files in this way to avoid exposing file naming logic and location to other tools. So, for the filenames of all (downloaded) Packages files: apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages' (the default output is deb822 stanzas you could grep with more powerful tools than the simple inbuilt line-based filter) Note that you either have to implement opening compressed files yourself or use `/usr/lib/apt/apt-helper cat-file`. That was historically the most common reason to fiddle with sources.list parsing, hence Julian referring to it, but this seems not what autpkgtest is aiming for. On a casual look (well, grep) I see only: * lib/adt_testbed.py apt-pocket codepath seems to want to construct new sources.list entries based on existing ones. That should be possible with some indextargets busywork in general, but I am not completely sure what is going on here and the gymnastics should be similar to… * setup-commands/setup-testbed tries to find the mirror (and release) used for target distribution based on your current system. I am a bit surprised that works actually… Anyway, the later could perhaps be implement with: apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \ grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2 (That is one line only for posterity – as you see, I am trying to fix the too general search by checking against Origin as defined by the Release file of a repository, but that would need work still to eliminate same-origin-but-different-repo cases) Parsing of the sources files is not really indextargets job through, so it might not always work for that task: It e.g. doesn't work if the data files are not on disk which might or might not be okay for you (there is 'guess' mode, but that of course has no metadata extracted from the Release files – the Origin I was using above). The apt family doesn't really have a publicly exposed way of reasoning about sources.list (or .sources) files and I am not quite sure it really can as subtil differences between repositories make it hard to give them all a common interface which makes sense. (I will probably be proven wrong by Julian though.) Like, for example, if stable is in the sources, make sure there is also updates and security there and/or add them. What for Debian are three distinct repositories might for others very well be components. Assuming you even know which line refers to Debian: I was using Origin above for this task as we can't really guess based on the URI. And even then… that logic above finds the tor+mirror+file source I am using, that won't work for autopkgtest, but I am special and this just a default fallback, so I might be thinking way too much about it… Anyway, if you have specific needs/questions feel free to ask on deity@ or #debian-apt. I am sure we will work something out even if in this case it might very well be new code nobody really uses for years (as is common in apt land – backward compat be damned ). Best regards David Kalnischkies signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote: > On Wed, Nov 03, 2021 at 04:23:52PM +, Holger Levsen wrote: > > You didn't say so explicitly, but do you plan to support old style > > /etc/apt/sources.list until forever? ;) Or do you envision automatic > > migration of that file? Or? > > I don't know, to be honest, have not thought about it yet. > > With APT's 5 year interface stability and major version bump, > The first time we could remove support for old releases would be > trixie/apt 3.0 in 2025 (that schedule just happens because I don't > want to release .10 versions, but keep the 2nd component single > digit :D) > > I think an automatic migration might be to painful what with all the > juju and ansible and saltstack (I feel like it'd be nice to have > those tools migrate config to new formats). > > Of course, once everyone and their dog has migrated, I might feel > different and complain about legacy sources.list and deprecate > them. I think it would be a mistake to ever drop support for the older format. Instructions for using it are scattered all over the place, and there'll be an enormous long tail of things to update, not to mention things like old forum posts that can never really be updated. I agree we might feel different in ten years' time! -- Colin Watson (he/him) [cjwat...@debian.org]
Re: deb822 sources by default for bookworm
On 11/3/21 5:59 PM, Holger Levsen wrote: > On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote: >> I don't know, to be honest, have not thought about it yet. > > many thanks for your verbose reply! /me likes this timeline. > >> I think an automatic migration might be to painful what with all the >> juju and ansible and saltstack (I feel like it'd be nice to have >> those tools migrate config to new formats). > > I guess it would be nice if those tools (and users not using those > tools) could run one standard tool doing the job :) This feels like a very nicely thought plans, which I 100% support. I especially like the timeline for deprecating and removal, which gives enough time for everyone to support the new format. So I'm joining to say thank you to Julian. Cheers, Thomas Goirand (zigo)
Re: deb822 sources by default for bookworm
On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote: > Hi Julian, > > On 03-11-2021 16:45, Julian Andres Klode wrote: > > There is some software "parsing" sources.list on its own, most of that > > is better served by `apt-get indextargets` (and for downloading stuff > > based on the urls, `apt-helper download-file`, such that it respects > > proxies and supports all transports users may use in sources.list) > > Like autopkgtest. When I was working on it to support Debian's migration > testing, I looked at python-apt and because that didn't support it, > stopped thinking. With indextargets and download-file I guess we could > work on it again. When were those introduced? Ubuntu needs it on old > releases so before autopkgtest can change it, we'd need support for a while. download-file gained hash checking support in 0.9.16, did not find when it was introduced. indextargets has been around since 1.1. (I'm not subscribed, please Cc me, I don't check archives/nntp often) -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
Hi Julian, On 03-11-2021 16:45, Julian Andres Klode wrote: > There is some software "parsing" sources.list on its own, most of that > is better served by `apt-get indextargets` (and for downloading stuff > based on the urls, `apt-helper download-file`, such that it respects > proxies and supports all transports users may use in sources.list) Like autopkgtest. When I was working on it to support Debian's migration testing, I looked at python-apt and because that didn't support it, stopped thinking. With indextargets and download-file I guess we could work on it again. When were those introduced? Ubuntu needs it on old releases so before autopkgtest can change it, we'd need support for a while. Paul OpenPGP_signature Description: OpenPGP digital signature
Re: deb822 sources by default for bookworm
On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote: > I don't know, to be honest, have not thought about it yet. many thanks for your verbose reply! /me likes this timeline. > I think an automatic migration might be to painful what with all the > juju and ansible and saltstack (I feel like it'd be nice to have > those tools migrate config to new formats). I guess it would be nice if those tools (and users not using those tools) could run one standard tool doing the job :) -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ This too shall pass. signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
On Wed, Nov 03, 2021 at 04:23:52PM +, Holger Levsen wrote: > Hi Julian, > > this sounds like a nice and useful plan and feature(s), thank you! > just one question: > > On Wed, Nov 03, 2021 at 04:45:15PM +0100, Julian Andres Klode wrote: > > I'd like us to move from > > /etc/apt/sources.list > > to > > /etc/apt/sources.list.d/debian.sources > [...] > > #timeline > > You didn't say so explicitly, but do you plan to support old style > /etc/apt/sources.list until forever? ;) Or do you envision automatic > migration of that file? Or? I don't know, to be honest, have not thought about it yet. With APT's 5 year interface stability and major version bump, The first time we could remove support for old releases would be trixie/apt 3.0 in 2025 (that schedule just happens because I don't want to release .10 versions, but keep the 2nd component single digit :D) I think an automatic migration might be to painful what with all the juju and ansible and saltstack (I feel like it'd be nice to have those tools migrate config to new formats). Of course, once everyone and their dog has migrated, I might feel different and complain about legacy sources.list and deprecate them. So presumably, we'd have both as supported in bookworm, sources.list deprecated for trixie, and then removed in 2030 in apt 4.0, but that's still 8 years to go. (all dates are 1 year after the change lands in unstable, as the changes land in odd numbers earlier in the cycle) -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
Hi Julian, this sounds like a nice and useful plan and feature(s), thank you! just one question: On Wed, Nov 03, 2021 at 04:45:15PM +0100, Julian Andres Klode wrote: > I'd like us to move from > /etc/apt/sources.list > to > /etc/apt/sources.list.d/debian.sources [...] > #timeline You didn't say so explicitly, but do you plan to support old style /etc/apt/sources.list until forever? ;) Or do you envision automatic migration of that file? Or? -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Only change is constant. signature.asc Description: PGP signature
Re: deb822 sources by default for bookworm
On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode wrote: > > Hi all, > > I'd like us to move from > > /etc/apt/sources.list > > to > /etc/apt/sources.list.d/debian.sources > While it's really a nice feature for the third-party repository, I don't see the benefits to change the default one, especially the path. I had to admit that I have countless scripts which run `sed /etc/apt/souces.list`, to change the default mirror, as well as in the Dockerfile. > in bookworm. > > # deb822 intro > > The deb822 format can be shorter and easier to read, to quote the > sources.list manual page: > >As an example, the sources for your distribution could look like this > in one-line-style format: > >deb http://deb.debian.org/debian bullseye main contrib non-free >deb http://security.debian.org bullseye-security main contrib > non-free > >or like this in deb822 style format: > >Types: deb >URIs: http://deb.debian.org/debian >Suites: bullseye >Components: main contrib non-free > >Types: deb >URIs: http://security.debian.org >Suites: bullseye-security >Components: main contrib non-free > > > Now if you mix unstable and testing, you could just have that it one > paragraph. > > The big advantage of deb822 sources is that you can embed larger data: > >Types: deb >URIs: https://deb.debian.org >Suites: stable >Components: main contrib non-free >Signed-By: > -BEGIN PGP PUBLIC KEY BLOCK- > . > > mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY > > CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk > > IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS > > dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG > 3bHcln8DMpIJVXht78sL > =IE0r > -END PGP PUBLIC KEY BLOCK- > > Oh yeah, a standalone sources file with embedded key - making third-party > repository management substantially more convenient. > > # issues > > several software does not support deb822 currently. I plan to work on > adding deb822 support to python-apt in the upcoming months, I don't know > what else is affected. > > There is some software "parsing" sources.list on its own, most of that > is better served by `apt-get indextargets` (and for downloading stuff > based on the urls, `apt-helper download-file`, such that it respects > proxies and supports all transports users may use in sources.list) > > In terms of setting up system, I guess debootstrap and d-i's apt-setup > module need changes? I admit I do not have a total overview. > > # timeline > > I do not know the total outcome of this. My hope would be that > we can switch the default in Summer 2022 and see what breaks, > although I don't know who's going to install from testing > d-i :D > > docker images probably can move earlier as they don't have > as much interactive users that use tools that might be broken > (e.g. software-properties). Possibly others, there's no need > to roll out in multiple places at the same time, as long as > we converge by freeze time. > > I invite people to test this out already on their own systems, > I know several people do; and extrepo also builds deb822 sources > files, but several people I guess do not know about it yet. Please > test and report bugs. > -- > debian developer - deb.li/jak | jak-linux.org - free software dev > ubuntu core developer i speak de, en > -- Shengjing Zhu
deb822 sources by default for bookworm
Hi all, I'd like us to move from /etc/apt/sources.list to /etc/apt/sources.list.d/debian.sources in bookworm. # deb822 intro The deb822 format can be shorter and easier to read, to quote the sources.list manual page: As an example, the sources for your distribution could look like this in one-line-style format: deb http://deb.debian.org/debian bullseye main contrib non-free deb http://security.debian.org bullseye-security main contrib non-free or like this in deb822 style format: Types: deb URIs: http://deb.debian.org/debian Suites: bullseye Components: main contrib non-free Types: deb URIs: http://security.debian.org Suites: bullseye-security Components: main contrib non-free Now if you mix unstable and testing, you could just have that it one paragraph. The big advantage of deb822 sources is that you can embed larger data: Types: deb URIs: https://deb.debian.org Suites: stable Components: main contrib non-free Signed-By: -BEGIN PGP PUBLIC KEY BLOCK- . mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG 3bHcln8DMpIJVXht78sL =IE0r -END PGP PUBLIC KEY BLOCK- Oh yeah, a standalone sources file with embedded key - making third-party repository management substantially more convenient. # issues several software does not support deb822 currently. I plan to work on adding deb822 support to python-apt in the upcoming months, I don't know what else is affected. There is some software "parsing" sources.list on its own, most of that is better served by `apt-get indextargets` (and for downloading stuff based on the urls, `apt-helper download-file`, such that it respects proxies and supports all transports users may use in sources.list) In terms of setting up system, I guess debootstrap and d-i's apt-setup module need changes? I admit I do not have a total overview. # timeline I do not know the total outcome of this. My hope would be that we can switch the default in Summer 2022 and see what breaks, although I don't know who's going to install from testing d-i :D docker images probably can move earlier as they don't have as much interactive users that use tools that might be broken (e.g. software-properties). Possibly others, there's no need to roll out in multiple places at the same time, as long as we converge by freeze time. I invite people to test this out already on their own systems, I know several people do; and extrepo also builds deb822 sources files, but several people I guess do not know about it yet. Please test and report bugs. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
