Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Sat, May 19, 2012 at 2:00 PM, Julian Andres Klode j...@debian.org wrote:
= Flat Repository Format =
A flat repository does not use the {{{dists}}} hierarchy of directories,
and instead places meta index and indices directly into the archive root
(or some part below it) In sources.list syntax, a flat repository is specified
like this:
{{{
deb uri directory/
}}}
I don't think defining sources.list syntax in a client-agnostic document
is a good move. APT has the 'sources.list' manpage for it and other clients
might or might not have different ways to specify repositories.
(beside, that it would be deb-src, too)
!InRelease, Release, Release.gpg meta-information are supported as well.
Diffs,
Translations, and Contents indices are not defined for that repository format.
Indices may be compressed just like in the standard Debian repository format.
Translations are supported, although with a different name: directory/en
(and co) instead of Translation-en. For Contents i am not sure, but i think
apt-file downloads these, too. (not sure if this should be a reason to include
it in a specification through or just keep it as some legacy cruft around)
Diffs are supported by apt, but it will not be used if not in Release.
(if no Release file is present, diffs will not be tried).
It's the same for the non-flat repository and true for other files as well
- and should be a reasonable thing to allow clients to do.
In that train of thought, I think it would be a good idea to require a
repository to have a Release (or InRelease) file including all files
[in their current state] composing this repository.
They are easy to create and this way a client could stop guessing if
they like to, avoiding possibly a lot of 404's.
Best combined with a strong recommendation on signing them.
Best regards
David Kalnischkies
P.S.: Could we please stop talking to three bugs and two mailinglists?
Especially as [0] suggests it is the wrong list…
[0] http://lists.debian.org/debian-devel/2012/05/msg00222.html
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/caaz6_fbybj8mqxssmdjm3naxq9v62usw3w-3juea-8eng...@mail.gmail.com
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Charles Plessy ple...@debian.org writes: How about integrating it with the Policy's chapter 5 (thus enlarging its scope) instead of having it as a separate document ? That would help to underline when a field is used in the same way or differently as in the package control data files. The primary reason why I'm hesitant to do this is that I don't think the normal Policy change process is useful for this document. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zk941z9w@windlord.stanford.edu
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Julian Andres Klode j...@debian.org writes:
On Fri, May 18, 2012 at 04:06:23PM +0200, Michal Suchanek wrote:
FWIW
posted on the wiki: http://wiki.debian.org/RepositoryFormat
Thanks
Michal
I have now documented the Contents indices and the diffs
as well, mostly (sans the exact format we use for the
patches), and Translation indices. Now we're basically
only missing details, it is fairly complete otherwise
(i.e. we should have mentioned every file and field
currently in use, but may not have explained all of
them completely).
We now have documented
dists/$DIST/Release (and InRelease, Release.gpg)
dists/$DIST/$COMP/binary-$ARCH/Packages
dists/$DIST/$COMP/source/Sources
dists/$DIST/$COMP/Contents-$ARCH.gz
dists/$DIST/$COMP/i18n/{Index,Translation-*.bz2}
*.diff/Index *.diff/%Y-%m-%d-%H%M.%S.gz
The other Release files have been omitted, as they are not
used anywhere. We are only missing udeb content files and
packages files now, which are just small subsentences.
In a few months, I'd like to rework this in DocBook form,
and submit it to debian-policy for inclusion into official
Policy, as a sub-policy like copyright-format.
This describes repositories of the form
deb uri suite component [...]
There should be a mention of flat repositories of the form
deb url path/
This changes nothing for the contents of files but it does change their
location and I think it's worth mentioning how that sources.list entry
maps to a repository.
MfG
Goswin
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqa0k5dn.fsf@frosties.localnet
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
* Paul Wise p...@debian.org [120519 01:39]: I would like to see the flat-style repository documented too, since some of the derivatives in the Debian derivatives census use it and I would like to lint their apt repositories. I my humble opinion the best documentation for the flat-style format is: don't use it, it's an ugly hack. Bernhard R. Link -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120519090516.ga5...@client.brlink.eu
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Sat, May 19, 2012 at 07:38:59AM +0800, Paul Wise wrote:
On Sat, May 19, 2012 at 12:58 AM, Julian Andres Klode wrote:
What's the opinion about the flat repository format, where you
just have one directory with Release, Packages, Sources, and
friends and no sub-directories?
Should they be documented as well then? We would then have two
kind of documented repository formats:
1. Debian-style, with a pool (or similar) and a dists directory
2. Flat-style, with just one directory
This should cover everything we currently support. Although I don't
know much about how much stuff we support in flat directories WRT
Translation, Contents, and diffs.
I would like to see the flat-style repository documented too, since
some of the derivatives in the Debian derivatives census use it and I
would like to lint their apt repositories.
I added (and others edited formatting a bit)
= Flat Repository Format =
A flat repository does not use the {{{dists}}} hierarchy of directories,
and instead places meta index and indices directly into the archive root
(or some part below it) In sources.list syntax, a flat repository is specified
like this:
{{{
deb uri directory/
}}}
Where {{{uri}}} specifies the archive root, and {{{directory}}} specifies the
position of the meta index and the indices relative to the archive root. In
Flat repositories, the following indices are supported:
* Packages (under the location {{{directory/Packages}}})
* Sources (under the location {{{directory/Sources}}})
!InRelease, Release, Release.gpg meta-information are supported as well. Diffs,
Translations, and Contents indices are not defined for that repository format.
Indices may be compressed just like in the standard Debian repository format.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120519135807.ga6...@debian.org
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Excerpts from David Kalnischkies's message of Thu May 17 18:21:59 +0200 2012: On Thu, May 17, 2012 at 3:16 PM, Michal Suchanek michal.sucha...@ruk.cuni.cz wrote: Excerpts from Ian Jackson's message of Thu May 17 14:53:30 +0200 2012: Michal Suchanek writes (Re: Bug#671503: general: APT repository format is not documented): Excerpts from Filipus Klutiero's message of Wed May 16 18:44:21 +0200 2012: Could you clarify how this differs from #481129? It's 4 years later. Sorry, forgot that I filed the bug already. It's quite some time. Given there is no feedback in 4 years I guess it is futile reporting this. Well, it's useful to bring it up again. Admittedly there is no text in social contract about using Debian-proprietary formats. And a format only defined by apt can read that is definitely Debian-proprietary there is no better term for that. Everyone agrees that it would be better if this were documented. (I have struggled on occasion myself due to the lack of documentation.) But I think the use of the word proprietary is going too far. It's certainly a special Debian format, but that wouldn't be changed if it were documented. But it's not secret and we publish at least two writer implementations and one reader implementation AFAIK, with proper Free licences. However, it's easier to reverse-engineer an existing repository than the source code so for all practical purposes it's the same as if it were closed source. That is non-sense. You said yourself that the repository is not sufficient to understand it, yet you say that it is easier to understand with it than with looking at the source (and the various bits and pieces where parts are documented). No, understanding the repository (or current apt source) is not sufficient to ensure that your repository will be readable by future apt or that future repositories will not become unreadable by your apt without any warning or explanation. Both has happened. But I don't know why we are still talking here. Russ already said he would like to have it as a subpolicy in the debian-policy. ftpmasters already said they would accept maintaining it. Everything left is writing this goddamn piece of documentation. So, maybe you should just write it… If you want to be extra fancy, start a wikipage in the debian wiki, but start typing. Go to debian-dak@l.d.o and discuss your work there. That would be awesome. Maybe he just forgot to CC this bug as well? Would be way more productive than talking about that this document is missing… Everbody knows that. Everybody doesn't like it. Now go and fix that. That everyone would like to have such a document but nobody has it so far is a strong indication that the current people are busy with other stuff. An opportunity to get involved, I would say. As said earlier, just writing a random document does not make apt not diverging from it. I'd say it's slightly discriminatory against software not part of Debian that cannot rely on getting notified when apt can read that silently changes, there is no document defining what apt should be able to read that software authors can rely on to interoperate with apt, one of the core Debian tools. Apt in turn relies on open standards like HTTP and FTP to interoperate with the rest of the world. I think this is not an appropriate use of the social contract or its concepts. Rather than complaining that this documentation doesn't exist, how about writing the document yourself ? It's not a trivial job but it should be feasible by looking at the apt source code. For me it is not feasible at all. I can, of course, describe what current repositories look like or what the current apt code accepts. However, that has silently changed in the past and is considered apt feature, not a bug. It hasn't silently changed. It was and is still the same. Your script was just horribly wrong and older APT versions just happened to work with that brokenness a little better. What you created with that script was NEVER intended to work, it just happened to be working out of complete luck (A Release file is supposed to include current data, not non-existent data, this conclusion is reachable even without too much guessing. Beside that this is actually documented in apt-secure and co, but that is the problem with most of the documentation, nobody really reads it even if it exists… which in the specific case of the Release file is even translated to a few languages -- i am to lazy to look it up now…). My script did exactly what apt-secure says. Well, at least so much as apt-secure is specific bout it. And the data was existing and well recognized by apt so it passed all available tests. At the time you reported that bug i also told you what was wrong in that script and how to fix it if you want to continue to use that script, so please don't
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Michal Suchanek michal.sucha...@ruk.cuni.cz writes: Excerpts from Ian Jackson's message of Thu May 17 14:53:30 +0200 2012: Michal Suchanek writes (Re: Bug#671503: general: APT repository format is not documented): Excerpts from Filipus Klutiero's message of Wed May 16 18:44:21 +0200 2012: Could you clarify how this differs from #481129? It's 4 years later. Sorry, forgot that I filed the bug already. It's quite some time. Given there is no feedback in 4 years I guess it is futile reporting this. Well, it's useful to bring it up again. Admittedly there is no text in social contract about using Debian-proprietary formats. And a format only defined by apt can read that is definitely Debian-proprietary there is no better term for that. Everyone agrees that it would be better if this were documented. (I have struggled on occasion myself due to the lack of documentation.) But I think the use of the word proprietary is going too far. It's certainly a special Debian format, but that wouldn't be changed if it were documented. But it's not secret and we publish at least two writer implementations and one reader implementation AFAIK, with proper Free licences. However, it's easier to reverse-engineer an existing repository than the source code so for all practical purposes it's the same as if it were closed source. I'd say it's slightly discriminatory against software not part of Debian that cannot rely on getting notified when apt can read that silently changes, there is no document defining what apt should be able to read that software authors can rely on to interoperate with apt, one of the core Debian tools. Apt in turn relies on open standards like HTTP and FTP to interoperate with the rest of the world. I think this is not an appropriate use of the social contract or its concepts. Rather than complaining that this documentation doesn't exist, how about writing the document yourself ? It's not a trivial job but it should be feasible by looking at the apt source code. For me it is not feasible at all. I can, of course, describe what current repositories look like or what the current apt code accepts. However, that has silently changed in the past and is considered apt feature, not a bug. Once such a document exists, even if it's a bit sketchy or perhaps not entirely accurate, it will be much easier to insist that future changes are likewise documented. I am not so sure about that. So long as the document merely describes what apt happens to do at the moment rather than apt implementing what the document says there is no saying this document has any value. The status was 'documented' by existing repositories which stopped working. Thanks Michal I would suggest you look at existing repositories, whatever scraps of information is in the manuals and maybe a bit at the source and start to write a documentation. Once you have that offer it for review and other people can pitch in their bits of knowledge. Getting the current format documented right shouldn't be that hard if someone just starts. And once such a document exists it is much easier to get people do document changes or hit them over the head if they don't. Remember that you don't have to be 100% right in what you write. You only need to write a draft to start the process. Getting people to comment and correct any mistakes you simply don't know about is much much easier than getting someone else to write the whole thing. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87mx55lqbo.fsf@frosties.localnet
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
CC'ing the apt list de...@lists.debian.org. Goswin von Brederlow writes (Re: Bug#481129: Bug#671503: general: APT repository format is not documented): Michal Suchanek michal.sucha...@ruk.cuni.cz writes: [ discussions regarding documenting the apt repository format ] I would suggest you look at existing repositories, whatever scraps of information is in the manuals and maybe a bit at the source and start to write a documentation. Once you have that offer it for review and other people can pitch in their bits of knowledge. Getting the current format documented right shouldn't be that hard if someone just starts. Right. And once such a document exists it is much easier to get people do document changes or hit them over the head if they don't. Can the apt maintainers confirm that once such a document exists, they will insist that future contributions to apt which change the repository format update the document ? What form do the apt maintainers think the document should take ? Should it eventually be in the apt source package or somewhere else ? Remember that you don't have to be 100% right in what you write. You only need to write a draft to start the process. Getting people to comment and correct any mistakes you simply don't know about is much much easier than getting someone else to write the whole thing. Indeed so. Thanks, Ian. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20406.11351.808519.174...@chiark.greenend.org.uk
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 12:02:47PM +0100, Ian Jackson wrote: CC'ing the apt list de...@lists.debian.org. Goswin von Brederlow writes (Re: Bug#481129: Bug#671503: general: APT repository format is not documented): Michal Suchanek michal.sucha...@ruk.cuni.cz writes: [ discussions regarding documenting the apt repository format ] I would suggest you look at existing repositories, whatever scraps of information is in the manuals and maybe a bit at the source and start to write a documentation. Once you have that offer it for review and other people can pitch in their bits of knowledge. Getting the current format documented right shouldn't be that hard if someone just starts. Right. And once such a document exists it is much easier to get people do document changes or hit them over the head if they don't. Can the apt maintainers confirm that once such a document exists, they will insist that future contributions to apt which change the repository format update the document ? What form do the apt maintainers think the document should take ? Should it eventually be in the apt source package or somewhere else ? I do not think that APT is responsible for the repository format. The repository format is defined by ftpmaster, not by APT. APT has to my knowledge not defined anything new, but only implemented changes to the repository format after they were introduced by ftpmaster (see InRelease files). We currently have three independent implementations of the repository format in the archive: APT, cupt, smartpm. Furthermore, tools like debian-cd probably also have some knowledge about the repository format. The repository format should thus be part of Policy, not part of APT. APT is one of the users of that format, not the one defining it (it might just get stricter in behavior from time to time, just like compilers). Changes to the format should require approval of ftpmaster, as they have to implement them on the server-side. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. pgpWPZByoczjc.pgp Description: PGP signature
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 01:38:40PM +0200, Julian Andres Klode wrote: On Fri, May 18, 2012 at 12:02:47PM +0100, Ian Jackson wrote: CC'ing the apt list de...@lists.debian.org. Goswin von Brederlow writes (Re: Bug#481129: Bug#671503: general: APT repository format is not documented): Michal Suchanek michal.sucha...@ruk.cuni.cz writes: [ discussions regarding documenting the apt repository format ] I would suggest you look at existing repositories, whatever scraps of information is in the manuals and maybe a bit at the source and start to write a documentation. Once you have that offer it for review and other people can pitch in their bits of knowledge. Getting the current format documented right shouldn't be that hard if someone just starts. Right. And once such a document exists it is much easier to get people do document changes or hit them over the head if they don't. Can the apt maintainers confirm that once such a document exists, they will insist that future contributions to apt which change the repository format update the document ? What form do the apt maintainers think the document should take ? Should it eventually be in the apt source package or somewhere else ? I do not think that APT is responsible for the repository format. The repository format is defined by ftpmaster, not by APT. APT has to my knowledge not defined anything new, but only implemented changes to the repository format after they were introduced by ftpmaster (see InRelease files). We currently have three independent implementations of the repository format in the archive: APT, cupt, smartpm. Furthermore, tools like debian-cd probably also have some knowledge about the repository format. The repository format should thus be part of Policy, not part of APT. APT is one of the users of that format, not the one defining it (it might just get stricter in behavior from time to time, just like compilers). Changes to the format should require approval of ftpmaster, as they have to implement them on the server-side. A working draft could be something like the following. It mostly describes the current format for Release, Packages, and Sources files. It's thus missing Contents and Translations, pdiffs, and stuff, but it's a beginning. It specifies different requirements for servers and clients, in order to have clients be backwards compatible with more repositories, and forcing servers to be stricter. Don't know how good that is, though. = Debian Repository Format = This documents a subset of the Debian repository format. This is a work in progress. Release files === The file dists/$DIST/Release shall contain meta-information about the distribution and checksums for the indices. The file dists/$DIST/Release.gpg shall be an GPG signature of the Release file, compatible with the format used by the GPG options -a -b -s. The file dists/$DIST/InRelease shall be the Release file with a GPG clearsign signature compatible with the format used by the GPG options -a -s --clearsign. The following fields might be available: - Origin - Label - Suite - Codename - Date - Valid-Until - Architectures - Components - Description - MD5sum, SHA1, SHA256 - NotAutomatic and ButAutomaticUpgrades Servers shall provide the Release file, and its signed counterparts with at least the following keys: - SHA256 - Origin - Suite and/or Codename - Architectures Clients shall accept missing Release files, and Release files without the fields required for servers. They might reject Release files that do not contain at least one of the fields defined herein. Architectures - Whitespace separated unique single words identifying Debian machine architectures as described in Architecture specification strings, Section 11.1. Origin -- Shall indicate the origin of the repository. Label - Optional field including some kind of label. Suite - The Suite field shall describe the suite. In Debian, this shall be one of oldstable, stable, testing, unstable, or experimental; with optional suffixes separated by - (such as stable-updates). Codename The Suite field shall describe the codename of the release. This is mostly a free-form string used to give a name to a release. Date, Valid-Until - The Date field shall specify the time at which the Release file was created. The Valid-Until field shall specify at which time the Release file should be considered expired by the client. Client behaviour on expired Release files is unspecified. Components --- A whitespace separated list of areas. Example: Components: main contrib non-free MD5sum, SHA1, SHA256 Those fields shall be multi-line fields containing multiple lines of whitespace separated
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 01:38:40PM +0200, Julian Andres Klode wrote: I do not think that APT is responsible for the repository format. The repository format is defined by ftpmaster, not by APT. APT has to my knowledge not defined anything new, but only implemented changes to the repository format after they were introduced by ftpmaster (see InRelease files). OK, we actually do have some more extensions not used by Debian or Ubuntu. One of those is the Important: yes field, which is like Essential, but does not force installation of the package like Essential would do (and does not force immediate configuration nowadays, so that we can use it for custom meta packages [so that users cannot accidentally remove the meta package that configures the complete system]). I don't know of any other extensions, though. In any case, they should probably not be part of an official specification, but rather documented in APT. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120518144739.ga21...@debian.org
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
FWIW posted on the wiki: http://wiki.debian.org/RepositoryFormat Thanks Michal -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1337349939-sup-8...@virtual.ruk.cuni.cz
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 04:06:23PM +0200, Michal Suchanek wrote:
FWIW
posted on the wiki: http://wiki.debian.org/RepositoryFormat
Thanks
Michal
I have now documented the Contents indices and the diffs
as well, mostly (sans the exact format we use for the
patches), and Translation indices. Now we're basically
only missing details, it is fairly complete otherwise
(i.e. we should have mentioned every file and field
currently in use, but may not have explained all of
them completely).
We now have documented
dists/$DIST/Release (and InRelease, Release.gpg)
dists/$DIST/$COMP/binary-$ARCH/Packages
dists/$DIST/$COMP/source/Sources
dists/$DIST/$COMP/Contents-$ARCH.gz
dists/$DIST/$COMP/i18n/{Index,Translation-*.bz2}
*.diff/Index *.diff/%Y-%m-%d-%H%M.%S.gz
The other Release files have been omitted, as they are not
used anywhere. We are only missing udeb content files and
packages files now, which are just small subsentences.
In a few months, I'd like to rework this in DocBook form,
and submit it to debian-policy for inclusion into official
Policy, as a sub-policy like copyright-format.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
pgpvMrHdLCzhm.pgp
Description: PGP signature
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 04:06:23PM +0200, Michal Suchanek wrote: FWIW posted on the wiki: http://wiki.debian.org/RepositoryFormat What's the opinion about the flat repository format, where you just have one directory with Release, Packages, Sources, and friends and no sub-directories? Should they be documented as well then? We would then have two kind of documented repository formats: 1. Debian-style, with a pool (or similar) and a dists directory 2. Flat-style, with just one directory This should cover everything we currently support. Although I don't know much about how much stuff we support in flat directories WRT Translation, Contents, and diffs. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120518185407.ga15...@debian.org
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 06:45:00PM +0100, Wookey wrote: +++ Julian Andres Klode [2012-05-18 13:38 +0200]: We currently have three independent implementations of the repository format in the archive: APT, cupt, smartpm. I think reprepro is another? Of course, I was just only talking about clients. When it comes to creating we probably have much more than 3 programs needing some knowledge of the repository format. We have dak, apt-ftparchive, reprepro, debian-cd, everyone's small script, mini-dinstall (the latter using flat repositories, if I am not mistaken). -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. pgpLQZvyw4KlE.pgp Description: PGP signature
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
+++ Julian Andres Klode [2012-05-18 13:38 +0200]: We currently have three independent implementations of the repository format in the archive: APT, cupt, smartpm. I think reprepro is another? /usr/share/doc/reprepro/manual.html contains a 'repository basics' section which includes useful layout/format information. Wookey -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120518174500.gf11...@stoneboat.aleph1.co.uk
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Fri, May 18, 2012 at 08:12:16PM +0200, Michal Suchanek wrote: The formatting is not consistent but that will have to be changed for docbook anyway. Yes, and it will also be more readable then, than the current wiki version. Also would need some proof-reading. If nothing else somebody should look in a few weeks from now if it still makes sense ;-) I'd also like to hear bits from the launchpad team about their implementation and see whether they agree with everything then. I put a link on the RepositoryHowto page for more exposure. I am not so sure documenting Debian installer files is tremendously useful. I don't think anyone outside Debian Installer team makes Debian Installer repositories and there are other aspects of Debian Installer that would need to be documented in order for it to be usable for 'outside' people in non-default configurations. We still need to at least document the udeb stuff, the images and other stuff is not relevant to the core format and probably defined by the d-i team anyway (and installed by-hand). The udeb stuff is relatively easy to document, as it just adds one new directory and one new filename for Contents files, so can be done in about two sentences (and udebs are uploaded by standard .changes with the rest of the package, and are thus standard). -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120518201701.ga17...@debian.org
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Excerpts from Julian Andres Klode's message of Fri May 18 18:49:10 +0200 2012:
On Fri, May 18, 2012 at 04:06:23PM +0200, Michal Suchanek wrote:
FWIW
posted on the wiki: http://wiki.debian.org/RepositoryFormat
Thanks
Michal
I have now documented the Contents indices and the diffs
as well, mostly (sans the exact format we use for the
patches), and Translation indices. Now we're basically
only missing details, it is fairly complete otherwise
(i.e. we should have mentioned every file and field
currently in use, but may not have explained all of
them completely).
We now have documented
dists/$DIST/Release (and InRelease, Release.gpg)
dists/$DIST/$COMP/binary-$ARCH/Packages
dists/$DIST/$COMP/source/Sources
dists/$DIST/$COMP/Contents-$ARCH.gz
dists/$DIST/$COMP/i18n/{Index,Translation-*.bz2}
*.diff/Index *.diff/%Y-%m-%d-%H%M.%S.gz
The other Release files have been omitted, as they are not
used anywhere. We are only missing udeb content files and
packages files now, which are just small subsentences.
In a few months, I'd like to rework this in DocBook form,
and submit it to debian-policy for inclusion into official
Policy, as a sub-policy like copyright-format.
Yes, looks fairly complete.
The formatting is not consistent but that will have to be changed for
docbook anyway.
Also would need some proof-reading. If nothing else somebody should look
in a few weeks from now if it still makes sense ;-)
I put a link on the RepositoryHowto page for more exposure.
I am not so sure documenting Debian installer files is tremendously
useful. I don't think anyone outside Debian Installer team makes Debian
Installer repositories and there are other aspects of Debian Installer
that would need to be documented in order for it to be usable for
'outside' people in non-default configurations.
Thanks
Michal
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1337363087-sup-4...@virtual.ruk.cuni.cz
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Sat, May 19, 2012 at 12:58 AM, Julian Andres Klode wrote: What's the opinion about the flat repository format, where you just have one directory with Release, Packages, Sources, and friends and no sub-directories? Should they be documented as well then? We would then have two kind of documented repository formats: 1. Debian-style, with a pool (or similar) and a dists directory 2. Flat-style, with just one directory This should cover everything we currently support. Although I don't know much about how much stuff we support in flat directories WRT Translation, Contents, and diffs. I would like to see the flat-style repository documented too, since some of the derivatives in the Debian derivatives census use it and I would like to lint their apt repositories. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caktje6gw9k_p_7ch-kzwnsxnpiwdbs3hxu1cyw_qjmgaydh...@mail.gmail.com
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
Le Fri, May 18, 2012 at 06:49:10PM +0200, Julian Andres Klode a écrit : In a few months, I'd like to rework this in DocBook form, and submit it to debian-policy for inclusion into official Policy, as a sub-policy like copyright-format. Dear Julian and everybody, thank you for this documentation effort. I think it is very useful. How about integrating it with the Policy's chapter 5 (thus enlarging its scope) instead of having it as a separate document ? That would help to underline when a field is used in the same way or differently as in the package control data files. Have a nice week-end, -- Charles Plessy Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120519042845.gd5...@falafel.plessy.net
Re: Bug#481129: Bug#671503: general: APT repository format is not documented
On Thu, May 17, 2012 at 3:16 PM, Michal Suchanek michal.sucha...@ruk.cuni.cz wrote: Excerpts from Ian Jackson's message of Thu May 17 14:53:30 +0200 2012: Michal Suchanek writes (Re: Bug#671503: general: APT repository format is not documented): Excerpts from Filipus Klutiero's message of Wed May 16 18:44:21 +0200 2012: Could you clarify how this differs from #481129? It's 4 years later. Sorry, forgot that I filed the bug already. It's quite some time. Given there is no feedback in 4 years I guess it is futile reporting this. Well, it's useful to bring it up again. Admittedly there is no text in social contract about using Debian-proprietary formats. And a format only defined by apt can read that is definitely Debian-proprietary there is no better term for that. Everyone agrees that it would be better if this were documented. (I have struggled on occasion myself due to the lack of documentation.) But I think the use of the word proprietary is going too far. It's certainly a special Debian format, but that wouldn't be changed if it were documented. But it's not secret and we publish at least two writer implementations and one reader implementation AFAIK, with proper Free licences. However, it's easier to reverse-engineer an existing repository than the source code so for all practical purposes it's the same as if it were closed source. That is non-sense. You said yourself that the repository is not sufficient to understand it, yet you say that it is easier to understand with it than with looking at the source (and the various bits and pieces where parts are documented). Even if you don't like apt sources, debian has a lot of other tools working with the same repository in a bunch of different languages, so choose which you like most and you will properly find a tool in that language to either download from repositories or to create them. But I don't know why we are still talking here. Russ already said he would like to have it as a subpolicy in the debian-policy. ftpmasters already said they would accept maintaining it. Everything left is writing this goddamn piece of documentation. So, maybe you should just write it… If you want to be extra fancy, start a wikipage in the debian wiki, but start typing. Go to debian-dak@l.d.o and discuss your work there. Would be way more productive than talking about that this document is missing… Everbody knows that. Everybody doesn't like it. Now go and fix that. That everyone would like to have such a document but nobody has it so far is a strong indication that the current people are busy with other stuff. An opportunity to get involved, I would say. I'd say it's slightly discriminatory against software not part of Debian that cannot rely on getting notified when apt can read that silently changes, there is no document defining what apt should be able to read that software authors can rely on to interoperate with apt, one of the core Debian tools. Apt in turn relies on open standards like HTTP and FTP to interoperate with the rest of the world. I think this is not an appropriate use of the social contract or its concepts. Rather than complaining that this documentation doesn't exist, how about writing the document yourself ? It's not a trivial job but it should be feasible by looking at the apt source code. For me it is not feasible at all. I can, of course, describe what current repositories look like or what the current apt code accepts. However, that has silently changed in the past and is considered apt feature, not a bug. It hasn't silently changed. It was and is still the same. Your script was just horribly wrong and older APT versions just happened to work with that brokenness a little better. What you created with that script was NEVER intended to work, it just happened to be working out of complete luck (A Release file is supposed to include current data, not non-existent data, this conclusion is reachable even without too much guessing. Beside that this is actually documented in apt-secure and co, but that is the problem with most of the documentation, nobody really reads it even if it exists… which in the specific case of the Release file is even translated to a few languages -- i am to lazy to look it up now…). At the time you reported that bug i also told you what was wrong in that script and how to fix it if you want to continue to use that script, so please don't keep up the myth that we changed everything and nobody told you why and how and what not. If you are really that unsatisfied with apt, feel free to use something else, we have enough tools to choose from, it not like apt would be the only package manager in existence. It might be the most used one, but that doesn't say anything about documentation or available manpower. The status was 'documented' by existing repositories which stopped working. You would complain to the mechanic who fixed the
