Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On Mon, 2018-11-19 at 19:21 +, Roy Bamford wrote: > On 2018.11.19 18:35, Michał Górny wrote: > > Hi, > > > > On Sat, 2018-11-17 at 12:21 +0100, Michał Górny wrote: > > > Here's a pre-GLEP draft based on the earlier discussion on gentoo- > > > portage-dev mailing list. The specification uses GLEP form as it > > > provides for cleanly specifying the motivation and rationale. > > > > Changes in -r1: took into account the feedback and restructured > > the motivation into pointing out advantages of the existing format, > > and focusing on the two real issues of non-transparency and OpenPGP > > implementations deficiencies. Also added a section on why there's no > > explicit version number. > > > > > Also available via HTTPS: > > > > > > rst: https://dev.gentoo.org/~mgorny/tmp/glep-0078.rst > > > html: https://dev.gentoo.org/~mgorny/tmp/glep-0078.html > > > > > [snip] > > Team, > > Looks good to me. I can manually unpick the binpackage with tar. > Choose, if I will check the signatures or not, then spray files all > over my broken Gentoo with tar in the same way as I do now. > > Implementation detail question. > It appears that all members must be signed, or none of them since > > "The archive members support optional OpenPGP signatures. > The implementations must allow the user to specify whether OpenPGP > signatures are to be expected in remotely fetched packages." > > Or can the user specify that only some elements need to be signed? This is really out of scope. The only purpose of this paragraph is to explain that '(optional)' doesn't mean you can safely ignore the lack of this file. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On 2018.11.19 19:33, Rich Freeman wrote: > On Mon, Nov 19, 2018 at 2:21 PM Roy Bamford > wrote: > > > > "The archive members support optional OpenPGP signatures. > > The implementations must allow the user to specify whether OpenPGP > > signatures are to be expected in remotely fetched packages." > > > > Or can the user specify that only some elements need to be signed? > > > > Is it a problem if not all elements are signed with the same key? > > That could happen if one person makes a binpackage and someone > > else updates the metadata. > > > > IMO this is going a bit into PM details for a GLEP that is about > container formats. > Rich, Not really. The GLEP needs to be clear about the signing. Is it every element or none? The GLEP hints that a mix of is possible with If the implementation needs to manipulate archive members, it must either create a new signature or discard the existing signature. An individual binpackage could start life with all elements signed by the same key. Some element could be updated and the key for the signature of that element changed. Later still, another element can be changed an have its signature dropped. Should some combinations have no practical value, they should not be permitted by the GLEP. > -- > Rich > > > -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods pgpaUbIZBgWaT.pgp Description: PGP signature
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On Mon, Nov 19, 2018 at 2:40 PM Zac Medico wrote: > > On 11/19/18 11:33 AM, Rich Freeman wrote: > > On Mon, Nov 19, 2018 at 2:21 PM Roy Bamford wrote: > >> > >> "The archive members support optional OpenPGP signatures. > >> The implementations must allow the user to specify whether OpenPGP > >> signatures are to be expected in remotely fetched packages." > >> > >> Or can the user specify that only some elements need to be signed? > >> > >> Is it a problem if not all elements are signed with the same key? > >> That could happen if one person makes a binpackage and someone > >> else updates the metadata. > >> > > > > IMO this is going a bit into PM details for a GLEP that is about > > container formats. > > > > Presumably any package manager is going to need to figure out what > > keys are/aren't valid and allow the user to configure this behavior. > > Users who want to go editing package innards will presumably adjust > > their package manager settings to accept their modifications, whether > > it means accepting their own sigs or disabling them. > > With the GLEP as it is, the user *must* use a local signing key to sign > installed packages during the installation process if they want to be > able to verify signatures for installed packages at some point in the > future, since the binary package format does not provide a way to use > binary package signatures for this purpose. I think we might be talking about different signatures? I think you're referring to signatures of the package files after they are installed on the local filesystem, while I'm talking about verifying the integrity of the package file themselves. If these signatures are applied to different data then obviously you couldn't just have the one signature serve double duty (unless you hung onto the binary package, verified the signature on it, then verified the package contents against the live filesystem). The simplest solution would be to do as you seem to be suggesting - verify the signature on the package before installing it, and then during installation capture whatever metadata is already supported by portage and sign that using a user's trusted key. This seems like the most practical solution in any case since we aren't likely to ever go down the route of using a single signed squashfs for /usr like a release-based binary distro might. -- Rich
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On 11/19/18 11:33 AM, Rich Freeman wrote: > On Mon, Nov 19, 2018 at 2:21 PM Roy Bamford wrote: >> >> "The archive members support optional OpenPGP signatures. >> The implementations must allow the user to specify whether OpenPGP >> signatures are to be expected in remotely fetched packages." >> >> Or can the user specify that only some elements need to be signed? >> >> Is it a problem if not all elements are signed with the same key? >> That could happen if one person makes a binpackage and someone >> else updates the metadata. >> > > IMO this is going a bit into PM details for a GLEP that is about > container formats. > > Presumably any package manager is going to need to figure out what > keys are/aren't valid and allow the user to configure this behavior. > Users who want to go editing package innards will presumably adjust > their package manager settings to accept their modifications, whether > it means accepting their own sigs or disabling them. With the GLEP as it is, the user *must* use a local signing key to sign installed packages during the installation process if they want to be able to verify signatures for installed packages at some point in the future, since the binary package format does not provide a way to use binary package signatures for this purpose. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On Mon, Nov 19, 2018 at 2:21 PM Roy Bamford wrote: > > "The archive members support optional OpenPGP signatures. > The implementations must allow the user to specify whether OpenPGP > signatures are to be expected in remotely fetched packages." > > Or can the user specify that only some elements need to be signed? > > Is it a problem if not all elements are signed with the same key? > That could happen if one person makes a binpackage and someone > else updates the metadata. > IMO this is going a bit into PM details for a GLEP that is about container formats. Presumably any package manager is going to need to figure out what keys are/aren't valid and allow the user to configure this behavior. Users who want to go editing package innards will presumably adjust their package manager settings to accept their modifications, whether it means accepting their own sigs or disabling them. -- Rich
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
On 2018.11.19 18:35, Michał Górny wrote: > Hi, > > On Sat, 2018-11-17 at 12:21 +0100, Michał Górny wrote: > > Here's a pre-GLEP draft based on the earlier discussion on gentoo- > > portage-dev mailing list. The specification uses GLEP form as it > > provides for cleanly specifying the motivation and rationale. > > Changes in -r1: took into account the feedback and restructured > the motivation into pointing out advantages of the existing format, > and focusing on the two real issues of non-transparency and OpenPGP > implementations deficiencies. Also added a section on why there's no > explicit version number. > > > Also available via HTTPS: > > > > rst: https://dev.gentoo.org/~mgorny/tmp/glep-0078.rst > > html: https://dev.gentoo.org/~mgorny/tmp/glep-0078.html > > > [snip] Team, Looks good to me. I can manually unpick the binpackage with tar. Choose, if I will check the signatures or not, then spray files all over my broken Gentoo with tar in the same way as I do now. Implementation detail question. It appears that all members must be signed, or none of them since "The archive members support optional OpenPGP signatures. The implementations must allow the user to specify whether OpenPGP signatures are to be expected in remotely fetched packages." Or can the user specify that only some elements need to be signed? Is it a problem if not all elements are signed with the same key? That could happen if one person makes a binpackage and someone else updates the metadata. > -- > Best regards, > Michał Górny > -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods pgpX6ueFyt3EF.pgp Description: PGP signature
Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format
Hi, On Sat, 2018-11-17 at 12:21 +0100, Michał Górny wrote: > Here's a pre-GLEP draft based on the earlier discussion on gentoo- > portage-dev mailing list. The specification uses GLEP form as it > provides for cleanly specifying the motivation and rationale. Changes in -r1: took into account the feedback and restructured the motivation into pointing out advantages of the existing format, and focusing on the two real issues of non-transparency and OpenPGP implementations deficiencies. Also added a section on why there's no explicit version number. > Also available via HTTPS: > > rst: https://dev.gentoo.org/~mgorny/tmp/glep-0078.rst > html: https://dev.gentoo.org/~mgorny/tmp/glep-0078.html > --- GLEP: Title: Gentoo binary package container format Author: Michał Górny Type: Standards Track Status: Draft Version: 1 Created: 2018-11-15 Last-Modified: 2018-11-16 Post-History: 2018-11-17 Content-Type: text/x-rst --- Abstract This GLEP proposes a new binary package container format for Gentoo. The current tbz2/XPAK format is shortly described, and its deficiences are explained. Accordingly, the requirements for a new format are set and a gpkg format satisfying them is proposed. The rationale for the design decisions is provided. Motivation == The current Portage binary package format - The historical ``.tbz2`` binary package format used by Portage is a concatenation of two distinct formats: header-oriented compressed .tar format (used to hold package files) and trailer-oriented custom XPAK format (used to hold metadata) [#MAN-XPAK]_. The format has already been extended incompatibly twice. The first time, support for storing multiple successive builds of binary package for a single ebuild version has been added. This feature relies on appending additional hyphen, followed by an integer to the package filename. It is disabled by default (preserving backwards compatibility) and controlled by ``binpkg-multi-instance`` feature. The second time, support for additional compression formats has been added. When format other than bzip2 is used, the ``.tbz2`` suffix is replaced by ``.xpak`` and Portage relies on magic bytes to detect compression used. For backwards compatibility, Portage still defaults to using bzip2; compression program can be switched using ``BINPKG_COMPRESS`` configuration variable. Additionally, there have been minor changes to the stored metadata and file storage policies. In particular, behavior regarding ``INSTALL_MASK``, controllable file compression and stripping has changed over time. The advantages of tbz2/XPAK format -- The tbz2/XPAK format used by Portage has three interesting features: 1. **Each binary package is fully contained within a single file.** While this might seem unnecessary, it makes it easier for the user to transfer binary packages without having to be concerned about finding all the necessary files to transfer. 2. **The binary packages are compatible with regular compressed tarballs, most of the time.** With notable exceptions of historical versions of pbzip2 and the recent zstd compressor, tbz2/XPAK packages can be extracted using regular tar utility with a compressor implementation that discards trailing garbage. 3. **The metadata is uncompressed, and can be efficiently accessed without decompressing package contents.** This includes the possibility of rewriting it (e.g. as a result of package moves) without the necessity of repacking the files. Transparency problem with the current binary package format --- Notwithstanding its advantages, the tbz2/XPAK format has a significant design fault that consists of two issues: 1. **The XPAK format is a custom binary format with explicit use of binary-encoded file offsets and field lengths.** As such, it is non-trivial to read or edit without specialized tools. Such tools are currently implemented separately from the package manager, as part of the portage-utils toolkit, written in C [#PORTAGE-UTILS]_. 2. **The tarball compatibility feature relies on obscure feature of ignoring trailing garbage in compressed files**. While this is implemented consistently in most of the compressors, this feature is not really a part of specification but rather traditional behavior. Given that the original reasons for this no longer apply, new compressor implementations are likely to miss support for this. Both of the issues make the format hard to use without dedicated tools, or when the tools misbehave. This impacts the following scenarios: A. **Using binary packages for system recovery.** In case of serious breakage, it is really preferable that the format depends on as few tools a possible, and especially not on Gentoo-specific tools. B. **Inspecting binary packages in detail exceeding
