Re: [gentoo-dev] [pre-GLEP r3] Gentoo binary package container format
On 2018.11.26 18:58, Michał Górny wrote: > Here's the newest version. > > Changes: > > - added explicit notion of parent directory (missing in previous GLEP > but present in implementation), > > - explicitly named GNU tar format with list of permitted extensions, > > - changed volume label to 'gpkg-1.txt' file to improve portability; > made > it explicit version identifier as well, > > - added info on other package formats to rationale. > [snip] The image archive stores all the files to be installed by the binary package. It should be included as the last of the files in the binary package container. [snip] > > -- > Best regards, > Michał Górny > Its a nit today but that says that any future extensions, none yet planned, should be placed before the image archive. The specification needs to avoid the use of relative references. -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods pgpXu1HdOG3la.pgp Description: PGP signature
Re: [gentoo-dev] Re: [gentoo-dev-announce] Packages up for grabs from xmw@g.o
Am Montag, 26. November 2018, 02:37:14 CET schrieb Benda Xu: > Michał Górny writes: > > > > x11-wm/xpra > > > I take this, the "X Persistent Remote Apps". Happy to help, I'm also using this. (Awesome unknown app of the year. The one thing that makes Mathematica useful over a network link. :) -- Andreas K. Hüttel dilfri...@gentoo.org Gentoo Linux developer (council, toolchain, perl, libreoffice, comrel) signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format
On Wed, Nov 21, 2018 at 11:45:54AM +0100, Fabian Groffen wrote: > We agree it is hackish, and we agree we can do without. You simply > exaggerate the problem, IMO, which mostly isn't there, because it works > fine today. It can also be solved today using shell tools. I am sad that you don't see it as a productivity impediment that the user is required to know the custom tooling to do even such a trivial non-standard action as manual extraction. Maybe I will make myself look bad by admitting this, but I'm not meeting your expectations. I use Gentoo for ~11 years, and for about one year I am using my private binpkgs distributed to all my machines (i.e. I have read binary package guide fair number of times, but I stopped rereading it when I satisfied my needs). When in need, I still reached to trusty tar, and I did not even know what are the names of special tools (a toolchain?) qtbz2 and qxpak. Just few days ago I messed with binpkgs for investigation purpose. I just wanted to extract few to somewhere (definitely not into system root), and read a core dump with GDB asking it to use those extracted files for debug symbols. Of course I used `tar xaf`, because what I know is that it's honest tbz2 just with metadata appended. # tar xaf boost-1.65.0.tbz2 bzip2: (stdin): trailing garbage after EOF ignored Exit code is 0. But the notice is annoying (on subconscious level), because Silence Is Golden - "when a program has nothing interesting or surprising to say, it should shut up". > % head -c `grep -abo 'XPAKPACK' > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | sed 's/:.*$//'` > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | tar -jxf - > > results in no warnings/errors from bzip about trailing garbage, possible > thanks to the spec being smart enough about this. Thanks, this is a very concise **custom tool** to handle current binpkg format. > Not having to do this, when under stress and pressure to restore a > system to get it back into production, is a plus. Though, in that > scenario the trailing garbage warning wouldn't have been that bad > either. When understress and pressure, the irrelevant warning is not bad? I am sure it is really bad for operator's attention. signature.asc Description: Digital signature
Re: [gentoo-dev] [pre-GLEP r3] Gentoo binary package container format
On Mon, 2018-11-26 at 20:17 +0100, Ulrich Mueller wrote: > > > > > > On Mon, 26 Nov 2018, Michał Górny wrote: > > Specification > > = > > The container format > > > > The gpkg package container is an uncompressed .tar achive whose filename > > should use ``.gpkg.tar`` suffix. This archive contains the following > > members, all placed in a single directory whose name matches > > the basename of the package file, in order: > > I see no value in adding another directory indirection, and it will add > more overhead. Tar bomb is not a good design. Given tar padding, there will be no overhead unless the full path exceeds ustar limits which is unlikely. > Also, AFAICS the tar|tar pipeline that you previously > suggested won't work any more (or would at least require additional > arguments). I'm pretty sure the tar pipeline was actually written with account for the directory. > > > 1. The package identifier file ``gpkg-1.txt`` (required). > > [...] > > The implementations must include a package identifier file named > > ``gpkg-1.txt``. The filename includes package format version; > > implementations should reject packages which do not contain this file > > as unsupported format. > > The file can have any contents. Normally, it should be empty. > > If the file is empty, why is it named gpkg-1.txt (instead of just > gpkg-1)? > *shrug*. I can make it 'gpkg-1' or 'gpkg.1' or whatever you want ;-). -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [pre-GLEP r3] Gentoo binary package container format
> On Mon, 26 Nov 2018, Michał Górny wrote: > Specification > = > The container format > > The gpkg package container is an uncompressed .tar achive whose filename > should use ``.gpkg.tar`` suffix. This archive contains the following > members, all placed in a single directory whose name matches > the basename of the package file, in order: I see no value in adding another directory indirection, and it will add more overhead. Also, AFAICS the tar|tar pipeline that you previously suggested won't work any more (or would at least require additional arguments). > 1. The package identifier file ``gpkg-1.txt`` (required). > [...] > The implementations must include a package identifier file named > ``gpkg-1.txt``. The filename includes package format version; > implementations should reject packages which do not contain this file > as unsupported format. > The file can have any contents. Normally, it should be empty. If the file is empty, why is it named gpkg-1.txt (instead of just gpkg-1)? Ulrich signature.asc Description: PGP signature
Re: [gentoo-dev] [pre-GLEP r3] Gentoo binary package container format
Here's the newest version. Changes: - added explicit notion of parent directory (missing in previous GLEP but present in implementation), - explicitly named GNU tar format with list of permitted extensions, - changed volume label to 'gpkg-1.txt' file to improve portability; made it explicit version identifier as well, - added info on other package formats to rationale. --- GLEP: Title: Gentoo binary package container format Author: Michał Górny Type: Standards Track Status: Draft Version: 1 Created: 2018-11-15 Last-Modified: 2018-11-26 Post-History: 2018-11-17 Content-Type: text/x-rst --- Abstract This GLEP proposes a new binary package container format for Gentoo. The current tbz2/XPAK format is shortly described, and its deficiences are explained. Accordingly, the requirements for a new format are set and a gpkg format satisfying them is proposed. The rationale for the design decisions is provided. Motivation == The current Portage binary package format - The historical ``.tbz2`` binary package format used by Portage is a concatenation of two distinct formats: header-oriented compressed .tar format (used to hold package files) and trailer-oriented custom XPAK format (used to hold metadata) [#MAN-XPAK]_. The format has already been extended incompatibly twice. The first time, support for storing multiple successive builds of binary package for a single ebuild version has been added. This feature relies on appending additional hyphen, followed by an integer to the package filename. It is disabled by default (preserving backwards compatibility) and controlled by ``binpkg-multi-instance`` feature. The second time, support for additional compression formats has been added. When format other than bzip2 is used, the ``.tbz2`` suffix is replaced by ``.xpak`` and Portage relies on magic bytes to detect compression used. For backwards compatibility, Portage still defaults to using bzip2; compression program can be switched using ``BINPKG_COMPRESS`` configuration variable. Additionally, there have been minor changes to the stored metadata and file storage policies. In particular, behavior regarding ``INSTALL_MASK``, controllable file compression and stripping has changed over time. The advantages of tbz2/XPAK format -- The tbz2/XPAK format used by Portage has three interesting features: 1. **Each binary package is fully contained within a single file.** While this might seem unnecessary, it makes it easier for the user to transfer binary packages without having to be concerned about finding all the necessary files to transfer. 2. **The binary packages are compatible with regular compressed tarballs, most of the time.** With notable exceptions of historical versions of pbzip2 and the recent zstd compressor, tbz2/XPAK packages can be extracted using regular tar utility with a compressor implementation that discards trailing garbage. 3. **The metadata is uncompressed, and can be efficiently accessed without decompressing package contents.** This includes the possibility of rewriting it (e.g. as a result of package moves) without the necessity of repacking the files. Transparency problem with the current binary package format --- Notwithstanding its advantages, the tbz2/XPAK format has a significant design fault that consists of two issues: 1. **The XPAK format is a custom binary format with explicit use of binary-encoded file offsets and field lengths.** As such, it is non-trivial to read or edit without specialized tools. Such tools are currently implemented separately from the package manager, as part of the portage-utils toolkit, written in C [#PORTAGE-UTILS]_. 2. **The tarball compatibility feature relies on obscure feature of ignoring trailing garbage in compressed files**. While this is implemented consistently in most of the compressors, this feature is not really a part of specification but rather traditional behavior. Given that the original reasons for this no longer apply, new compressor implementations are likely to miss support for this. Both of the issues make the format hard to use without dedicated tools, or when the tools misbehave. This impacts the following scenarios: A. **Using binary packages for system recovery.** In case of serious breakage, it is really preferable that the format depends on as few tools a possible, and especially not on Gentoo-specific tools. B. **Inspecting binary packages in detail exceeding standard package manager facilities.** C. **Modifying binary packages in ways not predicted by the package manager authors.** A real-life example of this is working around broken ``pkg_*`` phases which prevent the package from being installed. OpenPGP extensibility problem - There are at
Re: [gentoo-dev] Packages up for grabs from xmw@g.o
On Mon, 2018-11-26 at 13:20 +0700, gro...@gentoo.org wrote: > On Sun, 25 Nov 2018, Michał Górny wrote: > > dev-util/pycharm-community > > I'll take this one. > FYI, zlogene has already taken it. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-dev] Re: netsurf Prefix breakage
OK Virgil, Virgil Dupras writes: > Discussing this change on the list? I haven't touched the eclass, I > simply fixed the xmw-abandoned netsurf package which had been broken for > 3 months prior to that (and by broken, I don't mean only on Prefix. Why > talk about 1-1 features in those situations?). Fixing packages without > touching eclasses doesn't require gentoo-dev review. > > Also, netsurf doesn't have any prefix keyword. Some of the lower-level > libs have ~m68k-mint, but it isn't clear why because the only > non-netsurf revdep to those libs is net-libs/libwebsockets and it isn't > keyworded for prefix either. > > Please, open a ticket on the appropriate package with a description > of an actual breakage and I'll be deligthed to address it. Well, all in all, thank you for stepping up to take care of netsurf-buildsystem. You're good. Yours, Benda signature.asc Description: PGP signature
Re: netsurf Prefix breakage (Was: [gentoo-dev] Packages up for grabs from xmw@g.o)
On Mon, 26 Nov 2018 20:34:56 +0800 Benda Xu wrote: > This is a serious regression for Prefix and caught a lot of users > without a warning. Could you please discuss on this kind of changes in > the mailing list beforehand? Discussing this change on the list? I haven't touched the eclass, I simply fixed the xmw-abandoned netsurf package which had been broken for 3 months prior to that (and by broken, I don't mean only on Prefix. Why talk about 1-1 features in those situations?). Fixing packages without touching eclasses doesn't require gentoo-dev review. Also, netsurf doesn't have any prefix keyword. Some of the lower-level libs have ~m68k-mint, but it isn't clear why because the only non-netsurf revdep to those libs is net-libs/libwebsockets and it isn't keyworded for prefix either. Please, open a ticket on the appropriate package with a description of an actual breakage and I'll be deligthed to address it. Regards, Virgil pgpafpvucDBRx.pgp Description: PGP signature
netsurf Prefix breakage (Was: [gentoo-dev] Packages up for grabs from xmw@g.o)
Hi Virgil, Virgil Dupras writes: > I've been getting rid of netsurf.eclass' usage. This can soon be > last-rited. Only two-stabilizations away from that goal. I am sorry to raise this, but your migration of netsurf.eclass into dev-util/netsurf-buildsystem is not a 1-1 feature copy of the old eclass. The new ebuild breaks Prefix. For example, NSSHARED=/usr/share/netsurf-buildsystem This is a serious regression for Prefix and caught a lot of users without a warning. Could you please discuss on this kind of changes in the mailing list beforehand? Yours, Benda signature.asc Description: PGP signature
