Re: [gentoo-portage-dev] [PATCH v2] Support disabling docompress for binary package builds
On 11/04/2018 01:20 AM, Michał Górny wrote: > Add FEATURES=binpkg-docompress that can be used whether docompress > compression is performed before or after creating binary packages. With > the feature enabled (the default), the current behavior of storing > compressed files in binpkg is preserved. With it disabled, uncompressed > files are stored inside binary package and are compressed when > installing. > > Storing uncompressed files in binary packages has two advantages: > > 1. Avoids the double-compression penalty, effectively improving binary >package compression speed and compression ratio. > > 2. Allows the same packages to be reused on systems with different >docompress configurations. > > The option is roughly backwards compatible. Old Portage versions will > install packages created with FEATURES=-binpkg-docompress correctly, > albeit without compression. Portage with FEATURES=binpkg-docompress > should install old binpackages semi-correctly, potentially recompressing > them (and throwing already-compressed warnings on format mismatch). > The new behavior is left off by default to avoid those problems. > > Signed-off-by: Michał Górny > --- > bin/misc-functions.sh | 43 +++--- > bin/phase-functions.sh | 2 +- > cnf/make.globals | 2 +- > lib/portage/const.py | 1 + > lib/portage/dbapi/vartree.py | 12 +++ > lib/portage/package/ebuild/doebuild.py | 4 ++- > man/ebuild.1 | 5 +++ > man/make.conf.5| 6 > 8 files changed, 68 insertions(+), 7 deletions(-) > > Changes in v2: > * added .instprepped file logic, > * added instprep -> install dependency, > * documented in ebuild.1. > > I've tested this thoroughly via ebuild(1) and emerge(1). Looks good, thanks, please merge. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH v2] Support disabling docompress for binary package builds
Add FEATURES=binpkg-docompress that can be used whether docompress
compression is performed before or after creating binary packages. With
the feature enabled (the default), the current behavior of storing
compressed files in binpkg is preserved. With it disabled, uncompressed
files are stored inside binary package and are compressed when
installing.
Storing uncompressed files in binary packages has two advantages:
1. Avoids the double-compression penalty, effectively improving binary
package compression speed and compression ratio.
2. Allows the same packages to be reused on systems with different
docompress configurations.
The option is roughly backwards compatible. Old Portage versions will
install packages created with FEATURES=-binpkg-docompress correctly,
albeit without compression. Portage with FEATURES=binpkg-docompress
should install old binpackages semi-correctly, potentially recompressing
them (and throwing already-compressed warnings on format mismatch).
The new behavior is left off by default to avoid those problems.
Signed-off-by: Michał Górny
---
bin/misc-functions.sh | 43 +++---
bin/phase-functions.sh | 2 +-
cnf/make.globals | 2 +-
lib/portage/const.py | 1 +
lib/portage/dbapi/vartree.py | 12 +++
lib/portage/package/ebuild/doebuild.py | 4 ++-
man/ebuild.1 | 5 +++
man/make.conf.5| 6
8 files changed, 68 insertions(+), 7 deletions(-)
Changes in v2:
* added .instprepped file logic,
* added instprep -> install dependency,
* documented in ebuild.1.
I've tested this thoroughly via ebuild(1) and emerge(1).
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index db7aaed5a..3b3a5e09c 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -109,10 +109,13 @@ install_qa_check() {
[[ -d ${ED%/}/usr/share/info ]] && prepinfo
- # Apply compression.
- "${PORTAGE_BIN_PATH}"/ecompress --queue "${PORTAGE_DOCOMPRESS[@]}"
- "${PORTAGE_BIN_PATH}"/ecompress --ignore "${PORTAGE_DOCOMPRESS_SKIP[@]}"
- "${PORTAGE_BIN_PATH}"/ecompress --dequeue
+ # If binpkg-docompress is enabled, apply compression before creating
+ # the binary package.
+ if has binpkg-docompress ${FEATURES}; then
+ "${PORTAGE_BIN_PATH}"/ecompress --queue
"${PORTAGE_DOCOMPRESS[@]}"
+ "${PORTAGE_BIN_PATH}"/ecompress --ignore
"${PORTAGE_DOCOMPRESS_SKIP[@]}"
+ "${PORTAGE_BIN_PATH}"/ecompress --dequeue
+ fi
export STRIP_MASK
if ___eapi_has_dostrip; then
@@ -160,6 +163,38 @@ install_qa_check() {
rm -f "${ED%/}"/usr/share/info/dir{,.gz,.bz2} || die "rm failed!"
}
+__dyn_instprep() {
+ if [[ -e ${PORTAGE_BUILDDIR}/.instprepped ]] ; then
+ __vecho ">>> It appears that '$PF' is already instprepped;
skipping."
+ __vecho ">>> Remove '${PORTAGE_BUILDDIR}/.instprepped' to force
instprep."
+ return 0
+ fi
+
+ if has chflags ${FEATURES}; then
+ # Save all the file flags for restoration afterwards.
+ mtree -c -p "${ED}" -k flags > "${T}/bsdflags.mtree"
+ # Remove all the file flags so that we can do anything
necessary.
+ chflags -R noschg,nouchg,nosappnd,nouappnd "${ED}"
+ chflags -R nosunlnk,nouunlnk "${ED}" 2>/dev/null
+ fi
+
+ # If binpkg-docompress is disabled, we need to apply compression
+ # before installing.
+ if ! has binpkg-docompress ${FEATURES}; then
+ "${PORTAGE_BIN_PATH}"/ecompress --queue
"${PORTAGE_DOCOMPRESS[@]}"
+ "${PORTAGE_BIN_PATH}"/ecompress --ignore
"${PORTAGE_DOCOMPRESS_SKIP[@]}"
+ "${PORTAGE_BIN_PATH}"/ecompress --dequeue
+ fi
+
+ if has chflags ${FEATURES}; then
+ # Restore all the file flags that were saved earlier on.
+ mtree -U -e -p "${ED}" -k flags < "${T}/bsdflags.mtree" &>
/dev/null
+ fi
+
+ >> "${PORTAGE_BUILDDIR}/.instprepped" || \
+ die "Failed to create ${PORTAGE_BUILDDIR}/.instprepped"
+}
+
preinst_qa_check() {
postinst_qa_check preinst
}
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index 51b480bfb..d8ebf3d3e 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -288,7 +288,7 @@ __dyn_clean() {
if [[ $EMERGE_FROM = binary ]] || ! has keepwork $FEATURES; then
rm -f
"$PORTAGE_BUILDDIR"/.{ebuild_changed,logid,pretended,setuped,unpacked,prepared}
\
-
"$PORTAGE_BUILDDIR"/.{configured,compiled,tested,packaged} \
+
"$PORTAGE_BUILDDIR"/.{configured,compiled,tested,packaged,instprepped} \
"$PORTAGE_BUILDDIR"/.die_hooks \
"$PORTAGE_BUILDDIR"/.ipc_{in,out,lock} \
Re: [gentoo-portage-dev] [PATCH] Support disabling docompress for binary package builds
On Sat, 2018-11-03 at 15:44 -0700, Zac Medico wrote:
> On 11/03/2018 01:10 AM, Michał Górny wrote:
> > On Fri, 2018-11-02 at 19:35 -0700, Zac Medico wrote:
> > > On 11/02/2018 12:35 PM, Michał Górny wrote:
> > > > On Fri, 2018-11-02 at 12:22 -0700, Zac Medico wrote:
> > > > > On 11/02/2018 12:16 PM, Michał Górny wrote:
> > > > > > On Fri, 2018-11-02 at 11:49 -0700, Zac Medico wrote:
> > > > > > > On 11/01/2018 02:50 AM, Michał Górny wrote:
> > > > > > > > diff --git a/lib/portage/package/ebuild/doebuild.py
> > > > > > > > b/lib/portage/package/ebuild/doebuild.py
> > > > > > > > index 9706de422..d7b535698 100644
> > > > > > > > --- a/lib/portage/package/ebuild/doebuild.py
> > > > > > > > +++ b/lib/portage/package/ebuild/doebuild.py
> > > > > > > > @@ -674,7 +674,7 @@ def doebuild(myebuild, mydo,
> > > > > > > > _unused=DeprecationWarning, settings=None, debug=0,
> > > > > > > > "config", "info", "setup", "depend",
> > > > > > > > "pretend",
> > > > > > > > "fetch", "fetchall", "digest",
> > > > > > > > "unpack", "prepare", "configure",
> > > > > > > > "compile", "test",
> > > > > > > > - "install", "rpm", "qmerge", "merge",
> > > > > > > > + "install", "instprep", "rpm", "qmerge",
> > > > > > > > "merge",
> > > > > > > > "package", "unmerge", "manifest",
> > > > > > > > "nofetch"]
> > > > > > > >
> > > > > > > > if mydo not in validcommands:
> > > > > > > > @@ -1402,6 +1402,7 @@ def _spawn_actionmap(settings):
> > > > > > > > "compile": {"cmd":ebuild_sh, "args":{"droppriv":droppriv,
> > > > > > > > "free":nosandbox, "sesandbox":sesandbox, "fakeroot":0}},
> > > > > > > > "test": {"cmd":ebuild_sh, "args":{"droppriv":droppriv,
> > > > > > > > "free":nosandbox, "sesandbox":sesandbox, "fakeroot":0}},
> > > > > > > > "install": {"cmd":ebuild_sh, "args":{"droppriv":0,
> > > > > > > > "free":0, "sesandbox":sesandbox, "fakeroot":fakeroot}},
> > > > > > > > +"instprep": {"cmd":misc_sh, "args":{"droppriv":0,
> > > > > > > > "free":0, "sesandbox":sesandbox, "fakeroot":fakeroot}},
> > > > > > > > "rpm": {"cmd":misc_sh, "args":{"droppriv":0,
> > > > > > > > "free":0, "sesandbox":0, "fakeroot":fakeroot}},
> > > > > > > > "package": {"cmd":misc_sh, "args":{"droppriv":0,
> > > > > > > > "free":0, "sesandbox":0, "fakeroot":fakeroot}},
> > > > > > > > }
> > > > > > >
> > > > > > > The feature seems find but the instprep phase is not needed if we
> > > > > > > use
> > > > > > > MiscFunctionsProcess instead of EbuildPhase. Here's a list of odd
> > > > > > > things
> > > > > > > about the instprep ebuild phase implementation as it is:
> > > > > > >
> > > > > > > 1) You can call instprep explicitly via the ebuild(1) command,
> > > > > > > but I
> > > > > > > doubt that we really want or need to allow that.
> > > > > >
> > > > > > I was actually wondering about that. I suppose it might be useful
> > > > > > for
> > > > > > debugging purposes.
> >
> > Ok, I've rechecked it and ebuild(1) currently rejects it as not a valid
> > phase name.
>
> I have your patches applied, and for me it treats 'instprep' as a valid
> phase which I'm able to call. It also lists it in the list of available
> phases if I choose an invalid phase named 'foo', since 'instprep' is
> listed in validcommands:
>
> !!! doebuild: 'foo' is not one of the following valid commands:
> !!! clean cleanrmcompileconfig configure depend
> !!! digest fetch fetchall help info install
> !!! instprep manifest merge nofetchpackagepostinst
> !!! postrm preinstprepareprerm pretendqmerge
> !!! rpmsetup test unmergeunpack
>
> > So we can either ignore the problem until somebody finds it
> > useful to call it directly, or work on it. Would the existing patch be
> > ok, provided that ebuild(1) doesn't allow calling prepinst manually?
>
> Lets clarify our intention here, since apparently it works for me but
> not for you when you tested. I think it's fine as-is, but let's make
> sure that the behavior is what we collectively intend.
My mistake. I typed 'prepinst' instead of 'instprep' ;-).
>
> > > Sure. In fact, we could split out separate instcompress and inststrip
> > > phases. I suppose we could split instprep into multiple phases in the
> > > future, if we find a use for it.
> > >
> > > > > > However, since I wasn't able to figure out how to
> > > > > > fully integrate it with the phase machinery, I went for the minimal
> > > > > > set
> > > > > > of code that wouldn't be definitive either way.
> > > > > >
> > > > > > > 2) Unlinke other ebuild phases, the doebuild function doesn't
> > > > > > > created a
> > > > > > > .instpreped file to indicate when the instprep phase has executed.
