[gentoo-dev] [PATCH] linux-mod.eclass: Support module compression
The Linux kernel supports the compression of modules utilizing GZIP, XZ
and ZSTD. Add code into linux-mod.eclass to support this for out of
tree modules utilizing the compression binary specified in the kernel
config.
Note that if the binary which provides the compression is not present on
the system the kernel would have failed to build with an error
indicating the missing binaries name.
Signed-off-by: Mike Pagano
---
eclass/linux-mod.eclass | 17 -
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index 6a820371b..b7c13cbf7 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -711,7 +711,22 @@ linux-mod_src_install() {
einfo "Installing ${modulename} module"
cd "${objdir}" || die "${objdir} does not exist"
insinto "${INSTALL_MOD_PATH}"/lib/modules/${KV_FULL}/${libdir}
- doins ${modulename}.${KV_OBJ} || die "doins ${modulename}.${KV_OBJ}
failed"
+
+ # check here for CONFIG_MODULE_COMPRESS_
(NONE, GZIP, XZ, ZSTD)
+ # and similarily compress the module being built if != NONE.
+
+ if linux_chkconfig_present MODULE_COMPRESS_XZ; then
+ xz ${modulename}.${KV_OBJ}
+ doins ${modulename}.${KV_OBJ}.xz || die "doins
${modulename}.${KV_OBJ}.xz failed"
+ elif linux_chkconfig_present MODULE_COMPRESS_GZIP; then
+ gzip ${modulename}.${KV_OBJ}
+ doins ${modulename}.${KV_OBJ}.gz || die "doins
${modulename}.${KV_OBJ}.gz failed"
+ elif linux_chkconfig_present MODULE_COMPRESS_ZSTD; then
+ zstd ${modulename}.${KV_OBJ}
+ doins ${modulename}.${KV_OBJ}.zst || die "doins
${modulename}.${KV_OBJ}.zst failed"
+ else
+ doins ${modulename}.${KV_OBJ} || die "doins
${modulename}.${KV_OBJ} failed"
+ fi
cd "${OLDPWD}"
generate_modulesd "${objdir}/${modulename}"
--
2.35.1
OpenPGP_signature
Description: OpenPGP digital signature
[gentoo-dev] [PATCH] python-r1.eclass: Warn about direct foreach_impl calls in d-r1
Warn if python_foreach_impl is called directly from within a phase
function that is already covered by a distutils-r1 sub-phase.
Signed-off-by: Michał Górny
---
eclass/distutils-r1.eclass | 1 +
eclass/python-r1.eclass| 18 ++
2 files changed, 19 insertions(+)
diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 0e069597b3ab..0696b096b3c3 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1750,6 +1750,7 @@ _distutils-r1_run_foreach_impl() {
set -- distutils-r1_run_phase "${@}"
if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
+ local _DISTUTILS_CALLING_FOREACH_IMPL=1
python_foreach_impl "${@}"
else
if [[ ! ${EPYTHON} ]]; then
diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 2f8cfa0316d9..3471e17bdde6 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -625,6 +625,24 @@ _python_multibuild_wrapper() {
python_foreach_impl() {
debug-print-function ${FUNCNAME} "${@}"
+ if [[ ${_DISTUTILS_R1} ]]; then
+ if has "${EBUILD_PHASE}" prepare configure compile test install
&&
+ [[ ! ${_DISTUTILS_CALLING_FOREACH_IMPL} &&
+ ! ${_DISTUTILS_FOREACH_IMPL_WARNED} ]]
+ then
+ eqawarn "python_foreach_impl has been called directly
while using distutils-r1."
+ eqawarn "Please redefine python_*() phase functions to
meet your expectations"
+ eqawarn "instead."
+ _DISTUTILS_FOREACH_IMPL_WARNED=1
+
+ if ! has "${EAPI}" 6 7 8; then
+ die "Calling python_foreach_impl from
distutils-r1 is banned in EAPI ${EAPI}"
+ fi
+ fi
+ # undo the eclass-set value to catch nested calls
+ local _DISTUTILS_CALLING_FOREACH_IMPL=
+ fi
+
local MULTIBUILD_VARIANTS
_python_obtain_impls
--
2.35.1
[gentoo-dev] [PATCH] distutils-r1.eclass: Add assertions for bindir correctness
The eclass code in distutils-r1_python_install makes some assumptions
specific to _distutils-r1_post_python_compile being called,
and scriptdir not being modified since. Make them more explicit by:
1) explicitly removing the files that we expect to have been created,
2) verifying that both the copied and the original scriptdir have
the same list of files.
Signed-off-by: Michał Górny
---
eclass/distutils-r1.eclass | 24 +++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index f3d224a51224..0e069597b3ab 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1514,11 +1514,33 @@ distutils-r1_python_install() {
local merge_root=
if [[ ${DISTUTILS_USE_PEP517} ]]; then
local root=${BUILD_DIR}/install
+ local wrapped_scriptdir=${root}$(python_get_scriptdir)
+
+ # we are assuming that _distutils-r1_post_python_compile
+ # has been called and ${root} has not been altered since
+ # let's explicitly verify these assumptions
+
+ # remove files that we've created explicitly
+ rm
"${root}${scriptdir}"/{"${EPYTHON}",python3,python,pyvenv.cfg} || die
+ # verify that scriptdir & wrapped_scriptdir both contain
+ # the same files
+ local ck1=$(
+ { cd "${root}${scriptdir}" && find . -type f -print0; }
|
+ sort -z | cksum
+ )
+ local ck2=$(
+ { cd "${wrapped_scriptdir}" 2>/dev/null &&
+ find . -type f -print0; } |
+ sort -z | cksum
+ )
+ if [[ ${ck1} != ${ck2} ]]; then
+ die "File lists for ${root}${scriptdir} and
${wrapped_scriptdir} differ"
+ fi
+
# remove the altered bindir, executables from the package
# are already in scriptdir
rm -r "${root}${scriptdir}" || die
if [[ ${DISTUTILS_SINGLE_IMPL} ]]; then
- local wrapped_scriptdir=${root}$(python_get_scriptdir)
if [[ -d ${wrapped_scriptdir} ]]; then
mv "${wrapped_scriptdir}" "${root}${scriptdir}"
|| die
fi
--
2.35.1
Re: [gentoo-dev] About EGO_SUM
On Thu, Jun 09, 2022 at 07:49:04PM +0200, Sebastian Pipping wrote: > On 08.06.22 22:42, Robin H. Johnson wrote: > > EGO_SUM vs dependency tarballs: > > [..] > > - EGO_SUM is verifiable/reproducible from Upstream Go systems > > Let's be explicit, there is a _security_ threat here: as a user of an > ebuild, dependency tarballs now take effort in manual review just to > confirm that the content full matches its supposed list of ingredients. > They are the perfect place to hide malicious code in plain sight. Now > with dependency tarballs, there is a new layer that by design will > likely be chronically under-audited. It gives me shivers, frankly. > Previously with a manifest and upstream-only URLs, only upstream can add > malicious code, not downstream in Gentoo. There are many packages in ::gentoo that use tarballs of patches written and hosted by Gentoo developers, or tarballs of source code generated by developers themselves. A (very) rough grep shows this is very prevalent: ~/gentoo/gentoo $ grep -r SRC_URI.*dev.gentoo.org | wc -l 2845 So this problem isn't really new. Users are required to trust Gentoo packagers that we don't do naughty things to the source code, more or less just like any other distribution. signature.asc Description: PGP signature
Re: [gentoo-dev] About EGO_SUM
On 08.06.22 22:42, Robin H. Johnson wrote: EGO_SUM vs dependency tarballs: [..] - EGO_SUM is verifiable/reproducible from Upstream Go systems Let's be explicit, there is a _security_ threat here: as a user of an ebuild, dependency tarballs now take effort in manual review just to confirm that the content full matches its supposed list of ingredients. They are the perfect place to hide malicious code in plain sight. Now with dependency tarballs, there is a new layer that by design will likely be chronically under-audited. It gives me shivers, frankly. Previously with a manifest and upstream-only URLs, only upstream can add malicious code, not downstream in Gentoo. Best Sebastian
[gentoo-dev] [PATCH] java-vm-2.eclass: add BDEPEND="app-arch/unzip"
Closes: https://bugs.gentoo.org/850415
Reported-by: Joonas Niilola
Signed-off-by: Florian Schmaus
---
eclass/java-vm-2.eclass | 5 +
1 file changed, 5 insertions(+)
diff --git a/eclass/java-vm-2.eclass b/eclass/java-vm-2.eclass
index 7c63e63fad76..c0aea3e9ea4a 100644
--- a/eclass/java-vm-2.eclass
+++ b/eclass/java-vm-2.eclass
@@ -24,6 +24,11 @@ RDEPEND="
app-eselect/eselect-java
"
DEPEND="${RDEPEND}"
+BDEPEND="app-arch/unzip"
+
+if [[ ${EAPI} == 6 ]]; then
+ DEPEND+="${BDEPEND}"
+fi
export WANT_JAVA_CONFIG=2
--
2.35.1
[gentoo-dev] Re: About EGO_SUM
* "Robin H. Johnson" : Wrote on Wed, 8 Jun 2022 20:42:48 +: > EGO_SUM vs dependency tarballs: > - bloats ebuilds > - bloats Manifests > - bloats metadata/md5-cache/ (SRC_URI etc) > - doesn't bloat mirrors with gentoo-unique distfiles > - EGO_SUM is verifiable/reproducible from Upstream Go systems > - less downloads on upgrades (only changed Go deps, not entire dep tarballs) > > EGO_SUM data right now adds, to every user's system: > - 2.6MB of text to ebuilds (340k after de-dupe) > - 7MB of text to Manifests (2M after de-dupe) > - 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to > calc deduped amount here) > On the server side: > - The sum total of Go distfiles mirrored on Gentoo mirrors right now > is only 3.4GB. > - less downloads > > Dependency tarballs: > - Right now ~15GiB on each mirror, plus storage of the primary copy > somewhere (dev.g.o right now, but not great) > - Conservatively if the remaining EGO_SUM packages converted to Dep > tarballs, it would need another 8GB each of primary location and > mirrors. > - larger downloads for users who DO want to upgrade a Go package (all > new deps tarball even if only one or two deps changed) > - must be preserved much longer, unless we can introduce a guaranteed > way to regenerate them for any prior ebuild. > > I was trying to introduce a third option, but I haven't had the time to > write an entire GLEP. > > The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries > to move just the metadata out of the tree, in a way that can be > regenerated (specifically, a 1:1 reproducible creation from a given go.sum). > It DOES need to contain slightly more data than the present Manifest, > specifically a full SRC_URI entry for each file (upstream URI plus what > to rename it to on Gentoo side) > > The 2nd-level Manifest would be listed as SRC_URI, and be handled in > src_fetch/src_unpack. Download & verify the extra distfiles, against the > Manifest checksum data (and for Golang against go.sum checksums). > > The Portage mirrordist code needs the most work in this case, as it > would need to fetch the 2nd-level Manifests so it can populate Gentoo > mastermirror with the distfiles mirrored from upstream. > > The storage costs for the proposed idea: > - same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are > mirrored 1:1 content, just different naming) > - Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it > could be split more or shared between some ebuilds/packages) > - Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package > - Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild. > - metadata/md5-cache: 1 src_uri line! > - mirrors: add the Metadata-Manifest [Without claiming to have fully understood the proposal above: around Apr 15th 22 I tried suggesting to WilliamH on IRC that perhaps portage should implement the dirhash approach that go has taken to solve the problem of upstream sources when they invented go.sum. from hash.go in sources go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go // Hash1 is "h1:" followed by the base64-encoded SHA-256 hash of a summary prepared as if by the Unix command:find . -type f | sort | sha256sum loosely speaking the "manifest" could publish this dirhash of contents of go-mod/cache (which would have been bundled in the -deps.tar.xz) The immediate motivation was to avoid the network when I already had the sources locally: instead of downloading a -deps.tar.xz I could create it locally and dump it in distdir. portage would check the (hypothetically) published dirhash and let it through. the local timestamps and uid in my tarball and the upstream tarball wouldn't upset it. One unchecked assumption is that go-mod/cache can be recreated by unpacking sources. If so then with a notion of a "second level manifest" (the equivalent of go.sum) the contents can be assembled without having to store or download the actual -deps tarball. I didn't get very far in convincing WilliamH of my need so I dropped the idea. (I'm not sure if I'm being any clearer, if I'm missing something, do let me know)
