Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Joonas Niilola wrote: Hey, I'll admit I didn't read everything, but I just want to point out you may not have to edit ebuilds at all. If xz-utils is package.provided portage should ignore the dependency without you removing the dep from an ebuild. Then you can utilize /etc/portage/patches to apply any patches and finally try using EXTRA_ECONF and MYMESONARGS to override configure options via package.env. -- juippis Hi Joonas, The local ebuilds in the guide were not created because of the xz-utils dep. If you search through ebuilds in the tree there are hundreds of packages that specify xz-utils as a hard dep, so yes, as you say, package.provided takes care of all of then. No, the ebuilds were needed for various customisations to build arguments. However, the dev-libs/libxmlb ebuild is no longer needed as, since I wrote the guide, libxmlb 0.3.17, which makes liblzma.so.5 dep optional, is now in Gentoo, thanks whoever added that :-) You might be able to dispense with the need for the separate net-mail/dovecot ebuild by using EXTRA_ECONF, as you say. However, AFAICS local dev-lang/python ebuilds are unavoidable, unfortunately, you'll see why if you look at the diffs for them in my guide. It would be wonderful if dev-lang/python made its liblzma dep optional. It would be a simple change to the ebuild. However, I suspect the developers might feel that *not* depending on liblzma.so.5 is unsupported because it results in Gemato failing due to lack of support in core python for liblzma. The only way around that issue I can see is for Gemato to instead use /usr/bin/xz like the rest of Portage does. If that were to happen then dev-lang/python could be modified to respect -lzma and I can't see that anything significant in Gentoo would miss it. Then if there are any dev-python packages that need liblzma in core python either presently or in future (I've not encountered any yet) then of course they would just need a hard dependency on dev-lang/python with lzma USE flag set. I now have many Gentoo systems running without xz-utils installed (using my wrapper script from the guide) and I've not had a single issue anywhere, everything working perfectly, so I'm delighted that it has been possible :-) Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 6.4.2024 14.57, Eddie Chapman wrote:
>
> --- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> +++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> @@ -43,7 +43,6 @@
>
> DEPEND="
> app-arch/bzip2
> - app-arch/xz-utils
> dev-libs/icu:=
> dev-libs/openssl:0=
> sys-libs/zlib:=
> @@ -126,7 +125,7 @@
> --disable-rpath \
> --with-bzlib \
> --without-libbsd \
> - --with-lzma \
> + --without-lzma \
> --with-icu \
> --with-ssl \
> --with-zlib \
>
> --- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild
> @@ -179,6 +179,7 @@
> # Avoid as many dependencies as possible for the cross build.
> cat >> Makefile <<-EOF || die
> MODULE_NIS_STATE=disabled
> + MODULE__LZMA_STATE=disabled
> MODULE__DBM_STATE=disabled
> MODULE__GDBM_STATE=disabled
> MODULE__DBM_STATE=disabled
> @@ -328,7 +329,7 @@
> fi
>
> # force-disable modules we don't want built
> - local disable_modules=( NIS )
> + local disable_modules=( NIS _LZMA )
> use gdbm || disable_modules+=( _GDBM _DBM )
> use sqlite || disable_modules+=( _SQLITE3 )
> use ssl || disable_modules+=( _HASHLIB _SSL )
>
>
> --- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild
> @@ -177,6 +177,7 @@
> cat > Modules/Setup.local <<-EOF || die
> *disabled*
> nis
> + _lzma
> _dbm _gdbm
> _sqlite3
> _hashlib _ssl
> @@ -299,6 +300,7 @@
> cat > Modules/Setup.local <<-EOF || die
> *disabled*
> nis
> + _lzma
> $(usev !gdbm '_gdbm _dbm')
> $(usev !sqlite '_sqlite3')
> $(usev !ssl '_hashlib _ssl')
>
>
> Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to
> upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15.
>
> I also needed to apply a very recent patch from upstream, from this
> commit, which makes LZMA support optional:
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639
>
> and I needed to make some small changes to the ebuild.
>
> So this is what you need to do at the time of writing (6th April 2024):
>
> 1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into
> your local ebuilds directory.
>
> 2. Rename the ebuild file from libxmlb-0.3.14.ebuild to
> libxmlb-0.3.15.ebuild
>
> 3. Download the raw patch, you can use this link:
>
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch
> rename it to:
> libxmlb-0.3.15-make_lzma_optional.patch
> and place it in the local "files" directory.
>
> 4. Modify the new ebuild according to the diff below. Then just rebuild it.
>
> --- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild
> +++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild
> @@ -14,15 +14,15 @@
> SLOT="0/2" # libxmlb.so version
>
> KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86"
> -IUSE="doc introspection stemmer test +zstd"
> +IUSE="doc introspection -lzma stemmer test +zstd"
>
> RESTRICT="!test? ( test )"
>
> RDEPEND="
> - app-arch/xz-utils
> dev-libs/glib:2
> sys-apps/util-linux
> stemmer? ( dev-libs/snowball-stemmer:= )
> + lzma? ( app-arch/xz-utils:= )
> zstd? ( app-arch/zstd:= )
> "
>
> @@ -43,6 +43,7 @@
>
> PATCHES=(
> "${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch
> + "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch
> )
>
> python_check_deps() {
> @@ -60,6 +61,7 @@
> $(meson_use stemmer)
> $(meson_use test tests)
> $(meson_use zstd)
> + $(meson_feature lzma)
> )
> meson_src_configure
> }
Hey,
I'll admit I didn't read everything, but I just want to point out you
may not have to edit ebuilds at all. If xz-utils is package.provided
portage should ignore the dependency without you removing the dep from
an ebuild. Then you can utilize /etc/portage/patches to apply any
patches and finally try using EXTRA_ECONF and MYMESONARGS to override
configure options via package.env.
-- juippis
OpenPGP_signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Sam James wrote: > Eddie Chapman writes: >> Below is a guide I've written to removing app-arch/xz-utils in case >> anyone else wants to do so. Attached is the current version of the Bash >> wrapper script I now use in place of /usr/bin/xz >> >> Comments, corrections on anything technical in the guide or script are >> welcome, apart from flames about how this is ridiculous and unnecessary >> :-). > > For an experiment I'm doing (distinct from trying to purge xz-utils, > just verification work), I've packaged the following: * app-arch/gxz (pure > Go impl.) > * app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip > was an unofficial port) > > You might find those useful too. That's fantastic. I wrote about p7zip vs. upstream 7zip in another mail in this thread and was intending to create a local ebuild for 7zip soon but won't have to know it's in tree :-) > At a glance, it appears https://github.com/fpgaminer/rust-lzma and > https://github.com/gendx/lzma-rs don't provide executables - just a > library - so I didn't bother looking further. >> >> Guide to removing xz utils on a Gentoo system >> >> [...] >> There is one significant thing that breaks, which is Gemato >> (app-portage/gemato). Gemato requires lzma support in core python in >> order to do GPG signature verification. This means you will have to say >> goodbye (for now) to verifying upstream GPG signatures on distfiles, and >> verification of Portage metadata after doing an emerge --sync. These >> features have been added to Portage relatively recently (2022?) so are >> "nice to have", without them your system is just less > > No.. much older. It was introduced around the time of the github mirror > being hacked. It's not just theatre! > > Like, this is very much NOT hypothetical. Thanks, couldn't remember when it was. > It's not just about metadata, it's about the ebuilds if using rsync, or > the whole git checkout if using git. Completely agree with you that this was a great feature to be added from a security point of view. Without it there was still a level of trust, however small, that could be placed in the choice of mirror. But there's no doubt gpg sigs of repo data are order of magnitudes better, so yes it was a little unjust to describe it as only "nice to have". But in the current situation I personally consider it so critically important to get rid of xz utils from my systems that a short, temporary period of not having this while switching to another method of verification I consider an acceptable tradeoff (side note to anyone reading: yes I know how at odds I am with the rest of the world on this, it has now been argued to death in this thread so for anyone thinking about replying about that, maybe lets do everyone a favour, agree to disagree, and move on :-) ) >> hardened, but still with the very high level of security that Gentoo >> systems have has always had prior to these features, in my opinion. >> Personally I can live without them for now. Verifying hashes >> in Manifest files still works fine and that's the main thing. You may >> disagree in which case, well, don't do this then. I'm going to figure >> out an alternative way I can verify Portage metadata soon, as there are >> other ways if you are creative. > > See grobian's reply which should help. > > >> [...] >> - Portage binary packages: You cannot use xz compression if you create >> Portage binary packages. You will need to use one of bzip2, gzip, >> lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz >> (if that is what you were using, or is it the default?). I have >> > > zstd is the default for "new" installs (since a few years ago), yeah. > >> [...] >> - sys-apps/fwupd might stop working properly (though it will still >> build fine) due to what you have to change with dev-libs/libxmlb below. >> I'm not sure as I haven't checked yet, I just suspect it >> will. So bear that in mind if you need to rely on sys-apps/fwupd at the >> moment. But this "might" is temporary, upstream has now decided to make >> lzma optional, so this will trickle down to Gentoo soon. > > Just for completeness, this is > https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/. > Thanks for all the useful additions of info :-) cheers, Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Fabian Groffen wrote: > If you just want to verify signatures and manifests after sync, > qmanifest from portage-utils can help you do this. > > Thanks, > Fabian Thanks for the pointer, and I see you are one of the authors, thanks for writing a very useful tool!
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eddie Chapman writes: > On 04/04/2024 15:24, Eddie Chapman wrote: >> Since there appears to be some interest I'll put together a single email >> to the list later today detailing everything, as I needed to do more >> things overall in addition to replacing /usr/bin/xz. > > Below is a guide I've written to removing app-arch/xz-utils in case > anyone else wants to do so. Attached is the current version of the > Bash wrapper script I now use in place of /usr/bin/xz > > Comments, corrections on anything technical in the guide or script are > welcome, apart from flames about how this is ridiculous and > unnecessary :-). For an experiment I'm doing (distinct from trying to purge xz-utils, just verification work), I've packaged the following: * app-arch/gxz (pure Go impl.) * app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip was an unofficial port) You might find those useful too. At a glance, it appears https://github.com/fpgaminer/rust-lzma and https://github.com/gendx/lzma-rs don't provide executables - just a library - so I didn't bother looking further. > > Best wishes, > Eddie > > > Guide to removing xz utils on a Gentoo system > > [...] > There is one significant thing that breaks, which is Gemato > (app-portage/gemato). Gemato requires lzma support in core python in > order to do GPG signature verification. This means you will have to > say goodbye (for now) to verifying upstream GPG signatures on > distfiles, and verification of Portage metadata after doing an emerge > --sync. These features have been added to Portage relatively recently > (2022?) so are "nice to have", without them your system is just less No.. much older. It was introduced around the time of the github mirror being hacked. It's not just theatre! Like, this is very much NOT hypothetical. It's not just about metadata, it's about the ebuilds if using rsync, or the whole git checkout if using git. > hardened, but still with the very high level of security that Gentoo > systems have has always had prior to these features, in my > opinion. Personally I can live without them for now. Verifying hashes > in Manifest files still works fine and that's the main thing. You may > disagree in which case, well, don't do this then. I'm going to figure > out an alternative way I can verify Portage metadata soon, as there > are other ways if you are creative. See grobian's reply which should help. > [...] > - Portage binary packages: You cannot use xz compression if you create > Portage binary packages. You will need to use one of bzip2, gzip, > lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of > xz (if that is what you were using, or is it the default?). I have zstd is the default for "new" installs (since a few years ago), yeah. > [...] > - sys-apps/fwupd might stop working properly (though it will still > build fine) due to what you have to change with dev-libs/libxmlb > below. I'm not sure as I haven't checked yet, I just suspect it > will. So bear that in mind if you need to rely on sys-apps/fwupd at > the moment. But this "might" is temporary, upstream has now decided > to make lzma optional, so this will trickle down to Gentoo soon. Just for completeness, this is https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/. > [...]
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 06-04-2024 12:57:23 +0100, Eddie Chapman wrote: > There is one significant thing that breaks, which is Gemato > (app-portage/gemato). Gemato requires lzma support in core python in > order to do GPG signature verification. This means you will have to say > goodbye (for now) to verifying upstream GPG signatures on distfiles, and > verification of Portage metadata after doing an emerge --sync. These > features have been added to Portage relatively recently (2022?) so are > "nice to have", without them your system is just less hardened, but > still with the very high level of security that Gentoo systems have has > always had prior to these features, in my opinion. Personally I can live > without them for now. Verifying hashes in Manifest files still works > fine and that's the main thing. You may disagree in which case, well, > don't do this then. I'm going to figure out an alternative way I can > verify Portage metadata soon, as there are other ways if you are creative. If you just want to verify signatures and manifests after sync, qmanifest from portage-utils can help you do this. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 2024.04.06 12:57, Eddie Chapman wrote: > On 04/04/2024 15:24, Eddie Chapman wrote: > > Since there appears to be some interest I'll put together a single > email > > to the list later today detailing everything, as I needed to do more > > things overall in addition to replacing /usr/bin/xz. > > Below is a guide I've written to removing app-arch/xz-utils in case > anyone else wants to do so. Attached is the current version of the > Bash > wrapper script I now use in place of /usr/bin/xz > > Comments, corrections on anything technical in the guide or script are > > welcome, apart from flames about how this is ridiculous and > unnecessary :-). > > Best wishes, > Eddie > >[snip method] "Because I can" is a good enough reason to do anything with Gentoo. -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods arm64 pgpwyJRBN8PRl.pgp Description: PGP signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
> On Sat, 06 Apr 2024, Eddie Chapman wrote: > [...] this is ridiculous and unnecessary :-). Indeed. SCNR, Ulrich
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 04/04/2024 15:24, Eddie Chapman wrote: Since there appears to be some interest I'll put together a single email to the list later today detailing everything, as I needed to do more things overall in addition to replacing /usr/bin/xz. Below is a guide I've written to removing app-arch/xz-utils in case anyone else wants to do so. Attached is the current version of the Bash wrapper script I now use in place of /usr/bin/xz Comments, corrections on anything technical in the guide or script are welcome, apart from flames about how this is ridiculous and unnecessary :-). Best wishes, Eddie Guide to removing xz utils on a Gentoo system === Introduction === This guide is for people who wish to remove xz utils (app-arch/xz-utils) from a Gentoo system. I've been able to remove xz utils from two Gentoo workstations with 2412 packages and KDE 5.x as the desktop, and it has not been painful at all. I've gone on to remove it from several Gentoo server systems without any pain. These are all SElinux systems. In this guide we replace app-arch/xz-utils with app-arch/p7zip which will do all the work of uncompressing xz distfiles for Portage going forward. It works perfectly fine for that right now. I've written a bash wrapper script which is designed to be installed as /usr/bin/xz, which is referred to in the instructions below. It is attached to this email as xz.txt. It tries to takes care of decompressing .xz files transparently whenever Portage runs /usr/bin/xz, by behaving like it but using app-arch/p7zip in the background. You will need it if you want to get rid of app-arch/xz-utils. But don't blindly use it, check it yourself first of course. If you don't like it you will either need to write your own script, or hack emerge/Portage in various places to use something else to decompress xz files. You're mileage may vary with any of this, proceed at your own risk, don't blame me if you break your system or lose data. === Warnings / Caveats / Breakages === Before you do this, you should identify whether you have applications or scripts which use the Tukaani xz utils, or that link against liblzma.so.5. This could include non-Gentoo apps or scripts you run which call any of the xz utils (xz, unxz, xzgrep|xzegrep|xzfgrep, xzcat, xzcmp, xzdec, xzdiff, lzma, unlzma, lzgrep|lzegrep|lzfgrep, lzmainfo, lzmadec, lzcmp, lzdiff, lzcat). Those programs will all be gone, so you should not do this if you want or need them and cannot use alternatives. 99% of packages in Gentoo work fine without xz utils, it's just that some might optionally link against liblzma.so.5 in order to provide support for xz (de)compression along with other algorithms. We will rebuild those packages so they don't link against liblzma.so.5 anymore. xx utils is a relative newcomer to the Linux/OSS/GNU world so you will find there aren't any low level system packages that absolutely need it to do their main job. You are highly unlikely to render your system completely unbootable doing this. But removing it does carry some risk. You might discover along the way there is some application you have installed that cannot function without xz utils. You might just have to uninstall it and find an alternative, if the situation cannot be resolved by creating your own custom ebuild and tweaking configure/meson options. But worst case if you have to uninstall a package and other packages depend on it, you might have to remove them too, and I'm sure you know how that remove list can potentially turn into a long one once all deps are worked out. You will lose some things. I've had to uninstall the following two packages for now: media-gfx/gimp kde-apps/ark (and kde-apps/kdeutils-meta which depends on it) (I'll probably figure out later how to coax them into working without xz. There might even be upstream updates soon that make xz optional, who knows. I'll also need to add to my world file at some point everything that was in kde-apps/kdeutils-meta.) If you run another desktop (e.g. Gnome) I've no idea what might or might not need xz utils. The situation with your desktop environment may be worse, more painful, or impossible. You will lose lzma support in the core Python language (dev-lang/python) in 3.x versions and higher (not sure when exactly support was introduced but 2.7 does not have it, 3.11 & 3.12 do), so if you have python scripts that happen to need that, well, they will definitely throw a big error after this :-) But I was able to rebuild the 179 dev-python packages on my workstations and everything in app-portage and none of them complained. I've been able to go on and do plenty of rebuilding with Portage after this without any problem, so core Python functionality in Gentoo is fine (although see next paragraph about Gemato). There is one significant thing that breaks, which is Gemato (app-portage/gemato). Gemato requires lzma support in core python in order
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Sam James wrote: > Eli Schwartz writes: > >> On 4/3/24 11:30 AM, Eddie Chapman wrote: >> >>> Just to report I've been able to remove app-arch/xz-utils from my own >>> workstation, with 2412 packages installed and running kde. I'm going >>> to roll it out to my other gentoo systems which have a lot less stuff >>> on them so am confident will be fine. It's not completely trivial but >>> not as difficult as I imagined it to be, certainly something an >>> advance Gentoo user could do if they wanted, with instructions. It >>> does involve a relatively small hack and functionality previously >>> provided by xz-utils is replaced by app-arch/p7zip. >> >> I'd just like to clarify my previous posts: what you're describing here >> is neat and productive and valid to my eyes. Actually, I wish this had >> been the topic of the *first* post in this thread. :) > > Completely agreed. We just prefer shorter text and focusing on technical > changes. > > This sounds fun! > > >> [...] >> > > thanks, sam Well, I didn't think my first post was so bad, but OK, I'll take that criticism onboard and in future will think more about how I bring something to this list, and will take into account what you and Eli have said. Thanks for sharing your thoughts above, both of you, in a constructive way. regards, Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eli Schwartz wrote: > On 4/3/24 11:30 AM, Eddie Chapman wrote: > >> Just to report I've been able to remove app-arch/xz-utils from my own >> workstation, with 2412 packages installed and running kde. I'm going to >> roll it out to my other gentoo systems which have a lot less stuff on >> them so am confident will be fine. It's not completely trivial but not >> as difficult as I imagined it to be, certainly something an advance >> Gentoo >> user could do if they wanted, with instructions. It does involve a >> relatively small hack and functionality previously provided by xz-utils >> is replaced by app-arch/p7zip. > > > I'd just like to clarify my previous posts: what you're describing here > is neat and productive and valid to my eyes. Actually, I wish this had been > the topic of the *first* post in this thread. :) > > Replacing implementations has several great uses. There's some prior art > in make.conf, but it doesn't go far enough: > > PORTAGE_BZIP2_COMMAND > BINPKG_COMPRESS > BINPKG_COMPRESS_FLAGS > > > Disregarding the security component entirely, one might wish to use pixz > or pigz instead of the default programs. Why not 7zip as well? One of my emails elsewhere in this thread (easy to miss in a long thread, I know) I discussed pixz, pigz and 7zip. The former two were not suitable for me as both rely on xz utils. However I will probably switch from p7zip to the latest upstream 7zip in the near future, for reasons discussed in that email. > In terms of security, this suggests an easy and simple way both to allow > users to depclean xz-utils without sacrificing the ability to install > packages using *.tar.xz sources, and for Gentoo to roll out an update that > would do this distribution-wide if necessary via a trivial configuration > change. > > https://dev.gentoo.org/~ulm/pms/head/pms.html#section-12.3.15 may need > updating to allow this. But it seems very valid to propose doing exactly > that. I am not sure why it specifies e.g. "must ensure that GNU gzip" with > heavy ties to implementations, when it doesn't specify such for > compression. That would certainly be a nice improvement for all users if it were ever to come to pass. > I'm guessing what you did was override/hook the unpack phase helper > function and divert it to 7zip instead. ;) It would be interesting to have > actual hooks for that instead. Yes it is in the unpack phase where emerge calls /usr/bin/xz mostly. In fact I didn't have to touch emerge/portage, it was more crude, I uninstalled app-arch/xz-utils (and put it in /etc/portage/profile/package.provided) and replaced /usr/bin/xz with a bash script to behave like what the unpack phase was expecting, but using /usr/lib64/p7zip/7za to do the decompression. However, I need to do some more work on this "wrapper" (though it's more than a wrapper) as I found one package where xz is called from the install phase and my script doesn't handle that yet it just throws an error for anything other than the unpack phase case (which is 99.9 percent of packages). But ultimately doing something along the lines of what you suggest instead would of course be much better than this dirty hack (though it works just fine for me for now). Since there appears to be some interest I'll put together a single email to the list later today detailing everything, as I needed to do more things overall in addition to replacing /usr/bin/xz.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
If that’s working, it could at least be on an user personnal page on the wiki as well. Le 04/04/2024 à 10:32, Sam James a écrit : Eli Schwartz writes: On 4/3/24 11:30 AM, Eddie Chapman wrote: Just to report I've been able to remove app-arch/xz-utils from my own workstation, with 2412 packages installed and running kde. I'm going to roll it out to my other gentoo systems which have a lot less stuff on them so am confident will be fine. It's not completely trivial but not as difficult as I imagined it to be, certainly something an advance Gentoo user could do if they wanted, with instructions. It does involve a relatively small hack and functionality previously provided by xz-utils is replaced by app-arch/p7zip. I'd just like to clarify my previous posts: what you're describing here is neat and productive and valid to my eyes. Actually, I wish this had been the topic of the *first* post in this thread. :) Completely agreed. We just prefer shorter text and focusing on technical changes. This sounds fun! [...] thanks, sam
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eli Schwartz writes: > On 4/3/24 11:30 AM, Eddie Chapman wrote: >> Just to report I've been able to remove app-arch/xz-utils from my own >> workstation, with 2412 packages installed and running kde. I'm going to >> roll it out to my other gentoo systems which have a lot less stuff on them >> so am confident will be fine. It's not completely trivial but not as >> difficult as I imagined it to be, certainly something an advance Gentoo >> user could do if they wanted, with instructions. It does involve a >> relatively small hack and functionality previously provided by xz-utils is >> replaced by app-arch/p7zip. > > > I'd just like to clarify my previous posts: what you're describing here > is neat and productive and valid to my eyes. Actually, I wish this had > been the topic of the *first* post in this thread. :) Completely agreed. We just prefer shorter text and focusing on technical changes. This sounds fun! > [...] thanks, sam
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 4/3/24 11:30 AM, Eddie Chapman wrote: > Just to report I've been able to remove app-arch/xz-utils from my own > workstation, with 2412 packages installed and running kde. I'm going to > roll it out to my other gentoo systems which have a lot less stuff on them > so am confident will be fine. It's not completely trivial but not as > difficult as I imagined it to be, certainly something an advance Gentoo > user could do if they wanted, with instructions. It does involve a > relatively small hack and functionality previously provided by xz-utils is > replaced by app-arch/p7zip. I'd just like to clarify my previous posts: what you're describing here is neat and productive and valid to my eyes. Actually, I wish this had been the topic of the *first* post in this thread. :) Replacing implementations has several great uses. There's some prior art in make.conf, but it doesn't go far enough: PORTAGE_BZIP2_COMMAND BINPKG_COMPRESS BINPKG_COMPRESS_FLAGS Disregarding the security component entirely, one might wish to use pixz or pigz instead of the default programs. Why not 7zip as well? In terms of security, this suggests an easy and simple way both to allow users to depclean xz-utils without sacrificing the ability to install packages using *.tar.xz sources, and for Gentoo to roll out an update that would do this distribution-wide if necessary via a trivial configuration change. https://dev.gentoo.org/~ulm/pms/head/pms.html#section-12.3.15 may need updating to allow this. But it seems very valid to propose doing exactly that. I am not sure why it specifies e.g. "must ensure that GNU gzip" with heavy ties to implementations, when it doesn't specify such for compression. I'm guessing what you did was override/hook the unpack phase helper function and divert it to 7zip instead. ;) It would be interesting to have actual hooks for that instead. -- Eli Schwartz OpenPGP_0x84818A6819AF4A9B.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Wed, 2024-04-03 at 16:30 +0100, Eddie Chapman wrote: > It does involve a > relatively small hack and functionality previously provided by xz-utils is > replaced by app-arch/p7zip. I did the same thing with app-arch/unzip a long time ago. You caught a lot of shit for your post, but I don't think it was out of line. Worst case? You spent a lot of time building a fragile solution to a non-problem that everyone said you were crazy for wanting in the first place. Hi, this is Gentoo, glad to have you.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Just to report I've been able to remove app-arch/xz-utils from my own workstation, with 2412 packages installed and running kde. I'm going to roll it out to my other gentoo systems which have a lot less stuff on them so am confident will be fine. It's not completely trivial but not as difficult as I imagined it to be, certainly something an advance Gentoo user could do if they wanted, with instructions. It does involve a relatively small hack and functionality previously provided by xz-utils is replaced by app-arch/p7zip. I haven't had to give up distfile checksum verifications, everything builds, boots and is working fine, I don't miss it at all. There is some small (at least small for me) functionality you'll lose, but nothing I'll miss. I also had to uninstall 2 desktop packages which I can live without, in order to get there, but I believe even they can be later coaxed into working with some persuasion. I would imagine the majority of Gentoo installations would not miss it unless they do a lot of work daily directly using xz-utils and they particularly like all the different permutations of how it can be run and inserted here and there. Also, it goes without saying if you run any sort of application that absolutely refuses to run without liblzma.so being present and the requirement cannot be compiled out, or you have scripts which rely on liblzma.so provided functionality inside the language they use, then of course this cannot be done. If anyone wants to know the details of how to do it no problem just ask. I won't post if no one asks, to not get on people's nerves here with and this mail getting longer and longer, especially when most here are not interested in doing this and don't believe it is necessary. For all the overworked Gentoo developers I'd like to compliment you for once rather than irritating you; it's only realistically possible to do because of the the powerful distribution you have created, particularly the great choices you've made along the way, that allows the Linux user to do advanced stuff that would just not be realistically achievable to users on the vast majority of other distros. Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 02/04/2024 20:46, Eli Schwartz wrote: On 4/2/24 4:43 AM, Eddie Chapman wrote: Well, they change one thing. It's hard for the security professionals at work to deal with things when they are constantly having to respond to the three-ring circus. This is a complaint I hear very often from the people working at the heart of things. Stop making noise, shut up, we're overworked here and dealing with your "complaints" just adds to our stress. I do understand and sympathise with those feelings, believe me I do, I feel them myself in other contexts. But I hope you understand this is not finding things to nitpick about for the sake of it. Does the Gentoo dev community want people on the "outside" to raise their concerns on their mailing list if those persons feel like said community have got something very wrong, yes or no? If not then put a note on the mailing list page saying "please don't bother us, we're too overworked, just post patches" or something to that effect. I would be delighted to hear reasonable concerns. That's not what I'm referring to by "three-ring circus". What does one have to do with the other? Why is it necessary to claim that based on some sort of vibe check "there is too much compassion going around in our communities, and this must mean that not enough effort is being expended on the technical and cleanup aspects"? I have not made such a claim, I've said I see lots of technical and cleanup aspects. I've only stated the things that *are* happening versus what is not happening at all (literally zilch) and which should be happening, which is efforts towards a solution *not* involving the xz utilities. You say that as though a solution *not* involving the xz utilities is a reasonable takeaway from this scenario. In order to demonstrate that such efforts deserve discussion at all, let alone efforts towards solving it, you first need to prove that: - the xz utilities as shipped by Gentoo are something that should be moved away from - the xz utilities as released in 2022, when the backdoor author had just as much access as you or I -- that is, none, aside for the right to submit patches as suggestions -- are something that should be moved away from You have made no effort to justify either approach aside for claiming that for unidentified reasons you believe this scenario demonstrates that the "apparently innocent maintainer" now has an incentive to "downplay the involvement of the bad actor". If you had, I would be infinitely more interested in what you have to say on the topic. Also, if you had started with such. Reading in between the lines, e.g. "trying desperately to salvage the situation with xz-utils", I suspect you are trying to subtly suggest that any second of time where gentoo hasn't yet removed xz-utils from gentoo as a dead end is "cavalier". Not quite, I've never advocated removing xz-utils at all, more than happy for it to remain for whoever wants to use it. The only reason I started this thread is I'm very unhappy about that fact that it is currently impossible to NOT execute xz utilities on the Gentoo systems I'm responsible for, without heavy customisation. I'm also not demanding anything, let alone demanding anything instantly. If I have please point out where. Thank you for clarifying. I am now just as convinced as I was yesterday, that you are trying to subtly suggest it, but I'm newly convinced that you're also being mendacious about it. "It is cavalier for Gentoo to allow xz-utils as a package in the @system set" is not meaningfully distinct from "it is cavalier for Gentoo to not work to allow me to depclean xz-utils". I understand that you are passionate about your suggestion to make portage not validate distfile hashes. That's incorrect, I've never suggested Portage should not validate distfile hashes. The current behaviour is that validating distfile hashes is something that can be disabled if a user wishes to, and I have no problem with that at all, would not change a thing. I've said that, in order to implement what I have suggested, a user would have to disable it, which is not ideal, but acceptable if the user controls the distfile distribution. And I only suggested that in order to try and make the idea more acceptable by not requiring impractical infra changes that would be needed to generate uncompressed hashes for the Manifests). In other words, you didn't care to find a robust solution, you just want something that you personally can use, and which requires being less secure than using xz-utils. But it's okay! It's not harassing portage devs with unreasonable demands! Because it's *optional*, and by default people would just... use xz-utils. Even though the ***entire premise*** of changing anything here is that xz-utils as shipped by Gentoo is somehow dangerous and users have a valid reason to want to avoid it entirely. If you're going to recommend a solution for users who consider xz-utils to be
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 4/2/24 4:43 AM, Eddie Chapman wrote: >> Well, they change one thing. It's hard for the security professionals at >> work to deal with things when they are constantly having to respond to the >> three-ring circus. > > This is a complaint I hear very often from the people working at the heart > of things. Stop making noise, shut up, we're overworked here and dealing > with your "complaints" just adds to our stress. I do understand and > sympathise with those feelings, believe me I do, I feel them myself in > other contexts. > > But I hope you understand this is not finding things to nitpick about for > the sake of it. Does the Gentoo dev community want people on the "outside" > to raise their concerns on their mailing list if those persons feel like > said community have got something very wrong, yes or no? If not then put a > note on the mailing list page saying "please don't bother us, we're too > overworked, just post patches" or something to that effect. I would be delighted to hear reasonable concerns. That's not what I'm referring to by "three-ring circus". >> What does one have to do with the other? Why is it necessary to claim >> that based on some sort of vibe check "there is too much compassion going >> around in our communities, and this must mean that not enough effort is >> being expended on the technical and cleanup aspects"? > > I have not made such a claim, I've said I see lots of technical and > cleanup aspects. I've only stated the things that *are* happening versus > what is not happening at all (literally zilch) and which should be > happening, which is efforts towards a solution *not* involving the xz > utilities. You say that as though a solution *not* involving the xz utilities is a reasonable takeaway from this scenario. In order to demonstrate that such efforts deserve discussion at all, let alone efforts towards solving it, you first need to prove that: - the xz utilities as shipped by Gentoo are something that should be moved away from - the xz utilities as released in 2022, when the backdoor author had just as much access as you or I -- that is, none, aside for the right to submit patches as suggestions -- are something that should be moved away from You have made no effort to justify either approach aside for claiming that for unidentified reasons you believe this scenario demonstrates that the "apparently innocent maintainer" now has an incentive to "downplay the involvement of the bad actor". If you had, I would be infinitely more interested in what you have to say on the topic. Also, if you had started with such. >> Reading in between the lines, e.g. "trying desperately to salvage the >> situation with xz-utils", I suspect you are trying to subtly suggest that >> any second of time where gentoo hasn't yet removed xz-utils from gentoo as >> a dead end is "cavalier". > > Not quite, I've never advocated removing xz-utils at all, more than happy > for it to remain for whoever wants to use it. The only reason I started > this thread is I'm very unhappy about that fact that it is currently > impossible to NOT execute xz utilities on the Gentoo systems I'm > responsible for, without heavy customisation. > > I'm also not demanding anything, let alone demanding anything instantly. > If I have please point out where. Thank you for clarifying. I am now just as convinced as I was yesterday, that you are trying to subtly suggest it, but I'm newly convinced that you're also being mendacious about it. "It is cavalier for Gentoo to allow xz-utils as a package in the @system set" is not meaningfully distinct from "it is cavalier for Gentoo to not work to allow me to depclean xz-utils". >> I understand that you are passionate about your suggestion to make >> portage not validate distfile hashes. > > That's incorrect, I've never suggested Portage should not validate > distfile hashes. The current behaviour is that validating distfile hashes > is something that can be disabled if a user wishes to, and I have no > problem with that at all, would not change a thing. I've said that, in > order to implement what I have suggested, a user would have to disable it, > which is not ideal, but acceptable if the user controls the distfile > distribution. And I only suggested that in order to try and make the idea > more acceptable by not requiring impractical infra changes that would be > needed to generate uncompressed hashes for the Manifests). In other words, you didn't care to find a robust solution, you just want something that you personally can use, and which requires being less secure than using xz-utils. But it's okay! It's not harassing portage devs with unreasonable demands! Because it's *optional*, and by default people would just... use xz-utils. Even though the ***entire premise*** of changing anything here is that xz-utils as shipped by Gentoo is somehow dangerous and users have a valid reason to want to avoid it entirely. If you're going to
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 01/04/2024 15:56, Azamat Hackimov wrote: There is no problem in the XZ/LZMA format itself as the reference algorithm is not compromised. It's all about trust between developers of application and developers of distribution. If you lost trust to xz-utils's developers, you may use alternatives like app-arch/pxz or app-arch/pixz. I don't see reasons why we should change format instead of changing a tool. Hello Azamat, Yes, I have no issue with the format at all, just with the xz utils project. But I was suggesting to have support for two compression algorithms as an improvement for the future, in case of some unknown other major problem with one of them emerges in future. I suppose kind of a similar reasoning, but not quite the same, that we have BLAKE2B and SHA512 hashes. But there are severe practical problems for Gentoo infra resources with having two of course. Thank you for the pointer to app-arch/pxz and app-arch/pixz. I've had a close look at them both but sadly they are not suitable as they both rely on the xz utils project to do the main work. Once calls the xz exe and the other links against liblzma. I have been looking around a bit since Friday at what true alternatives (no relying on liblzma) there are to just decompress existing XZ/LZMA binaries. There is p7zip which is a command line fork of 7zip that's been around a good while. However in the years since that fork was created the 7zip project themselves have begun doing source code releases with build instructions, with the command line tool apparently working fine. On balance the upstream 7zip actually looks like a better option than p7zip now since p7zip maintenance has stagnated somewhat. On the one hand 7zip is actively developed, of course because of the large Windows userbase, and security fixes would be available immediately when a new release comes about (there were sec issues fixed in 7zip last year for example which didn't make it into p7zip in a timely fashion). But on the other hand most distros have used p7zip and I've only seen Arch and Debian that currently package the 7zip releases, so the latest 7zip releases have had only minimum real world testing and code scrutiny in the Linux world (although it's likely much of the code will still be the same as what it was when p7zip was forked, so in that sense at least a significant portion of the code has had wider testing, in a manner of speaking). Still, I'm not sure about 7zip, doesn't seem ideal. Thomas Gall, elsewhere in this thread, pointed out a pure Rust implementation which is interesting. https://github.com/gendx/lzma-rs The GH page says it supports decompression of "LZMA, LZMA2 and a subset of the .xz file format". If anyone else knows of any other true alternatives please do let me know. I'm currently looking into the feasibility of hacking my Gentoo installations so that .xz distfiles are decompressed during the ebuild process using an alternative implementation, allowing me to get rid of xz utils. Thanks, Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Michał Górny wrote: > On Mon, 2024-04-01 at 08:57 +0100, Eddie Chapman wrote: > >> I stand by and reiterate my view that there is far too much of a >> cavalier attitude towards the matter in general out there including here >> in Gentoo. But not in particular here, it is everywhere where this is >> being discussed at the moment. > > I would like to point out that the xz/sshd issue was primarily a social > one, not a technical one. > > The primary problem in open source today isn't bad code. It's projects > relying on overburdened, burned out maintainers. And on top of that, users > who are complaining, demanding, outright hostile or primarily contributing > by walls of text on a mailing lists, that bring nothing to discussion > except for furthering the burnout of open source developers who are > actually trying to do something. > > Think about that. > > > -- > Best regards, > Michał Górny > I'm sorry for having contributed to your burnout. I have a lot of respect for you personally Michał, the quality of your contributions to Gentoo are outstanding, and have to admit I've often felt a little worried for you with the amount of work you do. I don't know you at all, I hope you don't mind me saying that. Don't worry I think it's quite unlikely I'll bring any new concerns to this list again in future, I'll certainly think twice about it. regards, Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
OK, I said I was done and this is a waste of time for everyone, but if people want to keep the discussion going I'll bite :-) Eli Schwartz wrote: > But also, please keep in mind that 98% of all people on the internet can > do whatever they want and it simply doesn't matter. They are public > commentators at a three-ring circus and their cavalier or panicked > attitudes change nothing. I disagree, think it is very important to have discussions about what the oss/linux community thinks, not just what they do. And I think those discussions do significantly influence what is actually done, whether the "doers" actually realise it or not. > Well, they change one thing. It's hard for the security professionals at > work to deal with things when they are constantly having to respond to the > three-ring circus. This is a complaint I hear very often from the people working at the heart of things. Stop making noise, shut up, we're overworked here and dealing with your "complaints" just adds to our stress. I do understand and sympathise with those feelings, believe me I do, I feel them myself in other contexts. But I hope you understand this is not finding things to nitpick about for the sake of it. Does the Gentoo dev community want people on the "outside" to raise their concerns on their mailing list if those persons feel like said community have got something very wrong, yes or no? If not then put a note on the mailing list page saying "please don't bother us, we're too overworked, just post patches" or something to that effect. > Please stop insulting the work of the people who are working very hard > to analyze and learn about this issue and taking steps to try to mitigate > it... I'm certainly not trying to insult anyone. I've expressed a lot of appreciation for their work. I have *criticised* the prevailing view though. > What does one have to do with the other? Why is it necessary to claim > that based on some sort of vibe check "there is too much compassion going > around in our communities, and this must mean that not enough effort is > being expended on the technical and cleanup aspects"? I have not made such a claim, I've said I see lots of technical and cleanup aspects. I've only stated the things that *are* happening versus what is not happening at all (literally zilch) and which should be happening, which is efforts towards a solution *not* involving the xz utilities. > Reading in between the lines, e.g. "trying desperately to salvage the > situation with xz-utils", I suspect you are trying to subtly suggest that > any second of time where gentoo hasn't yet removed xz-utils from gentoo as > a dead end is "cavalier". Not quite, I've never advocated removing xz-utils at all, more than happy for it to remain for whoever wants to use it. The only reason I started this thread is I'm very unhappy about that fact that it is currently impossible to NOT execute xz utilities on the Gentoo systems I'm responsible for, without heavy customisation. I'm also not demanding anything, let alone demanding anything instantly. If I have please point out where. > I understand that you are passionate about your suggestion to make > portage not validate distfile hashes. That's incorrect, I've never suggested Portage should not validate distfile hashes. The current behaviour is that validating distfile hashes is something that can be disabled if a user wishes to, and I have no problem with that at all, would not change a thing. I've said that, in order to implement what I have suggested, a user would have to disable it, which is not ideal, but acceptable if the user controls the distfile distribution. And I only suggested that in order to try and make the idea more acceptable by not requiring impractical infra changes that would be needed to generate uncompressed hashes for the Manifests). > But I don't understand how you think > it's a solution to the xz-utils problem. For a wide variety of reasons, > but the simplest one is that your proposal has zero plan of action for > solving this at the community level and is entirely designed to allow > "lone wolf" users to use throwaway systems performing > security-sensitive actions (decompressing and hosting distfiles) in a > networked environment that has the xz-utils installed, to feed into other > security-sensitive systems (daily drivers etc.) that don't, but do have to > trust the artifacts produced by the former. I'm not entirely clear what you're trying to say in this paragraph. But what I will say is I've tried very hard in any suggestions I've made to only suggest things which will NOT change any default behaviour or require big changes. The average user would not see any change from my revised suggestions at all. I accepted after the first responses in this thread that there was no appetite here to stop using xz utils. I then asked the list about an idea I had just to see how palatable it might be. It was not supposed to be a concrete plan, I was seeking
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 1.4.2024 23.07, James Le Cuirot wrote: > > That's not stupid at all, I'd been thinking exactly the same thing. I raised > this whole issue during a discussion at FOSDEM 2019, where I admitted that I > didn't check the code changes for packages I was bumping, knowing that few to > none of the other people in the room did so either. Despite speaking up then, > I still didn't do it because it's a heavy a burden and I'm not paid to do it. > Now I'm thinking I really should, but I could really use some help. I'll raise > this idea at work. You could say that we specialise in these things. :) > > Regards, > Chewi Offtopic but I'll just throw this out there: "pkgdiff-mg -b" from mgorny-dev-scripts does wonders when bumping packages. Not everyone knows about this so posting for awareness. (Maybe slightly related after all since it would've shown the suspicious CmakeLists.txt change at least) -- juippis OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Mon, 2024-04-01 at 20:51 +0200, Kévin GASPARD DE RENEFORT wrote: > > Thanks for clarifying that, it wasn't clear to me when I read the > > earlier e-mail. > > > > Personally I think the long term solution is to identify critical code > > bases that have a low bus factor before the bad actors do and make a > > concentrated community effort to help audit and maintain these code > > bases. > > Hi, > > I hope this is not a stupid suggestion, that is also my first mail here > so if something does not suits habits feel free to tell me please, but > after reading the whole topic here I did not find this suggestion. > > It’s merely a proposition out of my mind, also something I know very > little about. > > --- > > I read Linus T. speaking about usage of AI nowadays, in the IT field and > stating that is an awful idea to write code with it (at least, for now)… > But not to ask an AI to read the code and try to found by this way > security holes, bad habits, bugs and such. > > Again, my skill and knowledge about AI, specially nowadays, is very > small. But would take it lot of works to sets an AI to simple «read» > codes to look for undesired stuff ? That won’t even modify anything, > merely says : «Ah, found something weird, **here**.». Maybe, properly > configured, it would have detected this social-hacking. Maybe not. > > Since programming is a very hard works, specially when it’s about > security and bug, I also have very poor programing skill, but since the > whole purpose of a computer and it’s set of software is to do what an > human could NOT do properly (like being attentives while reading dozens > of hundreds line of code…) and automate stuff, it *seems* to perfectly > suits this need. > > I guess the process on Gentoo side while it’s about "packaging" is > writing the good ebuild that download source code, compressed (and that > is the whole problem here if I understand) and then unpack it, compile > it, etc… > > Could an AI reading the code could be a step somewhere ? > > On other distribution I would say it needs to act **before** the package > is made, while building it I guess, for Gentoo I do not know. > > But that is not the job of Gentoo’s ebuild writer to check other > projects code, that would be a non-sense ! Right ? > > I’m curious of what an AI could bring in this subject. > > If it’s a stupid suggestion, well, will keep reading this topic, very > interesting. And sorry for the noise. > > PS: Thanks for the works behind libre software, open-source and here, > Gentoo. I trust you since I do not have knowledge to judge properly the > works, but Gentoo is indeed one of the best Linux available, if not the > best in some field. Don’t let burn-out takes you and keep your real > priority among everything, even Gentoo or libre software. We are humans, > not machines. > > Regards, > GASPARD DE RENEFORT Kévin That's not stupid at all, I'd been thinking exactly the same thing. I raised this whole issue during a discussion at FOSDEM 2019, where I admitted that I didn't check the code changes for packages I was bumping, knowing that few to none of the other people in the room did so either. Despite speaking up then, I still didn't do it because it's a heavy a burden and I'm not paid to do it. Now I'm thinking I really should, but I could really use some help. I'll raise this idea at work. You could say that we specialise in these things. :) Regards, Chewi signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Thanks for clarifying that, it wasn't clear to me when I read the earlier e-mail. Personally I think the long term solution is to identify critical code bases that have a low bus factor before the bad actors do and make a concentrated community effort to help audit and maintain these code bases. Hi, I hope this is not a stupid suggestion, that is also my first mail here so if something does not suits habits feel free to tell me please, but after reading the whole topic here I did not find this suggestion. It’s merely a proposition out of my mind, also something I know very little about. --- I read Linus T. speaking about usage of AI nowadays, in the IT field and stating that is an awful idea to write code with it (at least, for now)… But not to ask an AI to read the code and try to found by this way security holes, bad habits, bugs and such. Again, my skill and knowledge about AI, specially nowadays, is very small. But would take it lot of works to sets an AI to simple «read» codes to look for undesired stuff ? That won’t even modify anything, merely says : «Ah, found something weird, **here**.». Maybe, properly configured, it would have detected this social-hacking. Maybe not. Since programming is a very hard works, specially when it’s about security and bug, I also have very poor programing skill, but since the whole purpose of a computer and it’s set of software is to do what an human could NOT do properly (like being attentives while reading dozens of hundreds line of code…) and automate stuff, it *seems* to perfectly suits this need. I guess the process on Gentoo side while it’s about "packaging" is writing the good ebuild that download source code, compressed (and that is the whole problem here if I understand) and then unpack it, compile it, etc… Could an AI reading the code could be a step somewhere ? On other distribution I would say it needs to act **before** the package is made, while building it I guess, for Gentoo I do not know. But that is not the job of Gentoo’s ebuild writer to check other projects code, that would be a non-sense ! Right ? I’m curious of what an AI could bring in this subject. If it’s a stupid suggestion, well, will keep reading this topic, very interesting. And sorry for the noise. PS: Thanks for the works behind libre software, open-source and here, Gentoo. I trust you since I do not have knowledge to judge properly the works, but Gentoo is indeed one of the best Linux available, if not the best in some field. Don’t let burn-out takes you and keep your real priority among everything, even Gentoo or libre software. We are humans, not machines. Regards, GASPARD DE RENEFORT Kévin
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Mon, 1 Apr 2024 12:01:13 -0400 Kenton Groombridge wrote: > On 24/04/01 08:40AM, orbea wrote: > > On Mon, 1 Apr 2024 11:14:15 -0400 > > Kenton Groombridge wrote: > > > > > On 24/03/31 12:13PM, Eddie Chapman wrote: > > > > Eli Schwartz wrote: > > > > > On 3/29/24 11:07 PM, Eddie Chapman wrote: > > > > > > > > > >> Given what we've learnt in the last 24hrs about xz utilities, > > > > >> you could forgive a paranoid person for seriously considering > > > > >> getting rid entirely of them from their systems, especially > > > > >> since there are suitable alternatives available. Some might > > > > >> say that's a bit extreme, xz-utils will get a thorough audit > > > > >> and it will all be fine. But when a malicious actor has been > > > > >> a key maintainer of something as complex as a decompression > > > > >> utility for years, I'm not sure I could ever trust that > > > > >> codebase again. Maybe a complete rewrite will emerge, but > > > > >> I'm personally unwilling to continue using xz utils in the > > > > >> meantime for uncompressing anything on my systems, even if > > > > >> it is done by an unprivileged process. > > > > > > > > > > > > > > > It suffices to downgrade to the version of xz before a social > > > > > engineering attack by a malicious actor to gain > > > > > maintainership of the xz project. > > > > > > > > > > Have you been linked to this yet? > > > > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html > > > > > > > > > > -- > > > > > Eli Schwartz > > > > > > > > > > > > > Yes I saw that yesterday. It only increased my level of concern > > > > about the project ten-fold rather than decreased it, > > > > particularly because of "he has been helping a lot off-list and > > > > is practically a co-maintainer already". > > > > > > > > > > > > > > I think it's important to realize that this could have potentially > > > happened to any number of various open source projects, not just > > > xz-utils. Simply ripping it out and replacing it is not enough to > > > prevent these kinds of issues from happening in the future. > > > > > > There is a major shifting of perspectives as a result of this > > > unfortunate compromise. Right now there are serious considerations > > > about banning (or otherwise auditing) binary blobs in some > > > projects. There are talks about banning the use of older build > > > systems like autotools in favor of ones more easily readable and > > > auditable. > > > > Talk about throwing the baby out with the bathwater... > > > > Let's not shoot the messenger here. :) > > I cited this specific example to highlight the shared intent behind > positive changes to auditing code not just in the program but also its > build system. I didn't mean to imply that this was a great solution. Thanks for clarifying that, it wasn't clear to me when I read the earlier e-mail. Personally I think the long term solution is to identify critical code bases that have a low bus factor before the bad actors do and make a concentrated community effort to help audit and maintain these code bases. > > > Its fully possible to write autotools build systems which are simple > > and easy to audit. Depending on what blob does it may be far from > > trivial or advisable to get rid of it. > > > > This attack as already has been clearly stated is social, not > > technical. If xz-utils used meson or cmake instead it would of not > > changed the situation. > > > > > Ultimately what is happening is a reflection on how we audit > > > critical system components and contributions made to them. Change > > > is not going to happen over night. > > > > > > We saw a similar shift with OpenSSL's heartbleed, which > > > ultimately led to positive changes in code quality and improving > > > their vulnerability reporting process. > > > > > > There is some good to come of this event, but it's important to > > > recognize what went wrong and how open source can improve as a > > > whole. > > > > >
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 24/04/01 08:40AM, orbea wrote: > On Mon, 1 Apr 2024 11:14:15 -0400 > Kenton Groombridge wrote: > > > On 24/03/31 12:13PM, Eddie Chapman wrote: > > > Eli Schwartz wrote: > > > > On 3/29/24 11:07 PM, Eddie Chapman wrote: > > > > > > > >> Given what we've learnt in the last 24hrs about xz utilities, > > > >> you could forgive a paranoid person for seriously considering > > > >> getting rid entirely of them from their systems, especially > > > >> since there are suitable alternatives available. Some might say > > > >> that's a bit extreme, xz-utils will get a thorough audit and it > > > >> will all be fine. But when a malicious actor has been a key > > > >> maintainer of something as complex as a decompression utility > > > >> for years, I'm not sure I could ever trust that codebase again. > > > >> Maybe a complete rewrite will emerge, but I'm personally > > > >> unwilling to continue using xz utils in the meantime for > > > >> uncompressing anything on my systems, even if it is done by an > > > >> unprivileged process. > > > > > > > > > > > > It suffices to downgrade to the version of xz before a social > > > > engineering attack by a malicious actor to gain maintainership of > > > > the xz project. > > > > > > > > Have you been linked to this yet? > > > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html > > > > > > > > -- > > > > Eli Schwartz > > > > > > > > > > Yes I saw that yesterday. It only increased my level of concern > > > about the project ten-fold rather than decreased it, particularly > > > because of "he has been helping a lot off-list and is practically a > > > co-maintainer already". > > > > > > > > > > I think it's important to realize that this could have potentially > > happened to any number of various open source projects, not just > > xz-utils. Simply ripping it out and replacing it is not enough to > > prevent these kinds of issues from happening in the future. > > > > There is a major shifting of perspectives as a result of this > > unfortunate compromise. Right now there are serious considerations > > about banning (or otherwise auditing) binary blobs in some projects. > > There are talks about banning the use of older build systems like > > autotools in favor of ones more easily readable and auditable. > > Talk about throwing the baby out with the bathwater... > Let's not shoot the messenger here. :) I cited this specific example to highlight the shared intent behind positive changes to auditing code not just in the program but also its build system. I didn't mean to imply that this was a great solution. > Its fully possible to write autotools build systems which are simple > and easy to audit. Depending on what blob does it may be far from > trivial or advisable to get rid of it. > > This attack as already has been clearly stated is social, not > technical. If xz-utils used meson or cmake instead it would of not > changed the situation. > > > Ultimately what is happening is a reflection on how we audit critical > > system components and contributions made to them. Change is not going > > to happen over night. > > > > We saw a similar shift with OpenSSL's heartbleed, which ultimately led > > to positive changes in code quality and improving their vulnerability > > reporting process. > > > > There is some good to come of this event, but it's important to > > recognize what went wrong and how open source can improve as a whole. > > > > -- Kenton Groombridge Gentoo Linux Developer, SELinux Project signature.asc Description: PGP signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Mon, 1 Apr 2024 11:14:15 -0400 Kenton Groombridge wrote: > On 24/03/31 12:13PM, Eddie Chapman wrote: > > Eli Schwartz wrote: > > > On 3/29/24 11:07 PM, Eddie Chapman wrote: > > > > > >> Given what we've learnt in the last 24hrs about xz utilities, > > >> you could forgive a paranoid person for seriously considering > > >> getting rid entirely of them from their systems, especially > > >> since there are suitable alternatives available. Some might say > > >> that's a bit extreme, xz-utils will get a thorough audit and it > > >> will all be fine. But when a malicious actor has been a key > > >> maintainer of something as complex as a decompression utility > > >> for years, I'm not sure I could ever trust that codebase again. > > >> Maybe a complete rewrite will emerge, but I'm personally > > >> unwilling to continue using xz utils in the meantime for > > >> uncompressing anything on my systems, even if it is done by an > > >> unprivileged process. > > > > > > > > > It suffices to downgrade to the version of xz before a social > > > engineering attack by a malicious actor to gain maintainership of > > > the xz project. > > > > > > Have you been linked to this yet? > > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html > > > > > > -- > > > Eli Schwartz > > > > > > > Yes I saw that yesterday. It only increased my level of concern > > about the project ten-fold rather than decreased it, particularly > > because of "he has been helping a lot off-list and is practically a > > co-maintainer already". > > > > > > I think it's important to realize that this could have potentially > happened to any number of various open source projects, not just > xz-utils. Simply ripping it out and replacing it is not enough to > prevent these kinds of issues from happening in the future. > > There is a major shifting of perspectives as a result of this > unfortunate compromise. Right now there are serious considerations > about banning (or otherwise auditing) binary blobs in some projects. > There are talks about banning the use of older build systems like > autotools in favor of ones more easily readable and auditable. Talk about throwing the baby out with the bathwater... Its fully possible to write autotools build systems which are simple and easy to audit. Depending on what blob does it may be far from trivial or advisable to get rid of it. This attack as already has been clearly stated is social, not technical. If xz-utils used meson or cmake instead it would of not changed the situation. > Ultimately what is happening is a reflection on how we audit critical > system components and contributions made to them. Change is not going > to happen over night. > > We saw a similar shift with OpenSSL's heartbleed, which ultimately led > to positive changes in code quality and improving their vulnerability > reporting process. > > There is some good to come of this event, but it's important to > recognize what went wrong and how open source can improve as a whole. >
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 24/03/31 12:13PM, Eddie Chapman wrote: > Eli Schwartz wrote: > > On 3/29/24 11:07 PM, Eddie Chapman wrote: > > > >> Given what we've learnt in the last 24hrs about xz utilities, you could > >> forgive a paranoid person for seriously considering getting rid > >> entirely of them from their systems, especially since there are suitable > >> alternatives available. Some might say that's a bit extreme, xz-utils > >> will get a thorough audit and it will all be fine. But when a > >> malicious actor has been a key maintainer of something as complex as a > >> decompression utility for years, I'm not sure I could ever trust that > >> codebase again. Maybe a complete rewrite will emerge, but I'm personally > >> unwilling to continue using xz utils in the meantime for uncompressing > >> anything on my systems, even if it is done by an unprivileged process. > > > > > > It suffices to downgrade to the version of xz before a social > > engineering attack by a malicious actor to gain maintainership of the xz > > project. > > > > Have you been linked to this yet? > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html > > > > -- > > Eli Schwartz > > > > Yes I saw that yesterday. It only increased my level of concern about the > project ten-fold rather than decreased it, particularly because of "he has > been helping a lot off-list and is practically a co-maintainer already". > > I think it's important to realize that this could have potentially happened to any number of various open source projects, not just xz-utils. Simply ripping it out and replacing it is not enough to prevent these kinds of issues from happening in the future. There is a major shifting of perspectives as a result of this unfortunate compromise. Right now there are serious considerations about banning (or otherwise auditing) binary blobs in some projects. There are talks about banning the use of older build systems like autotools in favor of ones more easily readable and auditable. Ultimately what is happening is a reflection on how we audit critical system components and contributions made to them. Change is not going to happen over night. We saw a similar shift with OpenSSL's heartbleed, which ultimately led to positive changes in code quality and improving their vulnerability reporting process. There is some good to come of this event, but it's important to recognize what went wrong and how open source can improve as a whole. -- Kenton Groombridge Gentoo Linux Developer, SELinux Project signature.asc Description: PGP signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
сб, 30 мар. 2024 г. в 06:07, Eddie Chapman : > > Given what we've learnt in the last 24hrs about xz utilities, you could > forgive a paranoid person for seriously considering getting rid entirely > of them from their systems, especially since there are suitable > alternatives available. Some might say that's a bit extreme, xz-utils > will get a thorough audit and it will all be fine. But when a malicious > actor has been a key maintainer of something as complex as a decompression > utility for years, I'm not sure I could ever trust that codebase again. > Maybe a complete rewrite will emerge, but I'm personally unwilling to > continue using xz utils in the meantime for uncompressing anything on my > systems, even if it is done by an unprivileged process. > > I see that many system package ebuilds unconditionally expect > app-arch/xz-utils to be installed simply to be able to decompress the > source archive in SRC_URI. So simply specifying -lzma on your system isn't > going to get rid of it. > > No one could have been expected to foresee what's happened with xz-utils, > but now that it's here, perhaps Gentoo (and other projects that do) should > consider not relying on a single decompression algorithm for source > archives, even just as an insurance against some other yet unknown > disaster with one algorithm or another in future? > > And yes I'm sure there will be individual packages that currently > absolutely need xz-utils installed during the build process, and one or > two that absolutely have to have it available at runtime, but those > bridges can be crossed as and when. > > Eddie > > There is no problem in the XZ/LZMA format itself as the reference algorithm is not compromised. It's all about trust between developers of application and developers of distribution. If you lost trust to xz-utils's developers, you may use alternatives like app-arch/pxz or app-arch/pixz. I don't see reasons why we should change format instead of changing a tool. -- >From Siberia with Love!
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Mon, 2024-04-01 at 08:57 +0100, Eddie Chapman wrote: > I stand by and reiterate my view that there is far too much of a cavalier > attitude towards the matter in general out there including here in Gentoo. > But not in particular here, it is everywhere where this is being discussed > at the moment. I would like to point out that the xz/sshd issue was primarily a social one, not a technical one. The primary problem in open source today isn't bad code. It's projects relying on overburdened, burned out maintainers. And on top of that, users who are complaining, demanding, outright hostile or primarily contributing by walls of text on a mailing lists, that bring nothing to discussion except for furthering the burnout of open source developers who are actually trying to do something. Think about that. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 4/1/24 3:57 AM, Eddie Chapman wrote: > No, I don't need to do that. I don't appreciate suggestions to "just calm > down", especially when I'm not being hysterical. Your comment to me just > reinforces what I mean when I say there is far too much of a cavalier > attitude. I think you're making a big mistake by confusing "approach the issue with a calm and clearheaded approach, be methodical about how you analyze and react to trouble spots" with "everyone is being cavalier". But also, please keep in mind that 98% of all people on the internet can do whatever they want and it simply doesn't matter. They are public commentators at a three-ring circus and their cavalier or panicked attitudes change nothing. Well, they change one thing. It's hard for the security professionals at work to deal with things when they are constantly having to respond to the three-ring circus. > I stand by and reiterate my view that there is far too much of a cavalier > attitude towards the matter in general out there including here in Gentoo. > But not in particular here, it is everywhere where this is being discussed > at the moment. I don't care where this is being "discussed", scare quotes intentional. > But please think a little about what I mean when I say a "cavalier > attitude", and what it does NOT mean. It does not mean that a lot of > people are not working very hard to analyse and learn about this issue and > taking steps to try to mitigate it. It does not mean people are not well > intentioned, everyone wants to fix this. I have great appreciation and > admiration for a lot of fantastic work I see going on including by people > involved in Gentoo. But I believe it will only really be beneficial in the > far future, not right now. Please stop insulting the work of the people who are working very hard to analyze and learn about this issue and taking steps to try to mitigate it... > How are people in general being cavalier? By trying desperately to salvage > the situation with xz-utils above all else, by focussing too much on how > the original author of xz-utils and rallying round them (absolutely a > great thing to do but has absolutely nothing to do with what is good or > not good for users as a whole right now), there is too much clouded > judgment. There is more I could argue about why I use that word, but I > know by now that I am going against the grain of what the majority want > and it's not what people want to hear so I'm done, this discussion is now > a waste of everyone's time here including mine. ... by implying that people who are NOT part of that process "rallying around the original author" (an act of human compassion!!! which you admit is a good thing) is, somehow, detrimental to the process of working very hard to analyze and learn about this issue and taking steps to try to mitigate it. What does one have to do with the other? Why is it necessary to claim that based on some sort of vibe check "there is too much compassion going around in our communities, and this must mean that not enough effort is being expended on the technical and cleanup aspects"? ... Reading in between the lines, e.g. "trying desperately to salvage the situation with xz-utils", I suspect you are trying to subtly suggest that any second of time where gentoo hasn't yet removed xz-utils from gentoo as a dead end is "cavalier". Considering the fact that xz-utils is widely used and on the critpath for people to actually get work done, including to actually acquire extremely important software that already exists and must somehow be dealt with, I do indeed think that the situation needs salvaging and the community needs some form of xz decompressor. Fortunately, as you've agreed, we know the original xz-utils circa 2020 and before is trustworthy, so using that is viable and under discussion. I understand that you are passionate about your suggestion to make portage not validate distfile hashes. But I don't understand how you think it's a solution to the xz-utils problem. For a wide variety of reasons, but the simplest one is that your proposal has zero plan of action for solving this at the community level and is entirely designed to allow "lone wolf" users to use throwaway systems performing security-sensitive actions (decompressing and hosting distfiles) in a networked environment that has the xz-utils installed, to feed into other security-sensitive systems (daily drivers etc.) that don't, but do have to trust the artifacts produced by the former. It's not being cavalier when zero portage developers responded by saying "good idea I'll drop everything so I can get right on this and implement it". But if you are absolutely positive this is the right solution, I have an offer for you: implement this yourself, submit patches, and then we'll have something to talk about. -- Eli Schwartz OpenPGP_0x84818A6819AF4A9B.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Matt Jolly wrote: > Hi Eddie, > > On 31/3/24 21:13, Eddie Chapman wrote: > >> At the moment there is far too much of >> a cavalier attitude about the whole thing being shown by too many, >> including here I'm sad to see. > > It's obvious that this is something that you are very worried about, but > I think that you need to take a deep breath and relax a little. I have > not seen a cavalier attitude towards this issue. No, I don't need to do that. I don't appreciate suggestions to "just calm down", especially when I'm not being hysterical. Your comment to me just reinforces what I mean when I say there is far too much of a cavalier attitude. > What I see instead are developers who have made an initial assessment > and disclosure, have taken some actions to mitigate the severity of the > issue, and are carefully continuing their investigations (over a major > holiday in large parts of the world) so that they can issue sane responses > to actual threats, and not a knee-jerk response like 'Gentoo should > re-encode all xzs in other formats' which, as was discussed above, adds > significant complexity for no real benefit. I stand by and reiterate my view that there is far too much of a cavalier attitude towards the matter in general out there including here in Gentoo. But not in particular here, it is everywhere where this is being discussed at the moment. But please think a little about what I mean when I say a "cavalier attitude", and what it does NOT mean. It does not mean that a lot of people are not working very hard to analyse and learn about this issue and taking steps to try to mitigate it. It does not mean people are not well intentioned, everyone wants to fix this. I have great appreciation and admiration for a lot of fantastic work I see going on including by people involved in Gentoo. But I believe it will only really be beneficial in the far future, not right now. How are people in general being cavalier? By trying desperately to salvage the situation with xz-utils above all else, by focussing too much on how the original author of xz-utils and rallying round them (absolutely a great thing to do but has absolutely nothing to do with what is good or not good for users as a whole right now), there is too much clouded judgment. There is more I could argue about why I use that word, but I know by now that I am going against the grain of what the majority want and it's not what people want to hear so I'm done, this discussion is now a waste of everyone's time here including mine. > I've seen from your previous emails to the list that you know what > paragraphs are and how to use them to break up your content into digestible > chunks. Please continue using them - it makes it significantly easier to > respond to your ideas and gives off an aura of professionalism that you > will need if you want your concerns to be taken seriously and addressed > directly. Yes you are right, I do apologise for not using paragraphs in my last message, I slipped there, thanks for pointing it out. I've tried to do so in this message. > Cheers, > > Matt And I should make more of an effort to sign off, it's a little more friendly :-) Regards, Eddie
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Hi Eddie, On 31/3/24 21:13, Eddie Chapman wrote: At the moment there is far too much of a cavalier attitude about the whole thing being shown by too many, including here I'm sad to see. It's obvious that this is something that you are very worried about, but I think that you need to take a deep breath and relax a little. I have not seen a cavalier attitude towards this issue. What I see instead are developers who have made an initial assessment and disclosure, have taken some actions to mitigate the severity of the issue, and are carefully continuing their investigations (over a major holiday in large parts of the world) so that they can issue sane responses to actual threats, and not a knee-jerk response like 'Gentoo should re-encode all xzs in other formats' which, as was discussed above, adds significant complexity for no real benefit. I've seen from your previous emails to the list that you know what paragraphs are and how to use them to break up your content into digestible chunks. Please continue using them - it makes it significantly easier to respond to your ideas and gives off an aura of professionalism that you will need if you want your concerns to be taken seriously and addressed directly. Cheers, Matt
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 2024-03-31 01:33, Eli Schwartz wrote: On 3/29/24 11:07 PM, Eddie Chapman wrote: Given what we've learnt in the last 24hrs about xz utilities, you could forgive a paranoid person for seriously considering getting rid entirely of them from their systems, especially since there are suitable alternatives available. Some might say that's a bit extreme, xz-utils will get a thorough audit and it will all be fine. But when a malicious actor has been a key maintainer of something as complex as a decompression utility for years, I'm not sure I could ever trust that codebase again. Maybe a complete rewrite will emerge, but I'm personally unwilling to continue using xz utils in the meantime for uncompressing anything on my systems, even if it is done by an unprivileged process. It suffices to downgrade to the version of xz before a social engineering attack by a malicious actor to gain maintainership of the xz project. Have you been linked to this yet? https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html Wed, 29 Jun 2022 13:07:07 -0700 This is 2 years ago. Had I seen someone say that a bad actor would spend years gaining the trust of FOSS project maintainers in order to gain commit access and introduce such sophisticated back doors, I would have told them to take their meds. This is insane. Not even this seems impossible anymore: https://01.me/en/2014/11/insert-backdoor-into-compiler/ If this happened to something like firefox, I don't think anyone would have found out. No one bats an eye if a website loads 0.5s longer. -- Linux-gentoo-x86_64-Intel-R-_Core-TM-_i5-7400_CPU_@_3.00GHz COMMON_FLAGS="-O3 -pipe -march=native -fno-stack-protector -ftree-vectorize -ffast-math -funswitch-loops -fuse-linker-plugin -flto -fdevirtualize-at-ltrans -fno-plt -fno-semantic-interposition -falign-functions=64 -fgraphite-identity -floop-nest-optimize" USE="-* git verify-sig rsync-verify man alsa X grub ssl ipv6 lto libressl olde-gentoo asm native-symlinks threads jit jumbo-build minimal strip system-man" INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd /usr/lib/modules-load.d /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus /lib/udev /usr/share/icons /usr/share/applications /usr/share/gtk-3.0/emoji"
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eli Schwartz wrote: > On 3/29/24 11:07 PM, Eddie Chapman wrote: > >> Given what we've learnt in the last 24hrs about xz utilities, you could >> forgive a paranoid person for seriously considering getting rid >> entirely of them from their systems, especially since there are suitable >> alternatives available. Some might say that's a bit extreme, xz-utils >> will get a thorough audit and it will all be fine. But when a >> malicious actor has been a key maintainer of something as complex as a >> decompression utility for years, I'm not sure I could ever trust that >> codebase again. Maybe a complete rewrite will emerge, but I'm personally >> unwilling to continue using xz utils in the meantime for uncompressing >> anything on my systems, even if it is done by an unprivileged process. > > > It suffices to downgrade to the version of xz before a social > engineering attack by a malicious actor to gain maintainership of the xz > project. > > Have you been linked to this yet? > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html > > -- > Eli Schwartz > Yes I saw that yesterday. It only increased my level of concern about the project ten-fold rather than decreased it, particularly because of "he has been helping a lot off-list and is practically a co-maintainer already". It's not possible to just downgrade to before the bad actor's commits and then feel fine about things because they have been heavily involved offline even before commit access. We'll never know how much and when because I also cannot trust what the apparently innocent maintainer (who is most likely a victim here as well) might say about that now. Not because of anything about them (I don't know them or anything about them), just because of what has happened, there is too much of an incentive for that person to now downplay the involvement of the bad actor. I'm sorry if that may seem harsh but, in my view, this situation is so severe it warrants it. The world is facing threats from very sophisticated and capable bad actors, mostly criminal organisations. If people here want to run systems that are actually secure and also have other people trust their stewardship, then things need to be taken seriously and high standards need to be maintained. Especially where it is a tool that is not super essential (it has just become heavily entrenched) and where there are great alternatives, there should be no hesitancy to jettison a project that has been infiltrated to such an extent as we have seen here (this is far beyond just some devs workstation got compromised and there was a few bad commits made it into the repo). At the moment there is far too much of a cavalier attitude about the whole thing being shown by too many, including here I'm sad to see.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 3/30/24 11:17 AM, Eddie Chapman wrote: > Yes that's a very good point, that was something I was wondering in > weighing up both sides, what the costs would be practically, as I don't > know the realities of running Gentoo infrastructure. And maybe the costs > is just too high of a price to pay. > > I wonder if increased use of git repos rather than distributed tarballs > could be part of a solution to those issues, although that could put quite > a storage burden on every user. Unless they were all shallow git pulls and > the user could optionally choose to tar up the git directory after clone > with compression. But yes granted then there is even more ebuild > complexity. Live ebuilds cannot have keywords, so using git repos is not a valid option. There's not really much to discuss here. Recompressing all distfiles in gentoo-specific ways is... definitely a decision. It's a decision that Debian has made, mind you, so it's not like Gentoo would be breaking new ground here, but frankly I don't really regard that as fundamentally palatable. -- Eli Schwartz OpenPGP_0x84818A6819AF4A9B.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On 3/29/24 11:07 PM, Eddie Chapman wrote: > Given what we've learnt in the last 24hrs about xz utilities, you could > forgive a paranoid person for seriously considering getting rid entirely > of them from their systems, especially since there are suitable > alternatives available. Some might say that's a bit extreme, xz-utils > will get a thorough audit and it will all be fine. But when a malicious > actor has been a key maintainer of something as complex as a decompression > utility for years, I'm not sure I could ever trust that codebase again. > Maybe a complete rewrite will emerge, but I'm personally unwilling to > continue using xz utils in the meantime for uncompressing anything on my > systems, even if it is done by an unprivileged process. It suffices to downgrade to the version of xz before a social engineering attack by a malicious actor to gain maintainership of the xz project. Have you been linked to this yet? https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html -- Eli Schwartz OpenPGP_0x84818A6819AF4A9B.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
"Eddie Chapman" writes: > Given what we've learnt in the last 24hrs about xz utilities, you could > forgive a paranoid person for seriously considering getting rid entirely > of them from their systems, especially since there are suitable > alternatives available. Some might say that's a bit extreme, xz-utils > will get a thorough audit and it will all be fine. But when a malicious > actor has been a key maintainer of something as complex as a decompression > utility for years, I'm not sure I could ever trust that codebase again. > Maybe a complete rewrite will emerge, but I'm personally unwilling to > continue using xz utils in the meantime for uncompressing anything on my > systems, even if it is done by an unprivileged process. My own view is that there'll be a time for introspection, reflection, and discussion of changes once the crisis is over. We're not there yet. But I don't think us fetching several variants of compression formats and testing & verifying all of them is feasible. I also think it's (and I don't mean this derogatorily towards you) naive for people in general to suggest that this is really specific to xz at all. Unfortunately, there's many. many projects this could've happened to. > > I see that many system package ebuilds unconditionally expect > app-arch/xz-utils to be installed simply to be able to decompress the > source archive in SRC_URI. So simply specifying -lzma on your system isn't > going to get rid of it. > > No one could have been expected to foresee what's happened with xz-utils, > but now that it's here, perhaps Gentoo (and other projects that do) should > consider not relying on a single decompression algorithm for source > archives, even just as an insurance against some other yet unknown > disaster with one algorithm or another in future? I think there's real discussions to be had about relying on dist tarballs and such but I don't really see how we could try to avoid compression algorithms. > > And yes I'm sure there will be individual packages that currently > absolutely need xz-utils installed during the build process, and one or > two that absolutely have to have it available at runtime, but those > bridges can be crossed as and when. > > Eddie thanks, sam
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eddie Chapman wrote:
> Michał Górny wrote:
>
>> On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote:
>>
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>> is wouldn't it be nice if there were at least 2 alternatives to
>>> choose from? That doesn't have to be disruptive in any way, people who
>>> wish to continue using and trusting xz-utils should be able to
>>> continue to do so without any friction whatsoever.
>>
>> So, you're basically saying we should go out of our way, recompress all
>> distfiles using two alternative compression formats, increase mirror
>> load four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Michał Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls
> and the user could optionally choose to tar up the git directory after
> clone with compression. But yes granted then there is even more ebuild
> complexity.
>
I've been thinking a little about how Gentoo without
compression/decompression of distfiles could work, as a feature, without
any impact on the existing world order, and no increased stress on Gentoo
infra. I was wondering how palatable the following idea might be to others
...
The basis of the idea is to add a feature to Portage which would let a
person optionally indicate in make.conf that whenever a path in SRC_URI
resolves to a file with a compression extension (.gz, .bz2, .xz, etc),
that Portage should attempt to fetch it without the compression extension.
So as an example, lets take sys-apps/pciutils, which currently has:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar.gz;
the feature would tell portage to simply translate this to:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar;
So perhaps it could be a flag that goes in FEATURES= called something like
"strip_dist_comp" or something similar, or maybe someone has a better idea
about that.
Now, of course, I'm not proposing that Gentoo infra keeps uncompressed
versions of distfiles. So by default Portage would encounter a 404 error
when it tries to fetch the uncompressed file from Gentoo mirrors.
However, this feature would then pave the way for a person to then
configure Portage to fetch distfiles from their own server as well as
Gentoo mirrors, and that person could then keep their own uncompressed
versions of distfiles on their server, for however many and whichever
distfiles they might wish to keep there, as the compressed version would
get fetched from a Gentoo mirror if the uncompressed version is not there.
Such a person would then have to obtain or create their own uncompressed
distfile independently.
A caveat of this solution would be that one would have to disable checksum
verification (and gpg checks?) for this to work, as of course there would
be no checksum for the uncompressed version in the Manifest, and Gentoo
infra certainly should not be expected to especially uncompress each
distfile once in order to generate an extra checksum for the Manifest. In
fact I'd consider than undesirable, as anyone paranoid enough to want to
do this would not trust such a checksum anyway, since it would be a
checksum of a file that has been compressed at source and then
decompressed on Gentoo infra, potentially introducing vulnerabilities.
However, the lack of checksum is not a problem for someone who wants to
keep distfiles on their own server, as such a person can also be
responsible themselves for first verifying whatever they put on there, and
for keeping said server secured from tampering.
This seems to me to be something that would probably be relatively
straightforward to implement within Portage, maybe with just a few lines
around the python code that fetches the SRC_URI, and zero extra work or
resources required from Gentoo infra.
I'd consider it a feature for anyone who wants to eliminate a whole
potential class of vulnerabilities that may or may not be present either
now or in future in compression algorithm tools. Surely that would be a
nice feature to have for some folk?
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Rich Freeman wrote: > On Sat, Mar 30, 2024 at 10:57 AM Eddie Chapman wrote: > >> No, this is the the bad actor *themselves* being a >> principal author of the software, working stealthily and in very >> sophisticated ways for years, to manoeuvrer themselves and their >> software into a position of trust in the ecosystem whereby they were >> almost able to pull off the mother of all security nightmares for the >> world. > > This is entirely speculative at this point. It isn't certain that the > author is the one behind the exploit, and if they were, it is not known for > how long their intentions were malicious, or even what their motivations > were. It is also unclear what pseudonymous accounts with what projects > are associated with the attacker. For the purposes of this discussion I'm not speculating nor interested in *who* is behind this, or whether or whoever committed commits was a victim of account takeover. Certain key actions that have been taken over time by whoever is/was behind this do not require any speculation, they speak for themselves, and are clearly malicious. There is no need to wait for anything more to be revealed to be able to plainly see how bad it is. While we wait and see, huge numbers of people might be suffering real and serious consequences of continued use of xz-utils. The consequences may be completely hidden, if you go by how well the bad actor here has managed to hide what they have done. If you are following developments you can see right now with your own eyes how incredibly difficult it is for our smartest people to unravel and pick through what this actor has done. To have faith that everything malicious that the perpetrator has done will be unravelled over time by our collective smart minds by going over the codebase with a fine tooth-comb puts far too much faith in human beings and takes unnecessary risks for something that is not worth that risk when there are alternatives. If you were looking for a compression tool for a new project, why would anyone sane take such risks for such little gain? You just wouldn't. Of course the reason there is hesitancy is because xz has become so deeply entrenched in our world, it's become almost too hard to extrapolate ourselves from it. I dare say the attacker realised this and probably sought to take advantage of that fact. However, I do acknowledge and realise the significant practical difficulties that would be involved in making xz-utils something optional within Gentoo.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Eddie Chapman wrote: > Michał Górny wrote: >> On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: >> >>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying >>> is wouldn't it be nice if there were at least 2 alternatives to choose >>> from? That doesn't have to be disruptive in any way, people who wish to >>> continue using and trusting xz-utils should be able to continue to do so >>> without any friction whatsoever. >> So, you're basically saying we should go out of our way, recompress all >> distfiles using two alternative compression formats, increase mirror load >> four times and add a lot of complexity to ebuilds, right? >> >> -- >> Best regards, >> Michał Górny >> > Yes that's a very good point, that was something I was wondering in > weighing up both sides, what the costs would be practically, as I don't > know the realities of running Gentoo infrastructure. And maybe the costs > is just too high of a price to pay. > > I wonder if increased use of git repos rather than distributed tarballs > could be part of a solution to those issues, although that could put quite > a storage burden on every user. Unless they were all shallow git pulls and > the user could optionally choose to tar up the git directory after clone > with compression. But yes granted then there is even more ebuild > complexity. > > > . > There is a lot of unknowns out there. From what I've read, the person responsible for writing the code inserted this hack. There may be no way to prevent this. Basically, the person that should have been trusted with this code violated that trust. Why is unknown but I'm as curious about that as anything. It's like when someone goes to a grocery store to buy a tomato. They want organic and there is a organic sticker on the tomato. You either trust that sticker, and the person/company who put it on there, or you don't trust that sticker at all and avoid buying all tomatoes. The trust starts with the person/company that puts that sticker on the tomato. The person who was trusted with that code, broke that trust. There is likely hundreds of packages out there in the exact same position. Any package that has few or only one person writing the code can do the same thing. While this should be analyzed as more info comes in, right now, we should let the devs get us back to as safe a place as possible. Since it appears to affect systemd users who don't use Gentoo, which is a huge target, they certainly need to react as quickly as they can to the devs actions. Let's just not overreact just yet. The devs has rolled back to a safe, safer, version. Let time and more info sort this out. If it is needed, xz will go away, which shouldn't come as a surprise. I'm sure the person who did this will never get that trust back. Long term, this is going to be interesting to see what all gets revealed. The why is one thing. Another is how to prevent if it can be at all. I'm going back to my hole now. Dale :-) :-) P. S. Links that some may want to follow, instead of a -dev thread. https://bugs.gentoo.org/928134 https://forums.gentoo.org/viewtopic.php?p=8821925
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Michał Górny wrote: > On Sat, 2024-03-30 at 15:17 +, Eddie Chapman wrote: > >> Michał Górny wrote: >> >>> On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: >>> >>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is wouldn't it be nice if there were at least 2 alternatives to choose from? That doesn't have to be disruptive in any way, people who wish to continue using and trusting xz-utils should be able to continue to do so without any friction whatsoever. >>> >>> So, you're basically saying we should go out of our way, recompress >>> all distfiles using two alternative compression formats, increase >>> mirror load four times and add a lot of complexity to ebuilds, right? >>> >>> -- >>> Best regards, >>> Michał Górny >>> >>> >> >> Yes that's a very good point, that was something I was wondering in >> weighing up both sides, what the costs would be practically, as I don't >> know the realities of running Gentoo infrastructure. And maybe the >> costs is just too high of a price to pay. >> >> I wonder if increased use of git repos rather than distributed tarballs >> could be part of a solution to those issues, although that could put >> quite a storage burden on every user. Unless they were all shallow git >> pulls and the user could optionally choose to tar up the git directory >> after clone with compression. But yes granted then there is even more >> ebuild complexity. >> > > Should we convert git repositories to Mercurial and Bazaar too, to avoid > relying too much on a single tool? > > -- > Best regards, > Michał Górny > I sense that question may have been slightly in jest :-) At least I hope so as it could also be interpreted as an attempt at ridicule. I'll take it as the former. In case you are seriously asking; of course not, that's totally unnecessary. The objective is simply to obtain the upstream source code intact. We don't need whatever version control of their source they are using, which of course is the whole point of fetching distributed tarballs. My suggestion of git pulls is just to address your point of resource usage on gentoo infra, it reduces the need to store binary dist files. I've also heard some argue that relying on distributed tarballs is part of the overall problem and what the bad actor was taking advantage of. They may have a point.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, 2024-03-30 at 15:17 +, Eddie Chapman wrote: > Michał Górny wrote: > > On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: > > > > > Note, I'm not advocating ripping xz-utils out of tree, all I'm saying > > > is wouldn't it be nice if there were at least 2 alternatives to choose > > > from? That doesn't have to be disruptive in any way, people who wish to > > > continue using and trusting xz-utils should be able to continue to do so > > > without any friction whatsoever. > > > > So, you're basically saying we should go out of our way, recompress all > > distfiles using two alternative compression formats, increase mirror load > > four times and add a lot of complexity to ebuilds, right? > > > > -- > > Best regards, > > Michał Górny > > > > Yes that's a very good point, that was something I was wondering in > weighing up both sides, what the costs would be practically, as I don't > know the realities of running Gentoo infrastructure. And maybe the costs > is just too high of a price to pay. > > I wonder if increased use of git repos rather than distributed tarballs > could be part of a solution to those issues, although that could put quite > a storage burden on every user. Unless they were all shallow git pulls and > the user could optionally choose to tar up the git directory after clone > with compression. But yes granted then there is even more ebuild > complexity. > Should we convert git repositories to Mercurial and Bazaar too, to avoid relying too much on a single tool? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, 30 Mar 2024 16:02:25 +0100 Michał Górny wrote: > On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: > > Note, I'm not advocating ripping xz-utils out of tree, all I'm > > saying is wouldn't it be nice if there were at least 2 alternatives > > to choose from? That doesn't have to be disruptive in any way, > > people who wish to continue using and trusting xz-utils should be > > able to continue to do so without any friction whatsoever. > > So, you're basically saying we should go out of our way, recompress > all distfiles using two alternative compression formats, increase > mirror load four times and add a lot of complexity to ebuilds, right? > How would Gentoo even recompress all .xz distfiles if xz-utils is not even in the repo? And this will certainly be a reoccurring theme...
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Michał Górny wrote: > On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: > >> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying >> is wouldn't it be nice if there were at least 2 alternatives to choose >> from? That doesn't have to be disruptive in any way, people who wish to >> continue using and trusting xz-utils should be able to continue to do so >> without any friction whatsoever. > > So, you're basically saying we should go out of our way, recompress all > distfiles using two alternative compression formats, increase mirror load > four times and add a lot of complexity to ebuilds, right? > > -- > Best regards, > Michał Górny > Yes that's a very good point, that was something I was wondering in weighing up both sides, what the costs would be practically, as I don't know the realities of running Gentoo infrastructure. And maybe the costs is just too high of a price to pay. I wonder if increased use of git repos rather than distributed tarballs could be part of a solution to those issues, although that could put quite a storage burden on every user. Unless they were all shallow git pulls and the user could optionally choose to tar up the git directory after clone with compression. But yes granted then there is even more ebuild complexity.
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, Mar 30, 2024 at 10:57 AM Eddie Chapman wrote: > > No, this is the the bad actor *themselves* being a > principal author of the software, working stealthily and in very > sophisticated ways for years, to manoeuvrer themselves and their software > into a position of trust in the ecosystem whereby they were almost able to > pull off the mother of all security nightmares for the world. This is entirely speculative at this point. It isn't certain that the author is the one behind the exploit, and if they were, it is not known for how long their intentions were malicious, or even what their motivations were. It is also unclear what pseudonymous accounts with what projects are associated with the attacker. You could end up being right, but it probably makes sense to at least give things a few days for more facts to become available, before making decisions to retool the entire distro. I think the bigger challenge is what could have been done to prevent this sort of problem in the first place. There are so many projects that end up with code running as root that have one or two people taking care of them, and if somebody does the work to become one of those maintainers, there aren't many people looking out for problems. I think one thing that would help here is for distros to have better ways to ensure that the code in the scm matches the code in the tarball. It is pretty common for releases to be manipulated in some way (even if only to gpg sign them, but often to switch from commit IDs to version numbers and so on), and that can be a place where stuff gets added. That still says nothing about obfuscated code, which this also involved. -- Rich
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, 2024-03-30 at 14:57 +, Eddie Chapman wrote: > Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is > wouldn't it be nice if there were at least 2 alternatives to choose from? > That doesn't have to be disruptive in any way, people who wish to continue > using and trusting xz-utils should be able to continue to do so without > any friction whatsoever. So, you're basically saying we should go out of our way, recompress all distfiles using two alternative compression formats, increase mirror load four times and add a lot of complexity to ebuilds, right? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
Rich, Duncan, Dale, orbea, you have to admit the situation with xz-utils is nothing like the typical scenario people usually worry about, where a bad actor manages to compromise a project and slip something into a widely used piece of software. No, this is the the bad actor *themselves* being a principal author of the software, working stealthily and in very sophisticated ways for years, to manoeuvrer themselves and their software into a position of trust in the ecosystem whereby they were almost able to pull off the mother of all security nightmares for the world. And many very smart people reviewed what they did and were fooled by them (which is no reflection on those people, it was just because the bad actor did a very, very good job of fooling them). I have to ask, if you still trust a codebase to be right at the heart of your system after that, what on earth would it take for you to start to feel uncomfortable??!! Sometimes, it's good when you're inside the house that is on fire, to *not* stand there and say to yourself "well the engineers who built this place must have built it to withstand a fire, I'm sure it will stop burning soon. And anyway, the fire brigade will be here soon, I'm sure it will all be fine". I'm not saying the world of OSS & Linux is on fire, of course not. This is a very isolated and rare situation with just 1 piece of software. No, I'm just using probably a probably bad analogy to make the following point: while almost all of the time a reasoned, "lets just calm down and think about this" approach is right, in some rare situations it is important to see a situation as serious as it and act accordingly. In this case, if I weigh up the benefits of using this piece of software versus another (relatively small gains in file size reduction, some gains in resource usage) against the risks of continuing to use it (and lets be realistic about those risks please rather than "I'm sure it will all be fine"), the risks are far greater. Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is wouldn't it be nice if there were at least 2 alternatives to choose from? That doesn't have to be disruptive in any way, people who wish to continue using and trusting xz-utils should be able to continue to do so without any friction whatsoever. Rich Freeman wrote: > On Sat, Mar 30, 2024 at 3:06 AM Dale wrote: > >> >> when I got to the part about it not likely to affect Gentoo, my level >> of concern dropped significantly. If this is still true, there's no >> need to be concerned. > > "not likely" is the best way to characterize this. The exploit has > not been fully analyzed, and it could have additional malicious behavior, > either designed by its author, or perhaps even unintended by its author. > > I just wanted to toss in that caveat, but agree that the defaults > deployed in Gentoo seem the most sensible for general use. There is > nothing magical about xz - ANY widely-used library could have something > like this embedded in it, and the attacker exploited what they had access > to in order to go after a configuration that was going to be widely > deployed and reachable (xz+deb/rpm+systemd+openssh). If the attacker had > an intended target that used gentoo+openrc and access to something in our > supply chain, this could have been a vulnerability that only impacted > Gentoo. > > > I think the big lesson here is that FOSS continues to suffer from core > dependencies that are challenged for resources, and that efforts to fix > this have to constantly keep up with the changing landscape. xz is going > on 15 years old, but I don't think it was nearly as commonly used until > fairly recently. > > libz has been a pretty well-known source of security flaws for ages > (granted, usually not intentional like this). It isn't too surprising > that in both cases compression libraries were targeted, as these are so > widely depended on. > > This is getting tangential, but part of me wonders if there is a > better way to do authentication. Programs like ssh tend to run as root so > that they can authenticate users and then fork and suid to the appropriate > user. Could some OS-level facility be created to allow unprivileged > processes to run the daemons and then as part of the authentication > process they can have the OS accept a controlled and minimal set of data > to create the process as the new user and hand over the connection? PAM > already has a large amount of control over the authentication process, so > it seems like we just need to change the security context that this runs > in. That's just brainstorming-level thinking though - there could be > obvious issues with this that just haven't occurred to me. > > -- > Rich > > > >
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, Mar 30, 2024 at 3:06 AM Dale wrote: > > when I got to the part about it not likely to affect Gentoo, my level of > concern dropped significantly. If this is still true, there's no need to be > concerned. "not likely" is the best way to characterize this. The exploit has not been fully analyzed, and it could have additional malicious behavior, either designed by its author, or perhaps even unintended by its author. I just wanted to toss in that caveat, but agree that the defaults deployed in Gentoo seem the most sensible for general use. There is nothing magical about xz - ANY widely-used library could have something like this embedded in it, and the attacker exploited what they had access to in order to go after a configuration that was going to be widely deployed and reachable (xz+deb/rpm+systemd+openssh). If the attacker had an intended target that used gentoo+openrc and access to something in our supply chain, this could have been a vulnerability that only impacted Gentoo. I think the big lesson here is that FOSS continues to suffer from core dependencies that are challenged for resources, and that efforts to fix this have to constantly keep up with the changing landscape. xz is going on 15 years old, but I don't think it was nearly as commonly used until fairly recently. libz has been a pretty well-known source of security flaws for ages (granted, usually not intentional like this). It isn't too surprising that in both cases compression libraries were targeted, as these are so widely depended on. This is getting tangential, but part of me wonders if there is a better way to do authentication. Programs like ssh tend to run as root so that they can authenticate users and then fork and suid to the appropriate user. Could some OS-level facility be created to allow unprivileged processes to run the daemons and then as part of the authentication process they can have the OS accept a controlled and minimal set of data to create the process as the new user and hand over the connection? PAM already has a large amount of control over the authentication process, so it seems like we just need to change the security context that this runs in. That's just brainstorming-level thinking though - there could be obvious issues with this that just haven't occurred to me. -- Rich
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
orbea wrote: > On Sat, 30 Mar 2024 03:07:13 - > "Eddie Chapman" wrote: > >> Given what we've learnt in the last 24hrs about xz utilities, you >> could forgive a paranoid person for seriously considering getting rid >> entirely of them from their systems, especially since there are >> suitable alternatives available. Some might say that's a bit >> extreme, xz-utils will get a thorough audit and it will all be fine. >> But when a malicious actor has been a key maintainer of something as >> complex as a decompression utility for years, I'm not sure I could >> ever trust that codebase again. Maybe a complete rewrite will emerge, >> but I'm personally unwilling to continue using xz utils in the >> meantime for uncompressing anything on my systems, even if it is done >> by an unprivileged process. >> >> I see that many system package ebuilds unconditionally expect >> app-arch/xz-utils to be installed simply to be able to decompress the >> source archive in SRC_URI. So simply specifying -lzma on your system >> isn't going to get rid of it. >> >> No one could have been expected to foresee what's happened with >> xz-utils, but now that it's here, perhaps Gentoo (and other projects >> that do) should consider not relying on a single decompression >> algorithm for source archives, even just as an insurance against some >> other yet unknown disaster with one algorithm or another in future? >> >> And yes I'm sure there will be individual packages that currently >> absolutely need xz-utils installed during the build process, and one >> or two that absolutely have to have it available at runtime, but those >> bridges can be crossed as and when. >> >> Eddie >> >> > I think this is an overreaction and we should wait for the dust to > settle before making drastic disruptive changes. > > >From the news item email: "Impact == Our current understanding of the backdoor is that is does not affect Gentoo systems, because 1. the backdoor only appears to be included on specific systems and Gentoo does not qualify; 2. the backdoor as it is currently understood targets OpenSSH patched to work with systemd-notify support. Gentoo does not support or include these patches; Analysis is still ongoing, however, and additional vectors may still be identified. For this reason we are still issuing this advisory as if that will be the case." When I started reading it, I was concerned as well as I know it is used on my system. However, when I got to the part about it not likely to affect Gentoo, my level of concern dropped significantly. If this is still true, there's no need to be concerned. If things has changed and it does affect Gentoo, I'm sure there will be changes made that will either fix the issue for good or at least provide a workaround until a solution is found. Gentoo has some awesome devs. Someone will find a solution. I notice that it has already been changed in the tree to a version that does not have the malicious code. That alone should be a solution until a new plan is made. While I'm a little concerned and hope for a proper solution, I'm not to worried. I certainly don't think we should overreact this early. Give the devs and upstream time to work this out. Just a users opinion. Dale :-) :-)
Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
On Sat, 30 Mar 2024 03:07:13 - "Eddie Chapman" wrote: > Given what we've learnt in the last 24hrs about xz utilities, you > could forgive a paranoid person for seriously considering getting rid > entirely of them from their systems, especially since there are > suitable alternatives available. Some might say that's a bit > extreme, xz-utils will get a thorough audit and it will all be fine. > But when a malicious actor has been a key maintainer of something as > complex as a decompression utility for years, I'm not sure I could > ever trust that codebase again. Maybe a complete rewrite will emerge, > but I'm personally unwilling to continue using xz utils in the > meantime for uncompressing anything on my systems, even if it is done > by an unprivileged process. > > I see that many system package ebuilds unconditionally expect > app-arch/xz-utils to be installed simply to be able to decompress the > source archive in SRC_URI. So simply specifying -lzma on your system > isn't going to get rid of it. > > No one could have been expected to foresee what's happened with > xz-utils, but now that it's here, perhaps Gentoo (and other projects > that do) should consider not relying on a single decompression > algorithm for source archives, even just as an insurance against some > other yet unknown disaster with one algorithm or another in future? > > And yes I'm sure there will be individual packages that currently > absolutely need xz-utils installed during the build process, and one > or two that absolutely have to have it available at runtime, but those > bridges can be crossed as and when. > > Eddie > > I think this is an overreaction and we should wait for the dust to settle before making drastic disruptive changes.
