Re: Security lessons from liblzma - libsystemd

2024-04-15 Thread Michael Paquier
On Fri, Apr 12, 2024 at 09:00:11AM -0700, Andres Freund wrote: > I'm actually fairly bothered by us linking to libxml2. It was effectively > unmaintained for most of the last decade, with just very occasional drive-by > commits. And it's not that there weren't significant bugs or such. Maintenance

Re: Security lessons from liblzma - libsystemd

2024-04-12 Thread Andres Freund
Hi, On 2024-04-04 01:10:20 +0200, Peter Eisentraut wrote: > On 03.04.24 23:19, Magnus Hagander wrote: > > When the code is this simple, we should definitely consider carrying it > > ourselves. At least if we don't expect to need *other* functionality > > from the same library in the future, which

Re: Security lessons from liblzma - libsystemd

2024-04-12 Thread Magnus Hagander
On Thu, Apr 4, 2024 at 1:10 AM Peter Eisentraut wrote: > On 03.04.24 23:19, Magnus Hagander wrote: > > When the code is this simple, we should definitely consider carrying it > > ourselves. At least if we don't expect to need *other* functionality > > from the same library in the future, which I

Re: Security lessons from liblzma

2024-04-09 Thread Joe Conway
On 4/8/24 22:57, Bruce Momjian wrote: On Sat, Mar 30, 2024 at 04:50:26PM -0400, Robert Haas wrote: An awful lot of what we do operates on the principle that we know the people who are involved and trust them, and I'm glad we do trust them, but the world is full of people who trusted somebody

Re: Security lessons from liblzma

2024-04-08 Thread Bruce Momjian
On Sat, Mar 30, 2024 at 04:50:26PM -0400, Robert Haas wrote: > An awful lot of what we do operates on the principle that we know the > people who are involved and trust them, and I'm glad we do trust them, > but the world is full of people who trusted somebody too much and > regretted it

Re: Security lessons from liblzma

2024-04-08 Thread Jacob Champion
On Fri, Apr 5, 2024 at 5:14 PM Michael Paquier wrote: > Saying that, my spidey sense tingles at the recent commit > 3311ea86edc7, that had the idea to introduce a 20k line output file > based on a 378 line input file full of random URLs. In my experience, > tests don't require to be that large

Re: Security lessons from liblzma - libsystemd

2024-04-08 Thread Étienne BERSAC
Hi, > There are many more interesting and scary libraries in the dependency > tree of "postgres", so just picking off one right now doesn't really > accomplish anything.  The next release of libsystemd will drop all > the compression libraries as hard dependencies, so the issue in that > sense is

Re: Security lessons from liblzma

2024-04-07 Thread Ranier Vilela
Hi, Sorry, without any connection with the technical part of the thread. But I couldn't help but record this, and congratulate Andres Freund, for the excellent work. It's not every day that a big press, in Brazil or around the world, publishes something related to technology people. Today I came

Re: Security lessons from liblzma

2024-04-05 Thread Michael Paquier
On Fri, Apr 05, 2024 at 11:40:46AM -0700, Jacob Champion wrote: > On Fri, Apr 5, 2024 at 6:24 AM Robert Haas wrote: >> I'm also sort of afraid that we're getting sucked into thinking real >> hard about this SSL certificate issue rather than trying to brainstorm >> all the other places that might

Re: Security lessons from liblzma

2024-04-05 Thread Jacob Champion
On Fri, Apr 5, 2024 at 6:24 AM Robert Haas wrote: > I wonder how hard it would be to just code up our own binary to do > this. If it'd be a pain to do that, or to maintain it across SSL > versions, then it's a bad plan and we shouldn't do it. But if it's not > that much code, maybe it'd be worth

Re: Security lessons from liblzma

2024-04-05 Thread Robert Haas
On Thu, Apr 4, 2024 at 4:48 PM Daniel Gustafsson wrote: > AFAIK we haven't historically enforced that installations have the openssl > binary in PATH, but it would be a pretty low bar to add. The bigger issue is > likely to find someone to port this to Windows, it probably won't be too hard >

Re: Security lessons from liblzma

2024-04-04 Thread Tom Lane
Jelte Fennema-Nio writes: > Okay turns out even generating them in parallel isn't any faster than > running that sequentially. I guess it's because of the strong random > generation being the slow part. Command I used was the following and > took ~5 seconds on my machine: > make -C src/test/ssl

Re: Security lessons from liblzma

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 23:06, Daniel Gustafsson wrote: > > > On 4 Apr 2024, at 23:02, Jelte Fennema-Nio wrote: > > > How about we make it meson/make targets, so they are simply cached > > just like any of our other build artefacts are cached. Then only clean > > builds are impacted, not every

Re: Security lessons from liblzma

2024-04-04 Thread Tom Lane
Jelte Fennema-Nio writes: > How about we make it meson/make targets, so they are simply cached > just like any of our other build artefacts are cached. Then only clean > builds are impacted, not every test run. Every buildfarm and CI run is "clean" in those terms.

Re: Security lessons from liblzma

2024-04-04 Thread Daniel Gustafsson
> On 4 Apr 2024, at 23:02, Jelte Fennema-Nio wrote: > How about we make it meson/make targets, so they are simply cached > just like any of our other build artefacts are cached. Then only clean > builds are impacted, not every test run. They already are (well, make not meson yet), they're just

Re: Security lessons from liblzma

2024-04-04 Thread Jelte Fennema-Nio
On Thu, 4 Apr 2024 at 22:56, Daniel Gustafsson wrote: > > > On 4 Apr 2024, at 22:47, Tom Lane wrote: > > > > Robert Haas writes: > >> On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: > >>> I don't disagree, like I said that very email: it's non-trivial and I > >>> wish we > >>> could

Re: Security lessons from liblzma

2024-04-04 Thread Greg Sabino Mullane
> > It would be better if we created the required test files as part of the > test run. (Why not? Too slow?) Alternatively, I have been thinking > that maybe we could make the output more reproducible by messing with > whatever random seed OpenSSL uses. Or maybe use a Python library to >

Re: Security lessons from liblzma

2024-04-04 Thread Bruce Momjian
On Thu, Apr 4, 2024 at 10:56:01PM +0200, Daniel Gustafsson wrote: > > On 4 Apr 2024, at 22:47, Tom Lane wrote: > > > > Robert Haas writes: > >> On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: > >>> I don't disagree, like I said that very email: it's non-trivial and I > >>> wish we >

Re: Security lessons from liblzma

2024-04-04 Thread Daniel Gustafsson
> On 4 Apr 2024, at 22:47, Tom Lane wrote: > > Robert Haas writes: >> On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: >>> I don't disagree, like I said that very email: it's non-trivial and I wish >>> we >>> could make it better somehow, but I don't hav an abundance of good ideas. >

Re: Security lessons from liblzma

2024-04-04 Thread Daniel Gustafsson
> On 4 Apr 2024, at 22:40, Robert Haas wrote: > > On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: >> I don't disagree, like I said that very email: it's non-trivial and I wish we >> could make it better somehow, but I don't hav an abundance of good ideas. > > Is the basic issue that we

Re: Security lessons from liblzma

2024-04-04 Thread Tom Lane
Robert Haas writes: > On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: >> I don't disagree, like I said that very email: it's non-trivial and I wish we >> could make it better somehow, but I don't hav an abundance of good ideas. > Is the basic issue that we can't rely on the necessary

Re: Security lessons from liblzma

2024-04-04 Thread Robert Haas
On Thu, Apr 4, 2024 at 4:25 PM Daniel Gustafsson wrote: > I don't disagree, like I said that very email: it's non-trivial and I wish we > could make it better somehow, but I don't hav an abundance of good ideas. Is the basic issue that we can't rely on the necessary toolchain to be present on

Re: Security lessons from liblzma

2024-04-04 Thread Daniel Gustafsson
> On 4 Apr 2024, at 21:38, Robert Haas wrote: > Essentially, your argument is the same as his, namely: hey, don't > worry, you could totally verify these test files if you wanted to! But > of course, nobody did, because it was hard, and everybody had better > things to do with their time. And I

Re: Security lessons from liblzma

2024-04-04 Thread Robert Haas
On Wed, Apr 3, 2024 at 3:42 PM Daniel Gustafsson wrote: > > So you just have to "trust" that what's there now makes sense. > > Not entirely, you can review the input files which are used to generate the > test data and verify that those make sense.. Yeah, I mean, in theory I suppose that's true,

Re: Security lessons from liblzma - libsystemd

2024-04-03 Thread Peter Eisentraut
On 03.04.24 23:19, Magnus Hagander wrote: When the code is this simple, we should definitely consider carrying it ourselves. At least if we don't expect to need *other* functionality from the same library in the future, which I doubt we will from libsystemd. Well, I've long had it on my list

Re: Security lessons from liblzma - libsystemd

2024-04-03 Thread Andres Freund
Hi, On 2024-04-03 17:58:55 -0400, Tom Lane wrote: > Magnus Hagander writes: > > On Wed, Apr 3, 2024 at 7:57 PM Andres Freund wrote: > >> Openssh has now integrated [1] a patch to remove the dependency on > >> libsystemd > >> for triggering service manager readyness notifications, by inlining

Re: Security lessons from liblzma - libsystemd

2024-04-03 Thread Tom Lane
Magnus Hagander writes: > On Wed, Apr 3, 2024 at 7:57 PM Andres Freund wrote: >> Openssh has now integrated [1] a patch to remove the dependency on >> libsystemd >> for triggering service manager readyness notifications, by inlining the >> necessary function. That's not hard, the protocol is

Re: Security lessons from liblzma - libsystemd

2024-04-03 Thread Magnus Hagander
On Wed, Apr 3, 2024 at 7:57 PM Andres Freund wrote: > Hi, > > As most will know by now, the way xz debacle was able to make sshd > vulnerable > was through a dependency from sshd to libsystemd and then from libsystemd > to > liblzma. One lesson from this is that unnecessary dependencies can

Re: Security lessons from liblzma

2024-04-03 Thread Daniel Gustafsson
> On 3 Apr 2024, at 20:09, Peter Eisentraut wrote: > > On 30.03.24 00:14, Daniel Gustafsson wrote: >> One take-away for me is how important it is to ship recipes for regenerating >> any testdata which is included in generated/compiled/binary format. Kind of >> how we in our tree ship the config

Re: Security lessons from liblzma

2024-04-03 Thread Peter Eisentraut
On 30.03.24 00:14, Daniel Gustafsson wrote: One take-away for me is how important it is to ship recipes for regenerating any testdata which is included in generated/compiled/binary format. Kind of how we in our tree ship the config for test TLS certificates and keys which can be manually

Re: Security lessons from liblzma - libsystemd

2024-04-03 Thread Andres Freund
Hi, As most will know by now, the way xz debacle was able to make sshd vulnerable was through a dependency from sshd to libsystemd and then from libsystemd to liblzma. One lesson from this is that unnecessary dependencies can still increase risk. It's worth noting that we have an optional

Re: Security lessons from liblzma

2024-04-01 Thread Andrew Dunstan
On 2024-03-31 Su 17:12, Andres Freund wrote: Hi, On 2024-03-31 12:18:29 +0200, Michael Banck wrote: If you ask where they are maintained, the answer is here: https://salsa.debian.org/postgresql/postgresql/-/tree/17/debian/patches?ref_type=heads the other major versions have their own

Re: Security lessons from liblzma

2024-04-01 Thread Tom Lane
Bruce Momjian writes: > On Mon, Apr 1, 2024 at 03:17:55PM -0400, Tom Lane wrote: >> AFAIK, every open-source distro makes all the pieces needed to >> rebuild their packages available to users. It wouldn't be much >> of an open-source situation otherwise. You do have to learn >> their package

Re: Security lessons from liblzma

2024-04-01 Thread Bruce Momjian
On Mon, Apr 1, 2024 at 03:17:55PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I was more asking if users have access to patches so they could recreate > > the build by using the Postgres git tree and supplied OS-specific > > patches. > > AFAIK, every open-source distro makes all the

Re: Security lessons from liblzma

2024-04-01 Thread Bruce Momjian
On Fri, Mar 29, 2024 at 06:37:24PM -0400, Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and

Re: Security lessons from liblzma

2024-04-01 Thread Tom Lane
Bruce Momjian writes: > I was more asking if users have access to patches so they could recreate > the build by using the Postgres git tree and supplied OS-specific > patches. AFAIK, every open-source distro makes all the pieces needed to rebuild their packages available to users. It wouldn't

Re: Security lessons from liblzma

2024-04-01 Thread Bruce Momjian
On Sun, Mar 31, 2024 at 02:12:57PM -0700, Andres Freund wrote: > Hi, > > On 2024-03-31 12:18:29 +0200, Michael Banck wrote: > > If you ask where they are maintained, the answer is here: > > > > https://salsa.debian.org/postgresql/postgresql/-/tree/17/debian/patches?ref_type=heads > > > > the

Re: Security lessons from liblzma

2024-04-01 Thread David E. Wheeler
On Apr 1, 2024, at 06:55, walt...@technowledgy.de wrote: > Also a configurable directoy to look up extensions, possibly even to be > changed at run-time like [2]. The patch says this: > >> This directory is prepended to paths when loading extensions (control and >> SQL files), and to the

Re: Security lessons from liblzma

2024-04-01 Thread walther
I looked through the repositories of 19 linux distros [1] to see what kind of patches are applied often. Many of them share the same package managers / repositories and thus the same patches. I made sure to look at some smaller, "other" distros as well, to see what kind of problems appear

Re: Security lessons from liblzma

2024-03-31 Thread Andres Freund
Hi, On 2024-03-31 12:18:29 +0200, Michael Banck wrote: > If you ask where they are maintained, the answer is here: > > https://salsa.debian.org/postgresql/postgresql/-/tree/17/debian/patches?ref_type=heads > > the other major versions have their own branch. Luckily these are all quite small,

Re: Security lessons from liblzma

2024-03-31 Thread Tom Lane
Michael Banck writes: > On Sun, Mar 31, 2024 at 01:05:40PM -0400, Joe Conway wrote: >> But it has always bothered me how many patches get applied to the upstream >> tarballs by the OS package builders. > I think this more an artifact of how RHEL development works, i.e. trying > to ship the same

Re: Security lessons from liblzma

2024-03-31 Thread Michael Banck
Hi, On Sun, Mar 31, 2024 at 01:05:40PM -0400, Joe Conway wrote: > But it has always bothered me how many patches get applied to the upstream > tarballs by the OS package builders. Some of them, e.g. glibc on RHEL 7, > include more than 1000 patches that you would have to manually vet if you >

Re: Security lessons from liblzma

2024-03-31 Thread Joe Conway
On 3/31/24 11:49, Tom Lane wrote: Joe Conway writes: I am saying maybe those patches should be eliminated in favor of our tree including build options that would produce the same result. I don't really see how that can be expected to work sanely. It turns the responsibility for

Re: Security lessons from liblzma

2024-03-31 Thread Devrim Gündüz
Hi, On Sat, 2024-03-30 at 21:52 -0400, Bruce Momjian wrote: > How would someone access the out-of-tree patches?  Here are the v17 patches: https://git.postgresql.org/gitweb/?p=pgrpms.git;a=tree;f=rpm/redhat/main/non-common/postgresql-17/main You can replace -17 with -16 (and etc) for the other

Re: Security lessons from liblzma

2024-03-31 Thread Tom Lane
Joe Conway writes: > I am saying maybe those patches should be eliminated in favor of our > tree including build options that would produce the same result. I don't really see how that can be expected to work sanely. It turns the responsibility for platform-specific build issues on its head,

Re: Security lessons from liblzma

2024-03-31 Thread Devrim Gündüz
Hi, On Sun, 2024-03-31 at 08:15 -0400, Joe Conway wrote: > > I am saying maybe those patches should be eliminated in favor of our > tree including build options that would produce the same result. Works for me, as a long as I can commit them and upcoming potential patches to PostgreSQL git

Re: Security lessons from liblzma

2024-03-31 Thread Joe Conway
On 3/30/24 21:52, Bruce Momjian wrote: On Sat, Mar 30, 2024 at 07:54:00PM -0400, Joe Conway wrote: Virtually every RPM source, including ours, contains out of tree patches that get applied on top of the release tarball. At least for the PGDG packages, it would be nice to integrate them into our

Re: Security lessons from liblzma

2024-03-31 Thread Michael Banck
On Sat, Mar 30, 2024 at 09:52:47PM -0400, Bruce Momjian wrote: > On Sat, Mar 30, 2024 at 07:54:00PM -0400, Joe Conway wrote: > > Virtually every RPM source, including ours, contains out of tree patches > > that get applied on top of the release tarball. At least for the PGDG > > packages, it would

Re: Security lessons from liblzma

2024-03-30 Thread Bruce Momjian
On Sat, Mar 30, 2024 at 07:54:00PM -0400, Joe Conway wrote: > Virtually every RPM source, including ours, contains out of tree patches > that get applied on top of the release tarball. At least for the PGDG > packages, it would be nice to integrate them into our git repo as build > options or

Re: Security lessons from liblzma

2024-03-30 Thread Tom Lane
Joe Conway writes: > On 3/30/24 19:54, Joe Conway wrote: >> Tom follows this, at least last time I checked: >> https://wiki.postgresql.org/wiki/Release_process > Reading through that, I wonder if this part is true anymore: >In principle this could be done anywhere, but again there's a

Re: Security lessons from liblzma

2024-03-30 Thread Joe Conway
On 3/30/24 19:54, Joe Conway wrote: On 2024-03-30 16:50:26 -0400, Robert Haas wrote: or what Tom does when he builds the release tarballs. Tom follows this, at least last time I checked: https://wiki.postgresql.org/wiki/Release_process Reading through that, I wonder if this part is true

Re: Security lessons from liblzma

2024-03-30 Thread Joe Conway
On 3/30/24 17:12, Andres Freund wrote: Hi, On 2024-03-30 16:50:26 -0400, Robert Haas wrote: We might also want to move toward signing commits and tags. One of the meson maintainers was recommending that on-list not long ago. I don't know how valuable the commit signing really is, but I

Re: Security lessons from liblzma

2024-03-30 Thread Bruce Momjian
On Sat, Mar 30, 2024 at 04:50:26PM -0400, Robert Haas wrote: > On Fri, Mar 29, 2024 at 7:00 PM Andres Freund wrote: > > I am doubtful that every committer would find something sneaky hidden in > > e.g. one of the test changes in a large commit. It's not too hard to hide > > something sneaky. I

Re: Security lessons from liblzma

2024-03-30 Thread Andres Freund
Hi, On 2024-03-30 16:50:26 -0400, Robert Haas wrote: > We might also want to move toward signing commits and tags. One of the > meson maintainers was recommending that on-list not long ago. I don't know how valuable the commit signing really is, but I strongly agree that we should sign both tags

Re: Security lessons from liblzma

2024-03-30 Thread Robert Haas
On Fri, Mar 29, 2024 at 7:00 PM Andres Freund wrote: > I am doubtful that every committer would find something sneaky hidden in > e.g. one of the test changes in a large commit. It's not too hard to hide > something sneaky. I comparison to that hiding something in configure.ac seems > less likely

Re: Security lessons from liblzma

2024-03-29 Thread Tom Lane
Daniel Gustafsson writes: > One take-away for me is how important it is to ship recipes for regenerating > any testdata which is included in generated/compiled/binary format. IMO that's a hard, no-exceptions requirement just for maintainability's sake, never mind security risks.

Re: Security lessons from liblzma

2024-03-29 Thread Daniel Gustafsson
> On 29 Mar 2024, at 23:59, Andres Freund wrote: > On 2024-03-29 18:37:24 -0400, Bruce Momjian wrote: >> Now, we don't take pull requests, and all our committers are known >> individuals, but this might have cautionary lessons for us. > > I am doubtful that every committer would find something

Re: Security lessons from liblzma

2024-03-29 Thread Andres Freund
Hi, On 2024-03-29 18:37:24 -0400, Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and Ubuntu. The

Re: Security lessons from liblzma

2024-03-29 Thread Thomas Munro
On Sat, Mar 30, 2024 at 11:37 AM Bruce Momjian wrote: > You might have seen reports today about a very complex exploit added to > recent versions of liblzma. Fortunately, it was only enabled two months > ago and has not been pushed to most stable operating systems like Debian > and Ubuntu. The

Security lessons from liblzma

2024-03-29 Thread Bruce Momjian
You might have seen reports today about a very complex exploit added to recent versions of liblzma. Fortunately, it was only enabled two months ago and has not been pushed to most stable operating systems like Debian and Ubuntu. The original detection report is: