Thanks for providing input, Mantas. I appreciate the second opinion. I'll try building an RPM without the patch (or at least with a new one re-instating the LZ4 preprocessor directives) to confirm it has an impact on the issue I'm investigating.
Robert Winslow Dalpe On Tue, Sep 24, 2019 at 10:27 PM Mantas Mikulėnas <[email protected]> wrote: > On Wed, Sep 25, 2019 at 6:32 AM Robert Winslow Dalpe <[email protected]> > wrote: > >> Hello, >> >> I'm trying to track down the reason for an issue I'm seeing between >> systemd-journald and how rsyslog where data is being truncated when read >> from the journal file ( https://github.com/rsyslog/rsyslog/issues/3784 ). >> >> The behavior seems present in systemd 219 (on amazon linux 2 and centos >> 7) and not in later versions (on arch linux). I figured the truncation may >> be due to the default threshold for sd_journal_get_data ( >> https://www.freedesktop.org/software/systemd/man/sd_journal_get_data.html >> ). From what I can tell, it looks like the only time sd_journal_get_data >> uses the threshold is when calling `decompress_blob` ( >> https://github.com/systemd/systemd/blob/a88abde72169ddc2df77df3fa5bed30725022253/src/journal/sd-journal.c#L1916 >> ). `decompress_blob` looks to see if the data is compressed using XZ or LZ4 >> and then decompresses appropriately ( >> https://github.com/systemd/systemd/blob/a88abde72169ddc2df77df3fa5bed30725022253/src/journal/compress.c#L214 >> ). It looks like only the XZ decompression code utilizes the threshold >> (`dst_max`--which contains the threshold value--is unused in >> `decompress_blob_lz4`). >> >> From what I can tell LZ4 compression is the default. Assuming I'm on the >> right track, I'm trying to determine how XZ decompression would be invoked. >> That led me to the RPM that centos7 (and thus presumably amazon linux 2) >> builds for systemd. That RPM applies the following patch: >> https://git.powerel.org/powerel7/base/src/commit/e9d94968812dc1b501e6c691ee8c01595d77f76d/SOURCES/0575-Add-support-to-read-lz4-compressed-journals.patch >> Although the patch says it "adds support to read lz4 compressed journals," >> it looks to me like it is actually making XZ compression the default by >> removing several preprocessor directives that would allow for LZ4 >> compression/decompression. Am I interpreting that patch correctly? Does any >> of what I've said seem even close to sane? I realize I can remove that >> patch and try building an RPM myself, but I'm hoping someone might know >> enough to provide any quick feedback before I go down that road. >> > > Without the patch, XZ is already the default. The patch adds LZ4 > decompression support, but tries to avoid making LZ4 the *default* format > for compression, making sure that e.g. compress_lz4 remains NULL despite > the LZ4 library being present. (I'm guessing one reason is that the LZ4 > library already changed its framing format once, and CentOS wants to stick > with the stable xz format?) > > -- > Mantas Mikulėnas >
_______________________________________________ systemd-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/systemd-devel
