I knew about lossless-compressions with navigation capabilities, but CLP <https://github.com/y-scope/clp> is new to me. Thanks for the pointer.
For those interested, I want to add some more context into "binary serialization" subject: - LOG4J2-3023 <https://issues.apache.org/jira/browse/LOG4J2-3023> proposes a layout wrapper compressing the output of the delegate - LOG4J2-1305 <https://issues.apache.org/jira/browse/LOG4J2-1305> (and LOG4J2-836 <https://issues.apache.org/jira/browse/LOG4J2-836>) proposes a binary layout – AFAIC, there is a proliferation of serialization options (Avro, Protocol Buffers, CBOR, CLP, etc.) and this makes any specialized solution opinionated. I have my doubts if just picking one is the right way forward. - OpenHFT Chronicle Logger <https://github.com/OpenHFT/Chronicle-Logger> implements an efficient file appender with built-in binary layout for SLF4J, Logback, Log4j 1, Log4j 2, JUL, and JCL. On Thu, Jan 18, 2024 at 11:16 PM Remko Popma <remko.po...@gmail.com> wrote: > > https://www.uber.com/en-JP/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/ > > tldr: > Uber created a CLP > < > https://github.com/y-scope/clp?uclick_id=e2bbfdc9-f79f-4d7a-999b-6e3c3ea19364 > > > appender for log4j that compresses on the fly. > Impressive 169x compression rate. > Resulting compressed files are searchable because of embedded dictionaries. >
