When the streaming compression interfaces were originally implemented in 2018 [1], there was not a distinction between LZ4 "raw" compression (which is Compression::LZ4) and LZ4 "frame" compression (Compression::LZ4_FRAME). So in that patch while the LZ4 raw compression method was being used for one-shot compression, the frame format was being used for streaming.
Commingling these codec types proved problematic, so it was necessary later (in 0.17.0 [2]) to distinguish the codecs so they can be invoked separately for one-shot compression, while streaming compression is only available for the frame format. [1]: https://github.com/apache/arrow/commit/65f03168f20c9429768f64b761209365a99cbada [2]: https://github.com/apache/arrow/commit/895f22009f945371fb18d00324cc5eb853dda51a On Tue, Jul 6, 2021 at 5:08 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > > > > > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow > > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe > > the meaning has changed between versions. > > > Sorry for the late reply. yes, I think there have been a few rounds to > disambiguate this frmo 0.16.0 through 3.0.0 > > On Tue, Jun 22, 2021 at 12:45 PM Rares Vernica <rvern...@gmail.com> wrote: > > > Thanks for the pointer. > > > > Interestingly arrow::Compression::type::LZ4 worked with streams in Arrow > > 0.16.0, but triggers this NotImprelmented exception in Arrow 3.0.0. Maybe > > the meaning has changed between versions. > > > > On Tue, Jun 22, 2021 at 12:33 PM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > > > > It sounds like you are trying to use the buffer level compression > > described > > > by the specification? If so only LZ4_FRAME is currently allowed [1] > > > > > > [1] https://github.com/apache/arrow/blob/master/format/Message.fbs#L45 > > > > > > > > > On Tue, Jun 22, 2021 at 12:28 PM Rares Vernica <rvern...@gmail.com> > > wrote: > > > > > > > Hello, > > > > > > > > Using Arrow 3.0.0 I tried to compress a stream with LZ4 and got this > > > error > > > > message: > > > > > > > > NotImplemented: Streaming compression unsupported with LZ4 raw format. > > > Try > > > > using LZ4 frame format instead. > > > > > > > > Is it because LZ4 raw was not enabled when the .so was compiled or > > > actually > > > > not implemented? > > > > > > > > Is LZ4 frame the recommended format for compressing stream/IPC buffers? > > > > > > > > Thank you! > > > > Rares > > > > > > > > >
