Would the scattering and gathering byte channel APIs in java.nio be helpful in splitting up a stream into chunks for parallel processing?
On 10 May 2017 at 02:57, Stefan Bodewig <bode...@apache.org> wrote: > On 2017-05-10, Gary Gregory wrote: > > > I think the question is can/should [Compress] use any of the stock code > > in java.util.zip in a multi-threaded fashion for performance gains. > > We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by > itself. > > But we could implement the same strategy pigz uses, which is to break up > the stream into chunks and work on the chunks in parallel. Combining the > output of several streams may become tricky using the Java API. > > If my first read of the comments in > https://github.com/madler/pigz/blob/master/pigz.c is correct then we'd > need to manipulate the output of Deflater in order to strip headers and > trailers and insert empty stored blocks as well as create headers and > trailers of our own for the combined output. > > In theory we could implement something like pigz on top of the LZ77 > support I've added for Snappy and LZ4 (and some additional Hufmann code > yet to be written) but it would be slower than zlib - probably a lot - > and likely eat up the speed gain provided by parallel processing. > > Stefan > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > -- Matt Sicker <boa...@gmail.com>
