samarthjain opened a new issue #647: Prefetch parquet data pages URL: https://github.com/apache/incubator-iceberg/issues/647 While doing a hot spot analysis of the benchmark runs we have for vectorized read path, reading and decompressing of the parquet data pages stood out as one of the major hot spots. We can significantly improve performance if we can take out this read/decompress from the hot code path by prefetching the next data page asynchronously. Preliminary benchmark runs showed up to 60% performance gains when using prefetching. An initial implementation is here - https://github.com/samarthjain/incubator-iceberg/tree/samarth_prefetch_vectorized This is going to need some Parquet level changes as the underlying native compressors used are not thread safe. https://github.com/apache/parquet-mr/pull/671 adds support for airlift based compressors which are written in pure Java and don't use any JNI resources.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
