[
https://issues.apache.org/jira/browse/PARQUET-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17682895#comment-17682895
]
ASF GitHub Bot commented on PARQUET-2159:
-----------------------------------------
wgtmac commented on PR #1011:
URL: https://github.com/apache/parquet-mr/pull/1011#issuecomment-1411584288
> > > Sorry for the delay. I have left some comments and the implementation
is overall looking good. Thanks @jiangjiguang for your effort!
> > > My main concern is the extensibility to support other instruction
sets. In addition, it seems to me that the java vector api is still incubating.
As I am not a java expert, do we have the risk of unstable API?
> >
> >
> > @wgtmac Jatin is a java expert， @jatin-bhateja Can you help give an
answer? thanks.
>
> Hi @wgtmac , our patch vectorizes unpacking algorithm for various decode
bit sizes, entire new functionality is exposed through a plugin interface
**ParquetReadRouter**, in order to prevent any performance regressions over
other targets we have enabled the new functionality only for X86 targets with
valid features, this limitation can be removed over time.
>
> VectorAPI made its appearance in JDK16 and has been maturing since then
with each successive release. I do not have a firm timeline for you at this
point on its incubation exit and being exposed as a preview feature. Intent
here is to enable parquet-mr community developers to make use of the plugin in
parquet reader and provide us with early feedback, we are also in process of
vectorizing packer algorithm.
>
> Being a large project we plan to do this incrementally, we seek your
guidance in pushing this patch through either on master or a separate
development branch.
Thanks for your explanation @jatin-bhateja!
So when vector API is finalized in the future java release, we may need to
change the VM options to enable it accordingly.
BTW, I may not be able to verify the generated code line by line. Please
advice the best practice to test them according to your experience. Thanks
@jatin-bhateja
> Parquet bit-packing de/encode optimization
> ------------------------------------------
>
> Key: PARQUET-2159
> URL: https://issues.apache.org/jira/browse/PARQUET-2159
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.13.0
> Reporter: Fang-Xie
> Assignee: Fang-Xie
> Priority: Major
> Fix For: 1.13.0
>
> Attachments: image-2022-06-15-22-56-08-396.png,
> image-2022-06-15-22-57-15-964.png, image-2022-06-15-22-58-01-442.png,
> image-2022-06-15-22-58-40-704.png
>
>
> Current Spark use Parquet-mr as parquet reader/writer library, but the
> built-in bit-packing en/decode is not efficient enough.
> Our optimization for Parquet bit-packing en/decode with jdk.incubator.vector
> in Open JDK18 brings prominent performance improvement.
> Due to Vector API is added to OpenJDK since 16, So this optimization request
> JDK16 or higher.
> *Below are our test results*
> Functional test is based on open-source parquet-mr Bit-pack decoding
> function: *_public final void unpack8Values(final byte[] in, final int inPos,
> final int[] out, final int outPos)_* __
> compared with our implementation with vector API *_public final void
> unpack8Values_vec(final byte[] in, final int inPos, final int[] out, final
> int outPos)_*
> We tested 10 pairs (open source parquet bit unpacking vs ours optimized
> vectorized SIMD implementation) decode function with bit
> width=\{1,2,3,4,5,6,7,8,9,10}, below are test results:
> !image-2022-06-15-22-56-08-396.png|width=437,height=223!
> We integrated our bit-packing decode implementation into parquet-mr, tested
> the parquet batch reader ability from Spark VectorizedParquetRecordReader
> which get parquet column data by the batch way. We construct parquet file
> with different row count and column count, the column data type is Int32, the
> maximum int value is 127 which satisfies bit pack encode with bit width=7,
> the count of the row is from 10k to 100 million and the count of the column
> is from 1 to 4.
> !image-2022-06-15-22-57-15-964.png|width=453,height=229!
> !image-2022-06-15-22-58-01-442.png|width=439,height=217!
> !image-2022-06-15-22-58-40-704.png|width=415,height=208!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
