1111nit opened a new issue, #3073: URL: https://github.com/apache/parquet-java/issues/3073
### Describe the usage question you have. Please include as many useful details as possible. Hello, I have recently become interested in using vector API Parquet Bit-Packing decode. In the course of researching the code, I found that in the ByteBitPackingVectorBenchmarks.java in the official test benchmarks in the parquet-plugins-benchmarks folder totalByteCountVector = totalBytesCount - inputByteCountPerVector; When this range is exceeded, unpack8Values() is used to decode the data, which ensures that there is enough room for a full vector operation at the end. But in readBatchUsing512Vector() totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512; I'm wondering if this affects the performance practicalities and logical implementation choices of different bit-width decoding methods. My queries are as follows: 1. In this case, if it is the same amount of data, the number of vectorized decoding will be reduced, won't this affect the optimization effect after combining with Spark? For example, bitwidth = 3, outputValues = 2048; when ByteBitPackingVectorBenchmarks.java totalByteCountVector = totalBytesCount - inputByteCountPerVector. Here the decoding is done by 63 times unpackValuesUsingVector and 4 times unpack8value. When totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512; here the decoding is done by 59 times unpackValuesUsingVector, 20 times unpack8value.  2. I would like to ask here to reserve 64 bytes is to take into account the data out of bounds and other data security considerations? In readBatchUsing512Vector actual code can be used totalByteCountVector = totalBytesCount - inputByteCountPerVector; here should readBatchUsing512Vector prevail or the logic in the benchmark prevail? Should this be based on the readBatchUsing512Vector or the logic in the benchmark? 3. Also if readBatchUsing512Vector keeps totalByteCountVector = totalBytesCount - BYTES_PER_VECTOR_512; the vectorization bounds here are enough to cover off the bounds-safe case, Loads a vector from an array of type byte[] starting at an offset and using a mask is necessary? In my understanding, if the performance loss from using a mask is greater than the static ByteVector fromArray(VectorSpecies<Byte> species, byte[] a, int offset); which loads the excess of the array in a way that is guaranteed to be boundary-safe, the performance consumption then. Would it be possible to consider just Loads a vector from an array of type byte[] starting at an offset, eliminating the use of mask? That would just add some extra data to the vector? Or is there some other reason here why I can't eliminate mask,and am just very eagerly awaiting an answer.Sorry for my poor English.  ### Component(s) Core, Benchmark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
