LuciferYang commented on code in PR #55932:
URL: https://github.com/apache/spark/pull/55932#discussion_r3868434457
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaLengthByteArrayReader.java:
##########
@@ -54,17 +54,26 @@ public void initFromPage(int valueCount,
ByteBufferInputStream in) throws IOExce
@Override
public void readBinary(int total, WritableColumnVector c, int rowId) {
- ByteBuffer buffer;
- ByteBufferOutputWriter outputWriter =
ByteBufferOutputWriter::writeArrayByteBuffer;
- int length;
+ // Compute total data length across all values so we can read everything
in a single
Review Comment:
Following up on the magnitude: 2.00x-2.33x in the description comes from
dividing the integer-millisecond Best Time (7 to 3, and 6 to 3 on JDK 25). The
`Rate(M/s)` column in the same files is computed from the unrounded time, and
by it the four payload sizes span 1.93x-2.28x, so both ends of the quoted range
read a little high. Rounding is also why JDK 17 and JDK 21 both show 2.33x when
their baseline rates differ (155.3 and 161.1).
The mechanism argument holds regardless: post-change `skipBinary` lands at
352.7/354.7/354.8 M/s across the three files, within 0.6% of each other, while
same-group `readBinary` stays flat in all of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
