saths008 commented on code in PR #18553:
URL: https://github.com/apache/iceberg/pull/18553#discussion_r4359759448
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java:
##########
@@ -371,7 +372,22 @@ private UTF8StringWriter(ColumnDescriptor desc) {
@Override
public void write(int repetitionLevel, UTF8String value) {
- column.writeBinary(repetitionLevel,
Binary.fromReusedByteArray(value.getBytes()));
+ long baseOffset = value.getBaseOffset();
+ if (value.getBaseObject() instanceof byte[] bytes
+ && baseOffset >= Platform.BYTE_ARRAY_OFFSET) {
+ // Write the backing array in place. UTF8String is often a slice of a
larger array, so
+ // getBytes would copy the value out before Parquet reads it. Use the
smaller wrapper for
+ // whole-array strings to avoid extra per-value allocation.
+ column.writeBinary(
+ repetitionLevel,
+ baseOffset == Platform.BYTE_ARRAY_OFFSET && value.numBytes() ==
bytes.length
+ ? Binary.fromReusedByteArray(bytes)
+ : Binary.fromReusedByteArray(
+ bytes, (int) (baseOffset - Platform.BYTE_ARRAY_OFFSET),
value.numBytes()));
+ } else {
+ // Off-heap values must be copied onto the heap.
Review Comment:
Removed comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
