MonsterChenzhuo opened a new pull request, #7854:
URL: https://github.com/apache/paimon/pull/7854
### Purpose
When writing a very wide primary-key table, the write path may fail during
commit with an unclear `NegativeArraySizeException`:
```text
26/05/13 09:08:37 WARN TaskSetManager: Lost task 1.0 in stage 6.0 (TID 160)
(10.166.23.92 executor 12): java.lang.NegativeArraySizeException
at
org.apache.paimon.memory.MemorySegmentUtils.getBytes(MemorySegmentUtils.java:191)
at org.apache.paimon.data.BinarySection.toBytes(BinarySection.java:101)
at
org.apache.paimon.format.parquet.writer.ParquetRowDataWriter$StringWriter.writeString(ParquetRowDataWriter.java:299)
at
org.apache.paimon.mergetree.SortBufferWriteBuffer.forEach(SortBufferWriteBuffer.java:162)
at
org.apache.paimon.mergetree.MergeTreeWriter.flushWriteBuffer(MergeTreeWriter.java:224)
at
org.apache.paimon.mergetree.MergeTreeWriter.prepareCommit(MergeTreeWriter.java:253)
.....
```
One root cause is that the table schema is too wide for the configured
page-size. BinaryRow requires its fixed-length part to fit into one memory
segment. If the fixed-length part is larger than page-size, later write/flush
code can read invalid string offsets or lengths and surface a misleading
low-level exception.
### Tests
paimon-core/src/test/java/org/apache/paimon/mergetree/SortBufferWriteBufferValidationTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
