zhongyujiang opened a new pull request, #8840:
URL: https://github.com/apache/paimon/pull/8840
## Problem
Wide tables (thousands of columns) throw a cryptic
`NegativeArraySizeException` during compaction / sort spill:
```
java.lang.NegativeArraySizeException
at
org.apache.paimon.memory.MemorySegmentUtils.getBytes(MemorySegmentUtils.java:186)
at org.apache.paimon.data.BinarySection.toBytes(BinarySection.java:101)
at
org.apache.paimon.format.parquet.writer.ParquetRowDataWriter$StringWriter.writeString(ParquetRowDataWriter.java:268)
```
## Root Cause
`BinaryRow`'s fixed-length part (`nullBitsSizeInBytes + 8 * arity`) must
reside within a single `MemorySegment`, because all field accessors
(`getString`, `getLong`, `getArray`, etc.) read from `segments[0]` via
`UNSAFE.getLong` **without bounds checking**.
The default `page-size` is 64 KB, which supports up to **8064 columns**
(fixed part = 65528 bytes). When a table exceeds this limit (e.g. 8347 columns
→ fixed part = 67824 bytes), the row silently spans segments during paged
sort/spill operations. `segments[0]` is only 64 KB, so field offsets beyond 64
KB read out-of-bounds heap memory via `UNSAFE`. The garbage `long` is
interpreted as a string length; when negative, `new byte[negative]` throws
`NegativeArraySizeException`.
`BinaryRowSerializer.checkSkipWriteForFixLengthPart` only `advance()`s to
the next page when the current page lacks space — it never validates that the
fixed part fits in a single page, and assumes `advance()` will always help
(which it can't when `fixedPart > pageSize`).
## Fix
Add a `checkArgument` guard at the top of `checkSkipWriteForFixLengthPart` —
the choke point for all paged writes (`BinaryInMemorySortBuffer`,
`InMemoryBuffer`, `BytesHashMap`, `SimpleObjectsCache`,
`InternalRowSerializer`). When `getSerializedRowFixedPartLength() >
segmentSize`, throw a clear `IllegalArgumentException` at the first write,
advising to increase `page-size`.
This converts the cryptic `NegativeArraySizeException` (which may surface
far from the root cause, or not surface at all for INT-only schemas — silent
data corruption) into an immediate, actionable error.
## Testing
- `BinaryRowSerializerPageSizeGuardTest`: 200 INT columns (fixed part 1636
bytes), verifies the guard throws with a clear message at `page-size=1024` and
succeeds at `page-size=4096` with round-trip.
- `WriterOperatorTest`: bumped `page-size` from 32 b to 64 b in 3 test
methods whose KeyValue combined rows (52/60 bytes fixed part) previously
exceeded 32 b and relied on silent corruption.
## Workaround (for existing tables)
```sql
ALTER TABLE <table> SET TBLPROPERTIES ('page-size' = '256kb');
```
`page-size` must be larger than the row's fixed-length part (`≈ 8.125 *
num_columns` bytes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
