morningman opened a new pull request, #65789:
URL: https://github.com/apache/doris/pull/65789
### Problem
When an Arrow Flight result column's data reaches 2GB within a single batch,
`FromBlockToRecordBatchConverter` silently rebuilt that column's array as
`large_utf8`/`large_binary` (int64 offsets) while the RecordBatch schema kept
`utf8`/`binary` (int32 offsets). The batch then violates the Arrow spec — its
array offsets (int64) disagree with its declared schema type (int32):
- generic Arrow clients (pyarrow, Flight SQL JDBC, …) read the int64 offset
buffer as int32 → corrupted string values;
- the Doris receiver (`read_column_from_arrow`, cross-cluster `type=doris`
catalog) derives negative lengths → out-of-bounds `memcpy` / crash.
Arrow Flight sends one fixed schema for the whole stream, so a per-batch type
swap can never be correct.
### Fix
Keep the schema authoritative and never swap array types:
- Remove the silent `large_utf8`/`large_binary` upgrade in the converter.
- Add `convert_to_arrow_batches()`: an O(1) fast-path check (no cost on the
normal path); when a string/binary column would exceed the int32 offset
limit, split the block by rows so every emitted batch stays valid `utf8`.
A single value that alone exceeds the limit returns a clear error instead
of
corrupting data or crashing.
- The Arrow Flight local/remote readers drain the (possibly several) batches
produced from one block.
- `Validate()` the RecordBatch after building it, so any future schema/array
divergence fails loudly here instead of silently corrupting downstream.
- Replace the hard-coded 2GB constant with BE config
`arrow_flight_result_max_utf8_bytes` (default 2^31) so tests can exercise
the
split path without materializing 2GB of data.
Columns under the limit are emitted exactly as before (`utf8`/`binary`) — no
wire change on the normal path; only the previously-broken >2GB path changes.
### Testing
- BE unit tests (`DataTypeSerDeArrowTest`): split-and-round-trip byte
identity,
no-split under the limit, single-value-over-limit error, and split driven
by
the tightest column.
- `arrow_flight_sql_p0` regression passes unchanged (no wire change on the
normal path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
