JunRuiLee opened a new pull request, #770:
URL: https://github.com/apache/paimon-rust/pull/770
## What
`deserialize_binary_array_str` validates only that each element's
variable-length region lies
inside the buffer — not how the regions relate to each other, nor the null
bitmap. It is the last
variable-length `BinaryArray` variant without that check: #746 added it for
the row variant, and
#755 asks for the same rule on the string variant in its own PR, before a C
entry point starts
accepting split bytes from outside the process.
## The writer's contract
`BinaryArrayWriter` (`paimon-common/.../data/BinaryArrayWriter.java`, Java
master `3e510cf132`):
- `cursor = fixedSize = roundNumberOfBytesToNearestWord(header + 8 * n)` at
construction.
- `AbstractBinaryWriter.writeString`: a body of `len <= 7` goes to the fixed
part via
`writeBytesToFixLenPart` and **leaves `cursor` alone**; anything longer
goes through
`writeBytesToVarLenPart`, which does `setOffsetAndSize(pos, cursor, len)`
then
`cursor += roundNumberOfBytesToNearestWord(len)`.
- `complete()`: `array.pointTo(segment, 0, cursor)` — the array's size
**is** the final cursor,
exact, not word-rounded again.
So bodies are consecutive and word-padded, the buffer ends at the last one,
and inline elements
consume no variable-length space. Requiring exactly that leaves no layout
the writer can emit,
which is why this needs no heuristic size cap.
## What it rejects
Three inputs the writer cannot produce:
- **Two elements pointing at one body.** Each element is cloned into a
`String` of its own and the
fixed region costs 8 bytes per element, so a body spanning most of the
buffer bounds the output
around `len²/8`.
- **A body addressed inside the fixed part** — it returns the array's own
header as a value. Those
bytes are valid UTF-8, so this decoded to a garbage string rather than
failing: the new test
reads `["\u{1}\0\0\0\0\0\0\0"]` on `main`.
- **A gap before a body, or bytes after the last one.**
Two narrower gaps, from the same reading of the writer:
- **A set null bit was ignored** and the slot read as if it held a value.
All three call sites
declare non-null elements — `_EXTRA_FILES` (`ArrayType(false,
newStringType(false))`),
`_VALUE_STATS_COLS` (`ARRAY(STRING().notNull())`), `_WRITE_COLS`
(`ArrayType(true, newStringType(false))`) — so the bit is not writer
output, and Java's reader
would return null for it. On `main` the new test reads `["hello"]`.
- **The inline marker's seven length bits** can claim up to 127 bytes while
the writer emits at
most 7 (`BinarySection.MAX_FIX_PART_DATA_SIZE`), so a length of 8 read the
marker byte itself as
content.
## Reachability
From `DataFileMeta` row decoding (`spec/data_file.rs`, fields 11/16/19),
therefore from the
`DataSplit` embedded in a `BucketVectorSearchSplit`. Every such buffer
reaching this crate today
comes from a manifest it wrote or from a trusted Java writer, so the
amplification is not
reachable from current callers — the two silent-wrong-value gaps are, given
a corrupt buffer.
## Out of scope
`deserialize_binary_array_int` and `_long` have no variable-length part, and
are already exact and
bounded respectively. The row variant is #746's.
## Testing
Seven tests; the six negative ones fail on `main` and pass here, and
`binary_array_str_all_inline_is_exactly_its_fixed_part` pins the rule that
keeps the cursor walk
from over-rejecting a legal all-inline array.
Regression evidence against real writer output: the existing goldens
(`split_v1_data.bin`, `split_v1_indexed.bin`,
`bucket_vector_search_split_v1*.bin`,
`datasplit_v8/v9.bin`) all decode `DataFileMeta` rows through this function
and still pass.
Gates from a clean HEAD: `cargo fmt --all -- --check`; `cargo clippy -p
paimon --all-targets -- -D
warnings` and with `--features fulltext`, both clean; `cargo test -p paimon
--lib` 2586 passed / 0
failed (2579 on `main`, so +7 and nothing skipped) and 2661 passed with
`--features fulltext`;
`cargo build -p paimon-datafusion` and `-p paimon-c` for the cross-crate
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
