JeonDaehong commented on code in PR #1144:
URL: https://github.com/apache/paimon-rust/pull/1144#discussion_r4301841244
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
+ // not change the logical values, including in keys and nested fields.
+ // This specific cast checks offset overflow without enabling value
casts.
+ (DataType::LargeUtf8, DataType::Utf8) => {
Review Comment:
`arrow_cast` checks the slice's offsets as absolute values, so a small slice
taken past the 2 GiB mark of a LargeUtf8 array fails even though the slice
itself is tiny. I built a 2-row array (2^31 + 16 bytes, then `"abc"`) and
normalized `slice(1, 1)`:
```
LargeString array too large to cast to String array
```
The same `"abc"` as a fresh array goes through. pyarrow `to_batches()` hands
over zero-copy slices, so a large_string chunk over 2 GiB split into small
batches would hit this. Rebasing the offsets (subtract the first one, slice the
values) before the cast should fix it, like `visible_children` already does for
lists.
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
Review Comment:
Question: the same reasoning applies to `LargeBinary` -> `Binary` for BYTES
columns, and that still fails on this head with `Arrow write type LargeBinary
is incompatible with table type Binary`. Is that planned for a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
