This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 7c88505f9e fix(arrow-string): sum view element lengths in usize when
concatenating (#11318)
7c88505f9e is described below
commit 7c88505f9e3bb5b5275aa2838e7ef0757962964b
Author: pawan <[email protected]>
AuthorDate: Sat Oct 3 03:10:36 2026 +0000
fix(arrow-string): sum view element lengths in usize when concatenating
(#11318)
### Which issue does this PR close?
- Closes #11317.
### Rationale for this change
the two element lengths are u32 and were added as u32, so a pair that
sums past u32::MAX wrapped before the size check could see it. the check
passed on the wrapped number and the result came back with an empty
value where the data belongs.
### What changes are included in this PR?
the addition is done in usize. nothing else moves, the existing i32::MAX
check now sees the real total and rejects it.
### Are these changes tested?
yes, two 2 GiB elements concatenated. the test is ignored by default
since it wants around 8 GiB, run it with cargo test --release -p
arrow-string -- --ignored. without the fix it returns Ok with an empty
element, with the fix it returns the overflow error.
### Are there any user-facing changes?
concat of view elements that together pass u32::MAX returns an error now
instead of wrong data.
---
arrow-string/src/concat_elements.rs | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arrow-string/src/concat_elements.rs
b/arrow-string/src/concat_elements.rs
index 0a7e76f045..fd83b45d22 100644
--- a/arrow-string/src/concat_elements.rs
+++ b/arrow-string/src/concat_elements.rs
@@ -277,23 +277,23 @@ where
let null_buffer = NullBuffer::union(left.nulls(), right.nulls());
// Compute the required data buffer size, excluding any elements that
are null
- // or are small enough to be stored inline.
- let data_size = match &null_buffer {
+ // or are small enough to be stored inline. Both lengths are `u32`,
and a
+ // pair of them can sum past that, so the addition is done in `usize`.
+ let inline_max = MAX_INLINE_VIEW_LEN as usize;
+ let data_size: usize = match &null_buffer {
None => left
.lengths()
.zip(right.lengths())
- .map(|(l, r)| l + r)
- .filter(|len| *len > MAX_INLINE_VIEW_LEN)
- .map(|len| len as usize)
+ .map(|(l, r)| l as usize + r as usize)
+ .filter(|len| *len > inline_max)
.sum(),
Some(nb) => left
.lengths()
.zip(right.lengths())
.zip(nb.iter())
.filter(|((_, _), not_null)| *not_null)
- .map(|((l, r), _)| l + r)
- .filter(|len| *len > MAX_INLINE_VIEW_LEN)
- .map(|len| len as usize)
+ .map(|((l, r), _)| l as usize + r as usize)
+ .filter(|len| *len > inline_max)
.sum(),
};
