felipecrv opened a new pull request, #43190:
URL: https://github.com/apache/arrow/pull/43190
## Rationale for this change
When arrays using 32-bit offsets into data buffers are concatenated and the
data buffers of the results grow beyond 2GB, `Concatenate` returns a bad
`Status` with a very simple message:
`"offset overflow while concatenating arrays"`
The contract that `Concatenate` honors is very simple: arrays of input type
T lead to output of the same type T, so we can't, for instance, return a
`LARGE_STRING` [1] array when the input is `STRING`.
But we can **suggest a cast** to the caller in case an overflow error is
detected. Either programatically (by taking an output parameter) or by giving a
better error message to users.
[1] `LARGE_STRING` can use 64-bit offsets
### What changes are included in this PR?
- Suggest casts when concatenation of the values of an FSL fail due to
overflow
- Suggest casts when concatenation of [LARGE_]LIST_VIEW array fails due to
overflow
- Suggest casts when concatenation of [LARGE_]LIST array fails due to
overflow
- Suggest a cast to LARGE_(BINARY|STRING) when offsets overflow
### Are these changes tested?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
