neilconway opened a new issue, #20458:
URL: https://github.com/apache/datafusion/issues/20458
### Describe the bug
* `ScalarValue::to_array_of_size` calls `list_to_array_of_size` for
list-like inputs
* `list_to_array_of_size` will be called with a one-element list containing
an Arrow array type, and it does:
```rust
let arrays = repeat_n(arr, size).collect::<Vec<_>>();
let ret = match !arrays.is_empty() {
true => arrow::compute::concat(arrays.as_slice())?,
false => arr.slice(0, 0),
};
```
* If the input is a `StringViewArray`, `repeat_n` will create `size` copies,
each with their own data buffers. `concat` preserves those data buffers. So if
the input is, say, a `StringViewArray` with 500 buffers and `size` is 1024, the
result will have 500k buffers.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
We probably didn't see this problem before because DF doesn't usually call
`ScalarValue::to_array_of_size` on a list whose underlying StringViewArray has
many data buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
