andygrove commented on code in PR #23880:
URL: https://github.com/apache/datafusion/pull/23880#discussion_r3660602220
##########
datafusion/spark/src/function/string/quote.rs:
##########
@@ -88,34 +89,55 @@ fn spark_quote_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
fn quote_array<T: OffsetSizeTrait>(array: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_generic_string_array::<T>(array)?;
- let result = str_array
- .iter()
- .map(|s| s.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result))
+ Ok(quote_impl(str_array.iter(), str_array.value_data().len()))
Review Comment:
Confirmed — `GenericByteArray::slice` clones the whole `value_data` buffer
and narrows only the offsets, so this over-allocates by however much of the
buffer the slice excludes. Now taking the length from the sliced offsets (`last
- first`) in 4a7577fe7.
##########
datafusion/spark/src/function/string/quote.rs:
##########
@@ -88,34 +89,55 @@ fn spark_quote_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
fn quote_array<T: OffsetSizeTrait>(array: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_generic_string_array::<T>(array)?;
- let result = str_array
- .iter()
- .map(|s| s.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result))
+ Ok(quote_impl(str_array.iter(), str_array.value_data().len()))
}
fn quote_view(str_view: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_string_view_array(str_view)?;
- let result = str_array
- .iter()
- .map(|opt_str| opt_str.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result) as ArrayRef)
+ Ok(quote_impl(
+ str_array.iter(),
+ str_array.get_buffer_memory_size(),
Review Comment:
Right, and it is wrong in the other direction too: strings of 12 bytes or
fewer live inline in the views, so the data buffers can be near-empty while the
array holds plenty of data. Switched to `total_bytes_len()`, which walks the
sliced views and counts inlined values, in 4a7577fe7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
