neilconway commented on code in PR #23880:
URL: https://github.com/apache/datafusion/pull/23880#discussion_r3650733629
##########
datafusion/spark/src/function/string/quote.rs:
##########
@@ -88,34 +89,55 @@ fn spark_quote_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
fn quote_array<T: OffsetSizeTrait>(array: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_generic_string_array::<T>(array)?;
- let result = str_array
- .iter()
- .map(|s| s.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result))
+ Ok(quote_impl(str_array.iter(), str_array.value_data().len()))
Review Comment:
`str_array.value_data().len()` will over-allocate for sliced arrays.
##########
datafusion/spark/src/function/string/quote.rs:
##########
@@ -88,34 +89,55 @@ fn spark_quote_inner(arg: &[ArrayRef]) -> Result<ArrayRef> {
fn quote_array<T: OffsetSizeTrait>(array: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_generic_string_array::<T>(array)?;
- let result = str_array
- .iter()
- .map(|s| s.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result))
+ Ok(quote_impl(str_array.iter(), str_array.value_data().len()))
}
fn quote_view(str_view: &ArrayRef) -> Result<ArrayRef> {
let str_array = as_string_view_array(str_view)?;
- let result = str_array
- .iter()
- .map(|opt_str| opt_str.map(compute_quote))
- .collect::<StringArray>();
- Ok(Arc::new(result) as ArrayRef)
+ Ok(quote_impl(
+ str_array.iter(),
+ str_array.get_buffer_memory_size(),
Review Comment:
Looks like `get_buffer_memory_size` sums the buffer capacities, not their
actual valid contents, so this will also over-allocate for sliced arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
