mbutrovich commented on PR #4927: URL: https://github.com/apache/datafusion-comet/pull/4927#issuecomment-4985863155
`data_capacity` uses `values.value_data().len()`, which is the full backing buffer of the cast StringArray, not the length of the offsets actually referenced. For a sliced or offset array this over-allocates the output buffer. It is only a hint so there is no correctness impact, and it never under-allocates, but it can waste memory on sliced inputs. - Evidence: to_json.rs capacity sum uses `values.value_data().len()`; arrow-rs `value_data()` returns the entire `value_data` slice (byte_array.rs:283-285), independent of the array's offset window. - Suggested change: if precision matters, compute the used span from the offsets (`last_offset - first_offset`) instead of `value_data().len()`. Given it is a hint, an inline comment noting the deliberate over-approximation is the lighter-weight alternative. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
