liningpan opened a new issue, #23763: URL: https://github.com/apache/datafusion/issues/23763
### Describe the bug The upper bound of the output buffer size for binary view is estimated using its memory buffer size. https://github.com/apache/datafusion/blob/050046a7f0c08d64d8dba9ebf0e3544cbda6cd38/datafusion/functions/src/encoding/inner.rs#L297 This is incorrect as it can underestimate required memory when the BinaryView contains significant duplication and views point to overlapping regions in the buffers. base64 decode will fail with `Output slice too small` due to this bad estimate. ### To Reproduce The following test case could trigger this bug `datafusion/datafusion/functions/src/encoding/inner.rs` ```rust #[test] fn test_estimate_view_size() { let mut builder = BinaryViewBuilder::new().with_deduplicate_strings(); for _ in 0..1000 { builder.append_value([65u8; 64]); } let arr = ArrayBuilder::finish(&mut builder); println!("{}", arr.get_buffer_memory_size()); Encoding::Base64Padded .decode_array::<_, i32>(&arr.as_binary_view(), arr.get_buffer_memory_size()) .unwrap(); } ``` ### Expected behavior _No response_ ### Additional context This bug might have a much larger blast radius. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
