yongster opened a new issue, #11082: URL: https://github.com/apache/arrow-rs/issues/11082
### Is your feature request related to a problem or challenge? `arrow_cast::base64::b64_encode` sizes the output from every physical offset pair and calls `encode_slice` for every row, including rows already masked out by the null bitmap: https://github.com/apache/arrow-rs/blob/master/arrow-cast/src/base64.rs `GenericByteArray::value` documents that the physical bytes of a null slot are arbitrary but well-defined. Variable-width arrays may legally retain non-empty payloads in null slots. This is not only a hand-built case: `filter` on byte arrays **explicitly copies** null-slot bytes so the validity bitmap can be handled separately (`filter_bytes` in `arrow-select`). The result is that encode: 1. computes an encoded length for unobservable null payloads 2. allocates those bytes in the output values buffer 3. actually Base64-encodes them `b64_decode` already skips decoding null rows, but it still sizes the output buffer from `array.values().len()` (the full physical buffer) and does not truncate the unused tail. At high null density with retained payloads, that is a large zeroed allocation that is never used. Logical values are unchanged if null slots get a zero output length: physical bytes in null slots are not part of the array's logical content, and array equality already ignores them. ### Describe the solution you'd like - When the input has nulls, give null rows output length 0 and skip `encode_slice` for them. - Keep a no-null fast path so the common dense case does not pay a per-row validity check. - For decode, size the buffer from valid input bytes (or truncate to the actual decoded length) so unused tail is not retained. - Preserve the current generic `i32`/`i64` offset API. - Add unit coverage for arrays that retain non-empty payloads in null slots, plus a Criterion bench. I independently remeasured on arm64 macOS against `381eea177`, 16,384 rows × 1,024-byte values, `BASE64_STANDARD`. Null slots in the "retained" case keep the full 1,024-byte payload. Logical encode output of a skip-null prototype matched the current kernel. | Case | Current | Skip-null prototype | Speedup | |---|---:|---:|---:| | encode, 50% null retained | 4.081 ms | 2.012 ms | 2.03× | | encode, 90% null retained | 3.930 ms | 434 µs | 9.07× | | encode, 99% null retained | 3.888 ms | 76 µs | 51.2× | | encode, no nulls | 4.001 ms | 3.935 ms | ~1.02× (no regression) | | decode, 99% null retained | 188 µs | 82 µs | 2.31× | | decode, no nulls | 4.292 ms | 4.277 ms | unchanged | On current encode, a 99% null array with **compact** (empty) null slots is already ~36× faster than the same logical array with retained payloads, which is the wasted work this change removes. I can open a PR covering encode and decode together: same file, same retained-payload semantics, shared tests. ### Describe alternatives you've considered - Compacting null payloads in `filter`/`concat` instead. That would help more kernels, but it is a much larger behavior change, and encode would still be wrong for FFI / manually constructed arrays. - Only truncating the decode buffer and leaving encode alone. That misses the large CPU cost on encode. - Always checking validity even with `null_count() == 0`. A dedicated no-null path is cheaper and matches other kernels. ### Additional context Related but not the same: #10284 / #10324 made `b64_encode` reject invalid UTF-8 from a misbehaving `Engine`. No open issue or PR currently covers skipping null payloads. There is currently no `base64_encode` bench in `arrow-cast`; a PR should add one, including a retained-null-payload case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
