morningman commented on PR #69581: URL: https://github.com/apache/doris/pull/69581#issuecomment-6621254086
### [P1] Avoid rebuilding the entire child validity bitmap for every Arrow output block Reviewed commit: `82260656f278db88cc48849083e19f2ccb0685df` In [`arrow_parent_nulls.cpp:181-186`](https://github.com/apache/doris/blob/82260656f278db88cc48849083e19f2ccb0685df/be/src/core/data_type_serde/arrow_parent_nulls.cpp#L181-L186), the first NULL parent in each requested range causes the helper to allocate and copy/fill a validity bitmap for the **entire backing child array**, before checking whether the parent actually has any child slots to mask. An ordinary NULL LIST or MAP with equal adjacent offsets therefore pays this cost even though no child bit needs changing. This also affects existing Arrow IPC reads: the stream reader retains a full Arrow array and repeatedly passes smaller row ranges to the SerDe. The temporary mask is discarded after each call, so a large batch with alternating `[1]` and NULL lists rebuilds the same full child bitmap for every output block. With `N` parent rows, `M` child values, and block size `B`, this adds Θ(M × ceil(N/B)) bitmap work. For example, `N = 2²⁴`, `M = 2²³`, and `B = 4096` imply 4 GiB of cumulative temporary bitmap allocations and 8 GiB of zero/fill writes. These are static calculations from the code, not benchmark results. MAP and STRUCT have related repeated-mask paths. Could we avoid allocation for empty or already-null spans and bound effective masking to the retained source batch or the consumed child interval, for example by caching a mask per source array or correctly rebasing a range-local mask? Skipping empty intervals alone does not address nonempty hidden spans or STRUCT. Please also add allocation-sensitive coverage for repeated small ranges of one large input, including empty NULL intervals and nonempty hidden spans. This is a performance regression against the merge base; no build, test, or benchmark was run as part of this review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
