tustvold commented on PR #6608: URL: https://github.com/apache/arrow-rs/pull/6608#issuecomment-2427656780
> For DataFusion at least the alternative is to all logical_nulls().map(|n| n.null_count()) which is strictly less performant than the alternative offered here The problem is for RunArray, DictionaryArray and UnionArray computing `logical_nulls` is potentially very expensive. Now I accept that there might be a marginal performance win from a specialized logical_null_count implementation, but other than perhaps for NullArray I would expect the difference to largely be a wash. For most types it is a couple of additional atomics, or will be completely dominated by the cost of computing the logical nulls. The problem with exposing a logical_null_count method is it makes the fact this is effectively computing a fresh null mask implicit, hiding this problem. In fact this PR as written actually regresses `is_nullable` performance, demonstrating this :sweat_smile: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
