emkornfield commented on pull request #7143: URL: https://github.com/apache/arrow/pull/7143#issuecomment-638966384
> My anecdotal experience suggests that 1% is a higher-than-average rate of nulls, but I don't know how accurate that is. The popcount strategy could be adapted to do 64 bits at a time (instead of the 256 that's implemented now) which would mean a higher incidence of no-nulls blocks in the 1% case. @wesm Yeah, it would still expect some performance improvements with this approach at 1% at 64 bit blocks (roughly 48% of blocks would still use this path). I don't think there is necessarily a scientific way of determining this. We should probably just draw a line in the sand. I'm willing to revisit the approach in this PR to use BitBlockCounter once it is checked in if you and @pitrou would prefer to avoid complexity here. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
