Rich-T-kid commented on PR #10441: URL: https://github.com/apache/arrow-rs/pull/10441#issuecomment-5373705846
> I I found some other way to make OOB checks faster in other PRs (parquet) > > * do the computation in small chunks (8-16) > * check if any exceeds the value (return error if this is the case) > * then perform the work on the chunk > This reduces the checking overhead 8-16x, mostly removing it. > > Perhaps this is a good strategy going forward @Rich-T-kid ? @Dandandan I did a couple of attempts over the past few days and the benchmarks are always within noise. I believe a factor in this comes down to the benchmarks using very small arrays; `[512,1024]`. I think it'd be interesting to tweak the benchmarks to use 8192 just to reflect what datafusion uses by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
