linzch3 commented on PR #12066: URL: https://github.com/apache/arrow-rs/pull/12066#issuecomment-6628251984
Thanks for running it! I pushed d06b18b to cut the overhead. Pure ASCII data can't have a multi-byte char split across a field boundary, so the per-offset `is_char_boundary` scan now only runs when the batch has non-ASCII bytes, and the scan is kept out of line. For ASCII input the only extra work is one `is_ascii` pass over the batch. Locally (x86, pinned core, subset of `csv_reader`), the geomean vs main went from +2.0% with the first commit to +0.8%. The string/mixed cases are back to baseline, and decimal32 is too at 128 and 1024 (4096 is still about +5% here, so I'd like to see whether that holds on your runner). The `*_small` integer cases are still around +1–3%, which is about the run-to-run noise on my machine. A non-ASCII batch still pays for the full scan, which I think is unavoidable without checking boundaries while decoding. Would you mind re-running the benchmark when you get a chance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
