js8544 commented on issue #37055: URL: https://github.com/apache/arrow/issues/37055#issuecomment-1774404687
Thanks! Since the original file requires registration and some other verificaiton processes, I downloaded a [demo file](https://physionet.org/content/mimic-iv-demo/2.2/hosp/labevents.csv.gz) with about 100K rows. Nevertheless I was able to optimize `value_counts()` to the same level as `combine_chunks().values_counts()`: ```python # Before 1.04 ms ± 6.88 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) 625 µs ± 19.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) # After 642 µs ± 4.94 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) 610 µs ± 2.71 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) ``` I'll write a formal C++ benchmark to further verify and send a PR shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
