[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-645498297 I think I improved some of the readability problems and addressed the other comments. I'd like to merge this soon once CI is green This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-645004792 I'll have to deal with the string optimization in a follow up PR, so I'm going to leave this for review as is. It would be good to get this merged sooner rather than later. EDIT: opened https://issues.apache.org/jira/browse/ARROW-9152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-644892130 @ursabot benchmark --help This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-644870737 I implemented some other optimizations, especially for the case where neither values nor filter contain nulls. I'm working on updated benchmarks Updated benchmarks: https://gist.github.com/wesm/ad07cec1613b6327926dfe1d95e7f4f0/revisions?diff=split This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-644513681 To show some simple numbers to show the perf before and after in Python, this example has a high selectivity (all but one value selected) and low selectivity filter (1/100 and 1/1000): ``` import numpy as np import pandas as pd import pyarrow as pa import pyarrow.compute as pc string_values = pa.array([pd.util.testing.rands(16) for i in range(1)] * 100) double_values = pa.array(np.random.randn(100)) all_but_one = np.ones(len(string_values), dtype=bool) all_but_one[50] = False one_in_100 = np.array(np.random.binomial(1, 0.01, size=100), dtype=bool) one_in_1000 = np.array(np.random.binomial(1, 0.001, size=100), dtype=bool) ``` before: ``` In [2]: timeit pc.filter(double_values, one_in_100) 2.06 ms ± 41.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [3]: timeit pc.filter(double_values, one_in_1000) 1.82 ms ± 3.69 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [4]: timeit pc.filter(double_values, all_but_one) 5.75 ms ± 15.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [5]: timeit pc.filter(string_values, one_in_100) 2.23 ms ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [6]: timeit pc.filter(string_values, one_in_1000) 1.85 ms ± 3.92 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [7]: timeit pc.filter(string_values, all_but_one) 11.6 ms ± 183 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` after ``` In [4]: timeit pc.filter(double_values, one_in_100) 1.1 ms ± 7.03 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [5]: timeit pc.filter(double_values, one_in_1000) 531 µs ± 8.52 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [7]: timeit pc.filter(double_values, all_but_one) 1.83 ms ± 7.36 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [10]: timeit pc.filter(string_values, one_in_100) 1.28 ms ± 3.16 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [11]: timeit pc.filter(string_values, one_in_1000) 561 µs ± 1.69 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [12]: timeit pc.filter(string_values, all_but_one) 6.66 ms ± 34.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` EDIT: updated benchmarks for low-selectivity optimization This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7442: ARROW-9075: [C++] Optimized Filter implementation: faster performance + compilation, smaller code size
wesm edited a comment on pull request #7442: URL: https://github.com/apache/arrow/pull/7442#issuecomment-644509797 Here's benchmark runs on my machine * BEFORE: https://gist.github.com/wesm/857a3179e7dbc928d3325b1e7f687086 * AFTER: https://gist.github.com/wesm/ad07cec1613b6327926dfe1d95e7f4f0 **If you want to benchmark yourself, please use this branch for the "before":** https://github.com/wesm/arrow/tree/ARROW-9075-comparison. It contains the RandomArrayGenerator::Boolean change and some other changes to the benchmarks without which the results will be non-comparable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
