[
https://issues.apache.org/jira/browse/ARROW-10423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou closed ARROW-10423.
----------------------------------
Resolution: Done
It appears the performance gap was fixed in ARROW-10696.
> [C++] Filter compute function seems slow compared to numpy nonzero + take
> -------------------------------------------------------------------------
>
> Key: ARROW-10423
> URL: https://issues.apache.org/jira/browse/ARROW-10423
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: compute
>
> From
> https://stackoverflow.com/questions/64581590/is-there-a-more-efficient-way-to-select-rows-from-a-pyarrow-table-based-on-conte
> I made a smaller, simplified example:
> {code:python}
> arr = pa.array(np.random.randn(1_000_000))
> # mask with only few True values
> mask1 = np.zeros(len(arr), dtype=bool)
> mask1[np.random.randint(len(arr), size=100)] = True
> mask1_pa = pa.array(mask1)
> # mask with larger proportion of True values
> mask2 = np.zeros(len(arr), dtype=bool)
> mask2[np.random.randint(len(arr), size=10_000)] = True
> mask2_pa = pa.array(mask2)
> {code}
> Doing timings of doing a Arrow {{Filter}} kernel vs using numpy to convert
> the mask into indices and then using a {{Take}} kernel:
> {code}
> # mask 1
> In [3]: %timeit arr.filter(mask1_pa)
> 132 µs ± 4.44 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> In [4]: %%timeit
> ...: indices = np.nonzero(mask1)[0]
> ...: arr.take(indices)
> 114 µs ± 2.62 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> # mask 2
> In [8]: %timeit arr.filter(mask2_pa)
> 711 µs ± 63.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> In [9]: %%timeit
> ...: indices = np.nonzero(mask2)[0]
> ...: arr.take(indices)
> 333 µs ± 6.32 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> {code}
> So in the first case, both are quite similar in timing. But in the second
> case, the numpy+take version is faster.
> I know this might depend on a lot on the actual proportion of True values and
> how they are positioned in the array (random vs concentrated) etc, so there
> is probably not a general rule of what should be faster.
> But, it still seems a potential indication that things can be optimized in
> the Filter kernel.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
