[
https://issues.apache.org/jira/browse/ARROW-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254708#comment-16254708
]
ASF GitHub Bot commented on ARROW-1559:
---------------------------------------
wesm commented on issue #1266: ARROW-1559: [C++] Add Unique kernel and refactor
DictionaryBuilder to be a stateful kernel
URL: https://github.com/apache/arrow/pull/1266#issuecomment-344807083
So we're about a factor of 2 or so away from `pandas.unique`:
```
In [1]: import pandas as pd
import numpy as np
import pyarrow as pa
...:
In [2]: arr = pa.array(list(range(10000)) * 100)
In [3]: v = arr.to_pandas().copy()
In [4]: timeit arr.unique()
pd.unique(100 loops, best of 3: 10.4 ms per loop
In [5]: timeit pd.unique(v)
100 loops, best of 3: 4.82 ms per loop
```
I suspect this is because of our hash function being slow, but we'll need to
do some more analysis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [C++] Kernel implementations for "unique" (compute distinct elements of array)
> ------------------------------------------------------------------------------
>
> Key: ARROW-1559
> URL: https://issues.apache.org/jira/browse/ARROW-1559
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Assignee: Uwe L. Korn
> Labels: Analytics, pull-request-available
> Fix For: 0.8.0
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)