[
https://issues.apache.org/jira/browse/ARROW-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435627#comment-17435627
]
Weston Pace commented on ARROW-14254:
-------------------------------------
I think we'd need two passes then. First, get the size of the dataset.
Second, some kind of "streaming take". Alternatively, some googling suggests
the following alternatives from SQL:
1. Generate a random value for each row, sort by this column, then use top-k
2. Grab a random percent of rows with something like "SELECT * FROM table WHERE
rand() < 0.1"
3. Some SQL servers have a
[TABLESAMPLE](https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/ms189108(v=sql.105)?redirectedfrom=MSDN)
command but it looks like that just randomly selects batches (similar to #2)
and then applies top-k to that result. This means you tend to get clusters of
values so I don't see what's better than #2.
4. An alternative to 1 & 2, if you don't have rand() implemented, is to hash
the index column and then AND it with some mask (this limits the percentages
you can apply to 1/power-of-two, but you can then chop down to the requested
size).
> [C++] Return a random sample of rows from a query
> -------------------------------------------------
>
> Key: ARROW-14254
> URL: https://issues.apache.org/jira/browse/ARROW-14254
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Nicola Crane
> Priority: Major
> Labels: kernel, query-engine
> Fix For: 7.0.0
>
>
> Please can we have a kernel that returns a random sample of rows? We've had a
> request to be able to do this in R:
> https://github.com/apache/arrow-cookbook/issues/83
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
