[
https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852929#comment-15852929
]
koert kuipers edited comment on SPARK-19428 at 2/4/17 8:57 PM:
---------------------------------------------------------------
generalizing to return top x (or first x) by some sorting is straightforward,
but you will have to write it yourself.
for small x this is best done with an aggregator that internally holds
something like a priority queue. this will push most of the work map-side.
for large x the most efficient way is to collect all data reduce-side per group
and sort it (aka secondary sort). this is currently somewhat awkward to do in
spark-sql, but i wrote a small library that does this for you:
https://github.com/tresata/spark-sorted
was (Author: koert):
generalizing to return top-x by some sorting is straightforward, but you will
have to write it yourself.
for small x this is best done with an aggregator that internally holds
something like a priority queue. this will push most of the work map-side.
for large x the most efficient way is to collect all data reduce-side per group
and sort it (aka secondary sort). this is currently somewhat awkward to do in
spark-sql, but i wrote a small library that does this for you:
https://github.com/tresata/spark-sorted
> Ability to select first row of groupby
> --------------------------------------
>
> Key: SPARK-19428
> URL: https://issues.apache.org/jira/browse/SPARK-19428
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Luke Miner
> Priority: Minor
>
> It would be nice to be able to select the first row from {{GroupedData}}.
> Pandas has something like this:
> {{df.groupby('group').first()}}
> It's especially handy if you can order the group as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
