[
https://issues.apache.org/jira/browse/SPARK-53885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032877#comment-18032877
]
Dongjoon Hyun commented on SPARK-53885:
---------------------------------------
To improve the visibility of this feature, I linked this to SPARK-51166 and
marked as `Resolved` according to the subtasks. I believe we can add more
subtasks (if needed) until we release Apache Spark 4.1.0 release. Thank you,
[~yhuang95] and [~Gengliang.Wang].
> Frequency estimation functions
> ------------------------------
>
> Key: SPARK-53885
> URL: https://issues.apache.org/jira/browse/SPARK-53885
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Gengliang Wang
> Assignee: Yuchuan Huang
> Priority: Major
> Fix For: 4.1.0
>
>
> Introduce the following frequency estimation functions:
> * *APPROX_TOP_K(expr[, k[, maxItemsTracked]])* – Returns an approximate list
> of the top _k_ most frequent values in the input expression using a
> probabilistic algorithm.
> * *APPROX_TOP_K_ACCUMULATE(expr[, maxItemsTracked])* – Creates a state
> object that accumulates frequency statistics for use in later estimation or
> combination.
> * *APPROX_TOP_K_ESTIMATE(state [, k] ])* – Extracts and returns the
> approximate top _k_ values and their estimated frequencies from an
> accumulated state.
> * *APPROX_TOP_K_COMBINE(expr[, maxItemsTracked])* – Merges intermediate
> APPROX_TOP_K state objects, allowing distributed or parallel computation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
