[
https://issues.apache.org/jira/browse/SPARK-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831570#comment-15831570
]
Hyukjin Kwon commented on SPARK-6802:
-------------------------------------
If this JIRA is about Python UDAF, there is a JIRA -
https://issues.apache.org/jira/browse/SPARK-10915
If this JIRA is about refactoring the UDAF in Scala/Java side, I feel I don't
know what I should fix.
{quote}
We realized that Custom Aggregate Functions aren't well supported across
various features/functions in Spark beyond what is supported by Hive
{quote}
{quote}
It would make sense to keep in consideration, the recently added DataFrame,
GroupedData, and possibly also sql.dataframe.Column, which looks different from
pandas.Series
{quote}
These two statements are too broad and vague. We should point out what to fix
or what is the concrete problem.
Please correct me if I misunderstood or am wrong.
> User Defined Aggregate Function Refactoring
> -------------------------------------------
>
> Key: SPARK-6802
> URL: https://issues.apache.org/jira/browse/SPARK-6802
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Environment: We use Spark Dataframe, SQL along with json, sql and
> pandas quite a bit
> Reporter: cynepia
>
> While trying to use custom aggregates in spark (something which is common in
> pandas), We realized that Custom Aggregate Functions aren't well supported
> across various features/functions in Spark beyond what is supported by Hive.
> There are futher discussions on the topic viz-a -viz the issue SPARK-3947,
> which points to similar improvement tickets opened earlier for refactoring
> the UDAF area.
> While we refactor the interface for aggregates, It would make sense to keep
> in consideration, the recently added DataFrame, GroupedData, and possibly
> also sql.dataframe.Column, which looks different from pandas.Series and isn't
> currently supporting any aggregations.
> Would like to get feedback from the folks, who are actively looking at this...
> We would be happy to participate and contribute, if there are any discussions
> on the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
