[
https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965298#comment-14965298
]
Jihong MA commented on SPARK-10646:
-----------------------------------
[~mengxr] to add chi-squared test support through UDAF framework, I will need
to keep around a HashMap for tracking counts for each category encountered, I
prototyped it using ImperativeAggregate interface, and realized current UDAF
infrastructure doesn't support varied length of aggregation attribute buffer
and cause GC pressure as well even with the support in place. I discussed with
[~yhuai] offline on this sometime back. it looks adding it through UDAF is not
feasible at this point, please kindly let me know how we should proceed? thanks!
> Bivariate Statistics: Pearson's Chi-Squared goodness of fit test
> ----------------------------------------------------------------
>
> Key: SPARK-10646
> URL: https://issues.apache.org/jira/browse/SPARK-10646
> Project: Spark
> Issue Type: Sub-task
> Components: ML, SQL
> Reporter: Jihong MA
> Assignee: Jihong MA
>
> Pearson's chi-squared goodness of fit test for observed against the expected
> distribution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
