Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1733#discussion_r15854417
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala
---
@@ -89,4 +90,76 @@ object Statistics {
*/
@Experimental
def corr(x: RDD[Double], y: RDD[Double], method: String): Double =
Correlations.corr(x, y, method)
+
+ /**
+ * :: Experimental ::
+ * Conduct the Chi-squared goodness of fit test of the observed data
against the
+ * expected distribution.
+ *
+ * Note: the two input RDDs need to have the same number of partitions
and the same number of
+ * elements in each partition.
+ *
+ * @param observed RDD[Double] containing the observed counts.
+ * @param expected RDD[Double] containing the expected counts. If the
observed total differs from
+ * the expected total, this RDD is rescaled to sum up to
the observed total.
+ * @param method String specifying the method to use for the Chi-squared
test.
+ * Supported: `pearson` (default)
+ * @return ChiSquaredTest object containing the test statistic, degrees
of freedom, p-value,
+ * the method used, and the null hypothesis.
+ */
+ @Experimental
+ def chiSquared(observed: RDD[Double],
+ expected: RDD[Double],
+ method: String): ChiSquaredTestResult = {
+ ChiSquaredTest.chiSquared(observed, expected, method)
+ }
+
+ /**
+ * :: Experimental ::
+ * Conduct the Chi-squared goodness of fit test of the observed data
against the
+ * expected distribution.
--- End diff --
mention `pearson` here?
minor: I think it should be fine to remove the rest of the doc and point
users to the method with the full set of parameters, so we only maintain one
copy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
