[
https://issues.apache.org/jira/browse/SPARK-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-15656.
-------------------------------
Resolution: Not A Problem
That's not what a uniform distribution is -- the expected frequencies are not
equal in your example. The code is correct right now since it uses
{{Array.tabulate(size)(_ => 1.0 / size)}}. You'd have to provide a concrete
example; I can imagine getting a statistic of 1 from lots of distributions, and
easily, in a degenerate example. If you mean that a statistic of 1 implies that
the distribution matches the hypothesis of a uniform distribution, it doesn't.
0 would mean an exact match.
> ChiSqTest for goodness of fit doesn't test against a wrong uniform
> distribution by default
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-15656
> URL: https://issues.apache.org/jira/browse/SPARK-15656
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.5.1, 1.6.1
> Reporter: Jieyuan Chen
> Labels: easyfix, mllib, stats
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> I've been running a ChiSqTest to test whether my samples fit a uniform
> distribution.
> The documentation says that If a second vector to test against is not
> supplied as a parameter, the test runs against a uniform distribution. But
> when I pass samples drawn from a normal distribution, the p-value calculated
> is 1.0, which is wrong.
> The problem is that in ChiSqTest.scala, the `chiSquared` function will
> generate a wrong uniform distribution if the expected vector is not supplied.
> The default generated samples should be
> val expArr = if (expected.size == 0) Array.tabulate(size)(i => i.toDouble /
> size) else expected.toArray
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
