Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1733#issuecomment-51298178
@dorx I checked R's implementation and finally figured out what is going on.
1. When only a vector `x` is given, it is treated as a vector containing
frequency counts for categories and tested against multinomial distribution.
2. When a matrix `x` is given, it is treated as a contingency table and the
test is for independence.
3. When both `x` and `y` are given, both vectors are treated as factors
(categorical values) and the test is for independence.
I want to suggest the following APIs:
~~~
// test observed frequencies against multinomial distribution with
// `p = (1/n, 1/n, ..., 1/n)`
def chiSqTest(counts: Vector)
// test observed frequencies against the given multinomial distribution
def chiSqTest(counts: Vector, p: Vector)
// test independence using the given contingency table
def chiSqTest(counts: Matrix)
// test independence using the given observed pairs (assuming categorical
values)
def chiSqTest[V1, V2](observations: RDD[(V1, V2)])
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
