Github user josepablocam commented on the pull request:
https://github.com/apache/spark/pull/7278#issuecomment-120178166
Indeed, it is not implemented in stats, the best implementation I found was
in the nortest library
(http://cran.r-project.org/web/packages/nortest/index.html). AD works as an
alternative to KS, and has a better ability to detect deviations at the tails
of the distributions. So the use cases are effectively the same.
While KS is very popular, there are some people who think it should be
replaced by some of the alternatives (AD being one of them). See
https://asaip.psu.edu/Articles/beware-the-kolmogorov-smirnov-test for an
example.
We implemented under a similar idea as the KS implementation, meaning,
there is a portion of the statistic that we can calculate per partition and
then adjust at the end.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
