GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/9670
[SPARK-10086] [PySpark] [MLlib] Fix flaky StreamingKMeans PySpark test
The test sometimes fails because `predictOn` would occur before `trainOn`
with the input stream. This test is testing that the model gets updated and is
able to predict correctly with the updated model. This fix allows more
flexibility about when the update happens by repeating the final batch as the
default RDD in the queueStream and checking that the last result matches the
expected prediction of the updated model, within a timeout.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark
flaky-StreamingKMeans-test-SPARK-10086
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9670.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9670
----
commit 44c9679d64c06954c9b2c2c61c85d5f3924f7444
Author: Bryan Cutler <[email protected]>
Date: 2015-11-12T18:37:07Z
[SPARK-10086] Fix flaky StreamingKMeans PySpark test by being flexible when
model update occurs on the input stream
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
