GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/19843
[SPARK-22644][ML][TEST][WIP] Make ML testsuite support StructuredStreaming
test
## What changes were proposed in this pull request?
We need to add some helper code to make testing ML transformers & models
easier with streaming data. These tests might help us catch any remaining
issues and we could encourage future PRs to use these tests to prevent new
Models & Transformers from having issues.
I add a `MLTest` trait which extends `StreamTest` trait, and override
`createSparkSession`. So ML testsuite can only extend `MLTest`, to use both ML
& Stream test util functions.
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WeichenXu123/spark ml_stream_test_helper
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19843.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19843
----
commit 072f4b9f330af61e7de07c8c2e57421448aa306b
Author: WeichenXu <[email protected]>
Date: 2017-11-28T14:04:35Z
init pr
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
