GitHub user lw-lin opened a pull request:
https://github.com/apache/spark/pull/12797
[SPARK-15022][SPARK-15023] Add support for testing against the
`ProcessingTime(intervalMS > 0)` trigger and `ManualClock`
## What changes were proposed in this pull request?
Currently in `StreamTest`, we have a `StartStream` which will start a
streaming query against trigger `ProcessTime(intervalMS = 0)` and `SystemClock`.
We also need to test cases against `ProcessTime(intervalMS > 0)`, which
often requires `ManualClock`.
This patch:
- fixes an issue of `ProcessingTimeExecutor`, where for a batch it should
run `batchRunner` only once but might run multiple times under certain
conditions;
- adds support for testing against the `ProcessingTime(intervalMS > 0)`
trigger and `ManualClock`, by specifying them as fields for `StartStream`, and
by adding an `AdvanceClock` action;
- adds a test, which takes advantage of the new `StartStream` and
`AdvanceClock`, to test against [PR#[SPARK-14942] Reduce delay between batch
construction and execution ](https://github.com/apache/spark/pull/12725).
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lw-lin/spark add-trigger-test-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12797.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12797
----
commit 90ed69285fc34ae43a0f454ceb25837618212e28
Author: Liwei Lin <[email protected]>
Date: 2016-04-30T01:32:48Z
fix an issue of nextBatchTime against ManualClock
commit 9d80b15e33151f5f87207d72f89f016c18a21b01
Author: Liwei Lin <[email protected]>
Date: 2016-04-30T01:33:48Z
Add support for testing against ProcessingTime(intervalMS>0)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
