GitHub user ahmed-mahran opened a pull request:
https://github.com/apache/spark/pull/14145
[SPARK-16487] [STREAMING] Fix some batches might not get marked as fully
processed in JobGenerator
## What changes were proposed in this pull request?
In `JobGenerator`, the code reads like that some batches might not get
marked as fully processed. In the following flowchart, the batch should get
marked fully processed before endpoint C however it is not. Currently, this
does not actually cause an issue, as the condition `(time - zeroTime) is
multiple of checkpoint duration?` always evaluates to `true` as the `checkpoint
duration` is always set to be equal to the `batch duration`.
This PR fixes this issue so as to improve code readability and to avoid any
potential issue in case there is any future change making checkpoint duration
to be set different from batch duration.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ahmed-mahran/spark
b-mark-batch-fully-processed
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14145.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14145
----
commit 4df7e367fa11578767f2c2fa23b40fa2cb6714c7
Author: Ahmed Mahran <[email protected]>
Date: 2016-07-12T00:43:29Z
Mark uncheckpointed batch as fully processed
The code reads like that some batches might not get marked as fully
processed. Currently, this does not actually cause an issue, as the
condition '(time - zeroTime) is multiple of checkpoint duration?'
always evaluates to true as the checkpoint duration is always set
to be equal to the batch duration.
commit 585870f04cf93fcc663fd2b910c6bb1df3ef1dc5
Author: Ahmed Mahran <[email protected]>
Date: 2016-07-12T00:46:38Z
Fix comment typo
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
