GitHub user zhangqiang2 opened a pull request:
https://github.com/apache/spark/pull/16993
Dynamic Batch Interval Adjustment
The current Spark Streaming version cannot support the change of batch
interval at runtime, given that the speed of input data streams may not highly
dynamic from current Internet applications. If we have to do so, one must stop
the program first, modify the corresponding code, and then restart the program.
However, this will interrupt the execution of entire program, and may cause the
data loss. Towards this end, our contribution is to implement a Dynamic Batch
Interval Adjustment functionality that can help change the batch interval size
at runtime.
This functionality contains two algorithms. One is dynamic adjustment, and
the other is static adjustment. The former can predict the size of input data
stream and as a result the processing time, by using the most recent processing
time and used batch interval. In this way, one can decide whether the batch
interval needs to be changed or not, to avoid the data backlog, and secure the
system stability. On the other hand, the static adjustment needs the user to
manually change the configuration file.
JIRA Issue: [https://issues.apache.org/jira/browse/SPARK-19663](url)
My report:
[https://github.com/floatingtony/System-Lever-Optimization-of-Spark-Streaming](url)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangqiang2/spark my_change
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16993.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16993
----
commit a565a3f293c3e935ab35493e48348c40d0187705
Author: zhangqiang2 <[email protected]>
Date: 2017-02-19T13:42:54Z
âmessaynamic Batch Interval Adjustmentt
commit 7706aa9695d4f197b892d66dedc6f06232c24340
Author: zhangqiang2 <[email protected]>
Date: 2017-02-19T14:34:43Z
Dynamic Batch Interval Adjustment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
