GitHub user frreiss opened a pull request:

    [SPARK-17386] [STREAMING] [WIP] Make polling rate adaptive

    ## What changes were proposed in this pull request?
    This change makes the scheduler in `StreamExecution` adjust its rate of 
polling to the rate of data arrival. I used the 
algorithm, aka TCP congestion avoidance, because it is simple. I renamed
    `spark.sql.streaming.pollingDelay` to `spark.sql.streaming.minPollingDelay` 
and added a second parameter `spark.sql.streaming.maxPollingDelay` to serve as 
an upper bound. This upper bound is necessary because the data arrival rate 
could be bursty, with infinite variance.
    This approach works, but I'm not completely satisfied with the current 
design. The blocking-system-call part of polling really ought to be delegated 
to the sources themselves, so that the main scheduler thread won't be blocked 
for unpredictable amounts of time. Also, the polling rate ought to be adaptive 
on a per-source basis, instead of there being a single global rate per session. 
Leaving this PR marked as WIP for now.
    ## How was this patch tested?
    I tested for three scenarios:
    * If data is not present, the scheduler ramps up the polling delay quickly 
to `spark.sql.streaming.maxPollingDelay`
    * If data arrives quickly, the scheduler keeps its polling delay pinned at 
    * If new batches arrive at a constant interval between the minimum and 
maximum delay, the scheduler keeps its polling delay within a factor of two of 
the actual arrival rate.
    The testing code is included in this PR. Note that the test cases for the 
two latter cases are disabled by default, because they are somewhat timing 

You can merge this pull request into a Git repository by running:

    $ git pull fred-17386a

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15162
commit 54a8272f1810514ec0a1e92e222359f505c5e58b
Author: frreiss <>
Date:   2016-09-10T05:15:22Z

    Made polling rate adaptive.

commit 24a1f74ff5c401e39858d887212fe4acc5b17bca
Author: frreiss <>
Date:   2016-09-14T22:04:19Z

    Added test case and corrected some bugs.

commit 9b44439d784e51ed75f4476b4432d13155dcdfb0
Author: frreiss <>
Date:   2016-09-14T22:26:59Z

    Merge branch 'master' of into fred-17386a

commit 4439d07e79ca48b5037ecb50017013c75172f6af
Author: frreiss <>
Date:   2016-09-20T16:09:29Z

    Merge branch 'master' of into fred-17386a


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to