GitHub user koeninger opened a pull request:

    [SPARK-17813][SQL][KAFKA] Maximum data per trigger

    ## What changes were proposed in this pull request?
    maxOffsetsPerTrigger option for rate limiting, proportionally based on 
volume of different topicpartitions.
    This is assuming SPARK-17812 is merged first due to common changes in test 
utils, if that ends up not being the case I can clean this up as a separate 
    ## How was this patch tested?
    Added unit test

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-17813

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15527
commit c45ded7109474fcb40f03c772192eb38398f328a
Author: cody koeninger <>
Date:   2016-10-14T04:23:02Z

    [SPARK-17812][SQL][KAFKA] parse json for topicpartitions and offsets

commit 12d3988c4fcef9bbbd88ce69295d2ff3e5baa5ba
Author: cody koeninger <>
Date:   2016-10-14T19:58:08Z

    Merge branch 'master' into SPARK-17812

commit 3120fd8ade24140777c29fc1487aa3f6e76152fb
Author: cody koeninger <>
Date:   2016-10-14T21:37:35Z

    [SPARK-17812][SQL][KAFKA] implement specified offsets and assign

commit 35bb8c3cfe77f2cb3d26f4afd3364caa6d0ec4cf
Author: cody koeninger <>
Date:   2016-10-16T03:00:20Z

    [SPARK-17812][SQL][KAFKA] doc and test updates

commit 2e53e5a3904305cb1d1b0f2325e31c9c434d16ec
Author: cody koeninger <>
Date:   2016-10-16T03:16:11Z

    [SPARK-17812][SQL][KAFKA] style fixes

commit 5e4511f0c7e84d15011a7eb8d208be13ed672b49
Author: cody koeninger <>
Date:   2016-10-16T03:52:39Z

    [SPARK-17812][SQL][KAFKA] additional paranoia on reset of starting offsets

commit cae967cb88a7682b6794d5d2ef90a0d9a1d3ea60
Author: cody koeninger <>
Date:   2016-10-18T03:14:31Z

    Merge branch 'SPARK-17812' into SPARK-17813
    Testing maxOffsetsPerTrigger requires the per-partition sendMessages 
testing added in SPARK-17812

commit 6c8d459f9795c6ff32e8bf78f8796869ca722ee3
Author: cody koeninger <>
Date:   2016-10-18T05:20:53Z

    [SPARK-17813][SQL][KAFKA] maxOffsetsPerTrigger proportional implementation


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to