GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/20698
[SPARK-23541][SS] Allow Kafka source to read data with greater parallelism
than the number of topic-partitions
## What changes were proposed in this pull request?
Currently, when the Kafka source reads from Kafka, it generates as many
tasks as the number of partitions in the topic(s) to be read. In some case, it
may be beneficial to read the data with greater parallelism, that is, with more
number partitions/tasks. That means, offset ranges must be divided up into
smaller ranges such the number of records in partition ~= total records in
batch / desired partitions. This would also balance out any data skews between
topic-partitions.
In this patch, I have added a new option called `minPartitions`, which
allows the user to specify the desired level of parallelism.
## How was this patch tested?
New tests in KafkaMicroBatchV2SourceSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-23541
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20698.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20698
----
commit ebb9b51c51a4411811a7e0e09fff8f8608faa017
Author: Tathagata Das <tathagata.das1565@...>
Date: 2018-03-01T01:28:32Z
Implemented
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
