[
https://issues.apache.org/jira/browse/SAMZA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011831#comment-16011831
]
ASF GitHub Bot commented on SAMZA-1293:
---------------------------------------
GitHub user lindong28 opened a pull request:
https://github.com/apache/samza/pull/193
SAMZA-1293: Enable partition expansion of input streams (SEP-4)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lindong28/samza SAMZA-1293
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/193.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #193
----
commit 0ffd0d1ef81f2ed9b1a5d30918d1ba6d5b783227
Author: Dong Lin <[email protected]>
Date: 2017-05-16T00:59:18Z
SAMZA-1293: Enable partition expansion of input streams (SEP-4)
----
> Enable partition expansion of input streams
> -------------------------------------------
>
> Key: SAMZA-1293
> URL: https://issues.apache.org/jira/browse/SAMZA-1293
> Project: Samza
> Issue Type: New Feature
> Reporter: Dong Lin
>
> Right now, Samza does not allow partitions of the input stream to increase
> after a stateful job is created. This causes problem when Kafka is used as
> the input system, because we need to expand partitions of an existing topic
> as the byte-in-rate of the topic increases over time in order to limit the
> size of the maximum partition in Kafka. Kafka broker may have performance
> issue if the size of a given partition is too large.
> This patch provides a solution to increase partition number of the input
> streams of a stateful Samza job while still ensuring the correctness of Samze
> job output. The solution should work when Kafka is used as the input system.
> We expect this solution to work similarly with other input system as well.
> The motivation of increasing partition number of Kafka topic is 1) increase
> performance of Kafka broker and 2) increase throughput of Kafka consumer in
> the Samza container.
> See SEP-4
> (https://cwiki.apache.org/confluence/display/SAMZA/SEP-4%3A+Enable+partition+expansion+of+input+streams)
> for the design and the interface change of this patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
