[
https://issues.apache.org/jira/browse/SAMZA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013281#comment-16013281
]
ASF GitHub Bot commented on SAMZA-1293:
---------------------------------------
Github user lindong28 closed the pull request at:
https://github.com/apache/samza/pull/193
> Enable partition expansion of input streams
> -------------------------------------------
>
> Key: SAMZA-1293
> URL: https://issues.apache.org/jira/browse/SAMZA-1293
> Project: Samza
> Issue Type: New Feature
> Reporter: Dong Lin
>
> Right now, Samza does not allow partitions of the input stream to increase
> after a stateful job is created. This causes problem when Kafka is used as
> the input system, because we need to expand partitions of an existing topic
> as the byte-in-rate of the topic increases over time in order to limit the
> size of the maximum partition in Kafka. Kafka broker may have performance
> issue if the size of a given partition is too large.
> This patch provides a solution to increase partition number of the input
> streams of a stateful Samza job while still ensuring the correctness of Samze
> job output. The solution should work when Kafka is used as the input system.
> We expect this solution to work similarly with other input system as well.
> The motivation of increasing partition number of Kafka topic is 1) increase
> performance of Kafka broker and 2) increase throughput of Kafka consumer in
> the Samza container.
> See SEP-5
> (https://cwiki.apache.org/confluence/display/SAMZA/SEP-5%3A+Enable+partition+expansion+of+input+streams)
> for the design and the interface change of this patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
