[
https://issues.apache.org/jira/browse/SAMZA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shanthoosh Venkataraman resolved SAMZA-1293.
--------------------------------------------
Resolution: Fixed
This feature is released in samza-1.1
> Enable partition expansion of input streams
> -------------------------------------------
>
> Key: SAMZA-1293
> URL: https://issues.apache.org/jira/browse/SAMZA-1293
> Project: Samza
> Issue Type: New Feature
> Reporter: Dong Lin
> Assignee: Shanthoosh Venkataraman
> Priority: Major
>
> Right now, Samza does not allow partitions of the input stream to increase
> after a stateful job is created. This causes problem when Kafka is used as
> the input system, because we need to expand partitions of an existing topic
> as the byte-in-rate of the topic increases over time in order to limit the
> size of the maximum partition in Kafka. Kafka broker may have performance
> issue if the size of a given partition is too large.
> This patch provides a solution to increase partition number of the input
> streams of a stateful Samza job while still ensuring the correctness of Samze
> job output. The solution should work when Kafka is used as the input system.
> We expect this solution to work similarly with other input system as well.
> The motivation of increasing partition number of Kafka topic is 1) increase
> performance of Kafka broker and 2) increase throughput of Kafka consumer in
> the Samza container.
> See SEP-5
> (https://cwiki.apache.org/confluence/display/SAMZA/SEP-5%3A+Enable+partition+expansion+of+input+streams)
> for the design and the interface change of this patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
