Lakshmi Manasa Gaduputi created SAMZA-2687:
----------------------------------------------
Summary: Elasticity: scale up task count beyond the input
partition count.
Key: SAMZA-2687
URL: https://issues.apache.org/jira/browse/SAMZA-2687
Project: Samza
Issue Type: New Feature
Reporter: Lakshmi Manasa Gaduputi
Assignee: Lakshmi Manasa Gaduputi
Problem: Throughput via parallelism is tied to the number of tasks which is
equal to the partition count of input streams. If a job is facing lag and is
already at the max container count = number of tasks = number of input
partitions, then the only choice it is left with is to repartition the input.
The need for this arises due to the process-time of the job’s logic which is
not under Samza’s control.
Solution: Proposed approach is to allow consumption of a portion of the
partition (SystemStreamPartition) by a virtual task. Virtual task is the same
as a task except that it consumes sub-ssp.
SEP to follow shortly
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
