[
https://issues.apache.org/jira/browse/SAMZA-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lakshmi Manasa Gaduputi updated SAMZA-2724:
-------------------------------------------
Parent: SAMZA-2687
Issue Type: Sub-task (was: Improvement)
> [Elasticity] optimizations to improve throughput when elasticity is enabled
> by filtering out unwanted messages within SystemConsumers before RunLoop
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SAMZA-2724
> URL: https://issues.apache.org/jira/browse/SAMZA-2724
> Project: Samza
> Issue Type: Sub-task
> Reporter: Lakshmi Manasa Gaduputi
> Assignee: Lakshmi Manasa Gaduputi
> Priority: Major
>
> When elasticity is enabled, the following optimizations can be done to
> improve throughput
>
> in SystemConsumers, filter the messages which are not going to be processed
> by the RunLoop - aka of the messages fetched from consumer.poll(ssp) remove
> those messages which belong to the key buckets of the ssp not consumed by the
> container's job model. This will ensure RunLoop gets only those messages that
> it needs to process
>
> Note that during prototyping this optimization, it was observed that this
> filtering causes a delay in the start of processing in all containers. this
> is due to all messages from an ssp being filtered out initially for
> ~7-10mins. This could be due to how the messages are fetched from the
> specific input topic. need a deeper investigation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
