Lakshmi Manasa Gaduputi created SAMZA-2724:
----------------------------------------------
Summary: [Elasticity] optimizations to improve throughput when
elasticity is enabled by filtering out unwanted messages within SystemConsumers
before RunLoop
Key: SAMZA-2724
URL: https://issues.apache.org/jira/browse/SAMZA-2724
Project: Samza
Issue Type: Improvement
Reporter: Lakshmi Manasa Gaduputi
Assignee: Lakshmi Manasa Gaduputi
When elasticity is enabled, the following optimizations can be done to improve
throughput
in SystemConsumers, filter the messages which are not going to be processed by
the RunLoop - aka of the messages fetched from consumer.poll(ssp) remove those
messages which belong to the key buckets of the ssp not consumed by the
container's job model. This will ensure RunLoop gets only those messages that
it needs to process
Note that during prototyping this optimization, it was observed that this
filtering causes a delay in the start of processing in all containers. this is
due to all messages from an ssp being filtered out initially for ~7-10mins.
This could be due to how the messages are fetched from the specific input
topic. need a deeper investigation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
