Anand Hariharan created STORM-4076:
--------------------------------------
Summary: KafkaTridentSpoutEmitters can poll all partitions at once
instead of one at a time
Key: STORM-4076
URL: https://issues.apache.org/jira/browse/STORM-4076
Project: Apache Storm
Issue Type: Improvement
Components: storm-client, storm-kafka-client
Reporter: Anand Hariharan
{{Currently 'KafkaTridentTransactionalSpoutEmitter' and
'KafkaTridentOpaqueEmitter' polls every partition assigned to the spout one by
one while emitting new batches. }}
{{But this can be improved by leveraging Kafka Consumer's usual polling
strategy. That is Kafka Broker will take care of choosing the right partition.}}
{{Advantages of this are}}
{{1. If a spout is assigned multiple Topic Partitions, the consumer doesn't
have to waste time on polling partitions with no data.}}
{{2. This change will give better control over the Trident batch size, by
adjusting Kafka Consumer properties.}}
{{Note: This change will affect only when the batch is emitted for the first
time.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
