Neha Narkhede created KAFKA-808:
-----------------------------------
Summary: Migration tool internal queue between consumer and
producer threads should be configurable
Key: KAFKA-808
URL: https://issues.apache.org/jira/browse/KAFKA-808
Project: Kafka
Issue Type: Improvement
Components: tools
Affects Versions: 0.8
Reporter: Neha Narkhede
Assignee: Neha Narkhede
Priority: Critical
Migration tool maintains an internal blocking queue to buffer messages between
the 0.7 consumer and the 0.8 producer. It is fixed in size and it set to the
number of producers. However, the consumer usually fetches data in large
chunks, several thousands of messages. Also each producer's internal queue
frees up in chunks of the producer's batch size. In performance profiling, I
saw that the migration tool's consumer thread spent time waiting for space to
free up in the internal queue. It seems that this internal queue should be
configurable and much larger than the number of producers. This will improve
the buffering and help bridge the gap between the consumer and producer's
throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
