Sam Meder created KAFKA-1417:
--------------------------------
Summary: Very slow initial high-level consumer startup in low
traffic/blocking fetch scenario
Key: KAFKA-1417
URL: https://issues.apache.org/jira/browse/KAFKA-1417
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.8.0
Reporter: Sam Meder
Assignee: Neha Narkhede
We're seeing very slow startup times when starting a high level consumer in a
low traffic/blocking fetch type setup. The example we've come across has a
consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout.
What happens is that the leader finder thread adds partitions one by one and
since the offset is not know this causes a call to figure out the offset. This
call uses the fetcher threads simple consumer instance and locks around the
call. Initially this is not a problem, but as soon as the fetcher thread has
some partitions it will start fetching and since this is a low traffic
situation the fetch will at least sometimes take up to 20s (again locking
around the simple consumer). This leads to behavior like:
# Finder thread adds a partition
# Data thread notices it has partitions to fetch data for, locks the consumer
for 20s
# Finder thread tries to add a partition, tries to lock consumer and blocks for
20s
# Rinse, repeat for each partition
--
This message was sent by Atlassian JIRA
(v6.2#6252)
