Eric Chang created KAFKA-21262:
----------------------------------
Summary: Share consumer benchmark system tests are flaky
Key: KAFKA-21262
URL: https://issues.apache.org/jira/browse/KAFKA-21262
Project: Kafka
Issue Type: Bug
Components: system tests, tools
Reporter: Eric Chang
Benchmark.test_share_consumer_throughput and
Benchmark.test_producer_and_share_consumer sometimes fail with:
{code}
ValueError: could not convert string to float: ' data.consumed.in.MB'
{code}
There are three problems.
h3. 1. ShareConsumerPerformance uses --timeout as a limit for the whole run
The option is documented as "The maximum allowed time in milliseconds between
returned records", and each consumer thread already follows that. But consume()
waits for the threads with awaitTermination(timeout + 100ms) and then calls
shutdownNow(). So the whole run can take at most timeout + 100ms, even when
records are still arriving.
When the run is longer, poll() is interrupted and throws InterruptException.
Since KAFKA-19450 the exception is rethrown, so the tool exits with code 1 and
prints only the header line. The Python service then tries to parse the header
line as numbers and fails with the ValueError above.
The tests use --timeout 20000. In a recent run, 9 of 10 measured runs took
longer than 20 seconds.
h3. 2. The first share group in a cluster has to wait for __share_group_state
The __share_group_state topic (50 partitions) is created when the first share
group is used. On a busy test machine this took more than 10 seconds. The
consumer was already a stable member of the group, but it got no records until
the topic was ready. With --timeout 20000 the consumer can give up before it
reads anything.
h3. 3. The tests do not check how many records were read
The service only parses the MB and rate columns. A run that reads 0 records
still passes.
With only problem 1 fixed, 5 of 10 runs passed with 0 records read.
h3. Proposed fix
* Remove the total time limit in ShareConsumerPerformance.consume() and wait
for each thread. Add a unit test.
* In the two benchmark tests, read and release one record with
kafka-console-share-consumer.sh before the measured run, so the share group is
ready.
* In the two benchmark tests, fail if the consumers read fewer records than
expected.
h3. Test results
Both tests were repeated 5 times on the kafka-e2e Jenkins job
(jenkins.opensource4you.tw).
* Only the tool fix: 10 of 10 passed, but 5 runs read 0 records.
* All three fixes: 10 of 10 passed, and every run read all records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
