[
https://issues.apache.org/jira/browse/KAFKA-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Chang updated KAFKA-21262:
-------------------------------
Summary: Share consumer benchmarks are flaky: the run is cut off at
--timeout (was: Share consumer benchmark system tests are flaky)
> Share consumer benchmarks are flaky: the run is cut off at --timeout
> --------------------------------------------------------------------
>
> Key: KAFKA-21262
> URL: https://issues.apache.org/jira/browse/KAFKA-21262
> Project: Kafka
> Issue Type: Bug
> Components: system tests, tools
> Reporter: Eric Chang
> Assignee: Eric Chang
> Priority: Major
> Attachments: build-968-no-fix-ValueError.txt,
> build-969-tool-fix-only-zero-records.txt, build-970-all-fixes.txt
>
>
> Benchmark.test_share_consumer_throughput and
> Benchmark.test_producer_and_share_consumer sometimes fail with:
> {code}
> ValueError: could not convert string to float: ' data.consumed.in.MB'
> {code}
> There are three problems.
> h3. 1. ShareConsumerPerformance uses --timeout as a limit for the whole run
> The option is documented as "The maximum allowed time in milliseconds between
> returned records", and each consumer thread already follows that. But
> consume() waits for the threads with awaitTermination(timeout + 100ms) and
> then calls shutdownNow(). So the whole run can take at most timeout + 100ms,
> even when records are still arriving.
> When the run is longer, poll() is interrupted and throws InterruptException.
> Since KAFKA-19450 the exception is rethrown, so the tool exits with code 1
> and prints only the header line. The Python service then tries to parse the
> header line as numbers and fails with the ValueError above.
> The tests use --timeout 20000. In a recent run, 9 of 10 measured runs took
> longer than 20 seconds.
> h3. 2. The first share group in a cluster has to wait for __share_group_state
> The __share_group_state topic (50 partitions) is created when the first share
> group is used. On a busy test machine this took more than 10 seconds. The
> consumer was already a stable member of the group, but it got no records
> until the topic was ready. With --timeout 20000 the consumer can give up
> before it reads anything.
> h3. 3. The tests do not check how many records were read
> The service only parses the MB and rate columns. A run that reads 0 records
> still passes.
> With only problem 1 fixed, 5 of 10 runs passed with 0 records read.
> h3. Proposed fix
> * Remove the total time limit in ShareConsumerPerformance.consume() and wait
> for each thread. Add a unit test.
> * In the two benchmark tests, read and release one record with
> kafka-console-share-consumer.sh before the measured run, so the share group
> is ready.
> * In the two benchmark tests, fail if the consumers read fewer records than
> expected.
> h3. Test results
> Both tests were repeated 5 times on the kafka-e2e Jenkins job
> (jenkins.opensource4you.tw).
> * Only the tool fix: 10 of 10 passed, but 5 runs read 0 records.
> * All three fixes: 10 of 10 passed, and every run read all records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
