This is an automated email from the ASF dual-hosted git repository.
guozhang pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3746bf2 MINOR: Need to have same wait as verify timeout broker
upgrade downgrade (#6127)
3746bf2 is described below
commit 3746bf2c848e28ad5551a1adcf7516da1d61dbc5
Author: Bill Bejeck <[email protected]>
AuthorDate: Fri Jan 11 14:35:33 2019 -0500
MINOR: Need to have same wait as verify timeout broker upgrade downgrade
(#6127)
When I originally refactored the
streams_upgrade_test#upgrade_downgrade_brokers test I removed the wait call
which would wait for up 24 minutes for the SmokeTestDriver class to publish and
verify all of its records.
Since most of the tests run in two minutes or less I set the monitor_log
timeout to three minutes. However, the SmokeTestDriver#verify method allows up
to six minutes to consume all records before verifying the monitor_log timeout
needs to be greater than 6 minutes. I've set the timeout to 8 minutes.
Additionally, the steps needed to update the streams_upgrade_test should be
documented as there are several components that need to get updated. So I've
documented those steps here on the test as a giant comment.
Reviewers: Guozhang Wang <[email protected]>
---
.../tests/streams/streams_upgrade_test.py | 41 +++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/tests/kafkatest/tests/streams/streams_upgrade_test.py
b/tests/kafkatest/tests/streams/streams_upgrade_test.py
index 885aefe..2810be2 100644
--- a/tests/kafkatest/tests/streams/streams_upgrade_test.py
+++ b/tests/kafkatest/tests/streams/streams_upgrade_test.py
@@ -36,6 +36,40 @@ metadata_2_versions = [str(LATEST_0_10_1),
str(LATEST_0_10_2), str(LATEST_0_11_0
backward_compatible_metadata_2_versions = [str(LATEST_0_10_2),
str(LATEST_0_11_0), str(LATEST_1_0), str(LATEST_1_1)]
metadata_3_or_higher_versions = [str(LATEST_2_0), str(LATEST_2_1),
str(DEV_VERSION)]
+"""
+After each release one should first check that the released version has been
uploaded to
+https://s3-us-west-2.amazonaws.com/kafka-packages/ which is the url used by
system test to download jars;
+anyone can verify that by calling
+curl
https://s3-us-west-2.amazonaws.com/kafka-packages/kafka_$scala_version-$version.tgz
to download the jar
+and if it is not uploaded yet, ping the dev@kafka mailing list to request it
being uploaded.
+
+This test needs to get updated, but this requires several steps
+which are outlined here:
+
+1. Update all relevant versions in tests/kafkatest/version.py this will
include adding a new version for the new
+ release and bumping all relevant already released versions.
+
+2. Add the new version to the "kafkatest.version" import above and include the
version in the
+ broker_upgrade_versions list above. You'll also need to add the new
version to the
+ "StreamsUpgradeTestJobRunnerService" on line 484 to make sure the correct
arguments are passed
+ during the system test run.
+
+3. Update the vagrant/bash.sh file to include all new versions, including the
newly released version
+ and all point releases for existing releases. You only need to list the
latest version in
+ this file.
+
+4. Then update all relevant versions in the tests/docker/Dockerfile
+
+5. Add a new "upgrade-system-tests-XXXX module under streams. You can
probably just copy the
+ latest system test module from the last release. Just make sure to update
the systout print
+ statement in StreamsUpgradeTest to the version for the release. After you
add the new module
+ you'll need to update settings.gradle file to include the name of the
module you just created
+ for gradle to recognize the newly added module
+
+6. Then you'll need to update any version changes in gradle/dependencies.gradle
+
+"""
+
class StreamsUpgradeTest(Test):
"""
Test upgrading Kafka Streams (all version combination)
@@ -139,8 +173,13 @@ class StreamsUpgradeTest(Test):
timeout_sec=60,
err_msg="Never saw output '%s'
on" % self.processed_msg + str(processor.node.account))
+ # SmokeTestDriver allows up to 6 minutes to consume all
+ # records for the verification step so this timeout is set to
+ # 6 minutes (360 seconds) for consuming of verification records
+ # and a very conservative additional 2 minutes (120 seconds) to
process
+ # the records in the verification step
driver_monitor.wait_until('ALL-RECORDS-DELIVERED\|PROCESSED-MORE-THAN-GENERATED',
- timeout_sec=180,
+ timeout_sec=480,
err_msg="Never saw output '%s' on" %
'ALL-RECORDS-DELIVERED|PROCESSED-MORE-THAN-GENERATED' +
str(self.driver.node.account))
self.driver.stop()
