PoAn Yang created KAFKA-17978:
---------------------------------
Summary: StreamsUpgradeTest#test_rolling_upgrade_with_2_bounces
system tests fail
Key: KAFKA-17978
URL: https://issues.apache.org/jira/browse/KAFKA-17978
Project: Kafka
Issue Type: Test
Reporter: PoAn Yang
Assignee: PoAn Yang
Run `TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py" /bin/bash
tests/docker/run_tests.sh` on trunk branch. The versions which can support
fk_joins can't pass `test_rolling_upgrade_with_2_bounces`.
{noformat}
[INFO:2024-11-09 22:24:00,601]: Triggering test 10 of 19...
[INFO:2024-11-09 22:24:00,611]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
'3.4.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
[INFO:2024-11-09 22:24:00,619]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
on run 1/1
[INFO:2024-11-09 22:24:00,621]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
Setting up...
[INFO:2024-11-09 22:24:00,623]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
Running...
[INFO:2024-11-09 22:26:26,343]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
Tearing down...
[INFO:2024-11-09 22:27:47,017]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker07")
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
351, in _do_run
data = self.run_test()
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
411, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
438, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 137, in test_rolling_upgrade_with_2_bounces
self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
extra_properties)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 402, in do_stop_start_bounce
err_msg="Never saw output '%s' on " % self.processed_data_msg +
str(node.account))
File
"/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 754, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker07
[WARNING:2024-11-09 22:27:47,017]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[INFO:2024-11-09 22:27:47,017]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.4.1.metadata_quorum=COMBINED_KRAFT:
Data: None
[INFO:2024-11-09 22:27:47,124]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-11-09 22:27:47,125]: Triggering test 11 of 19...
[INFO:2024-11-09 22:27:47,134]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
'3.5.2', 'metadata_quorum': 'COMBINED_KRAFT'}}
[INFO:2024-11-09 22:27:47,142]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
on run 1/1
[INFO:2024-11-09 22:27:47,144]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Setting up...
[INFO:2024-11-09 22:27:47,146]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Running...
[INFO:2024-11-09 22:30:12,455]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Tearing down...
[INFO:2024-11-09 22:31:34,101]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker13")
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
351, in _do_run
data = self.run_test()
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
411, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
438, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 137, in test_rolling_upgrade_with_2_bounces
self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
extra_properties)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 402, in do_stop_start_bounce
err_msg="Never saw output '%s' on " % self.processed_data_msg +
str(node.account))
File
"/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 754, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker13
[WARNING - 2024-11-09 22:31:34,101 - runner_client - log - lineno:459]:
RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[WARNING:2024-11-09 22:31:34,101]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[INFO:2024-11-09 22:31:34,101]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.5.2.metadata_quorum=COMBINED_KRAFT:
Data: None
[INFO:2024-11-09 22:31:34,208]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-11-09 22:31:34,208]: Triggering test 12 of 19...
[INFO:2024-11-09 22:31:34,218]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
'3.6.2', 'metadata_quorum': 'COMBINED_KRAFT'}}
[INFO:2024-11-09 22:31:34,226]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
on run 1/1
[INFO:2024-11-09 22:31:34,228]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Setting up...
[INFO:2024-11-09 22:31:34,229]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Running...
[INFO:2024-11-09 22:33:58,400]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Tearing down...
[INFO:2024-11-09 22:35:19,548]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker07")
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
351, in _do_run
data = self.run_test()
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
411, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
438, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 137, in test_rolling_upgrade_with_2_bounces
self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
extra_properties)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 402, in do_stop_start_bounce
err_msg="Never saw output '%s' on " % self.processed_data_msg +
str(node.account))
File
"/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 754, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker07
[WARNING - 2024-11-09 22:35:19,549 - runner_client - log - lineno:459]:
RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[WARNING:2024-11-09 22:35:19,550]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[INFO:2024-11-09 22:35:19,551]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.6.2.metadata_quorum=COMBINED_KRAFT:
Data: None
[INFO:2024-11-09 22:35:19,660]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-11-09 22:35:19,660]: Triggering test 13 of 19...
[INFO:2024-11-09 22:35:19,670]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
'3.7.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
[INFO:2024-11-09 22:35:19,678]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
on run 1/1
[INFO:2024-11-09 22:35:19,680]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Setting up...
[INFO:2024-11-09 22:35:19,681]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Running...
[INFO:2024-11-09 22:38:00,197]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Tearing down...
[INFO:2024-11-09 22:39:21,823]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
FAIL: TimeoutError("Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker12")
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
351, in _do_run
data = self.run_test()
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
411, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
438, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 137, in test_rolling_upgrade_with_2_bounces
self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
extra_properties)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 402, in do_stop_start_bounce
err_msg="Never saw output '%s' on " % self.processed_data_msg +
str(node.account))
File
"/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 754, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker12
[WARNING - 2024-11-09 22:39:21,823 - runner_client - log - lineno:459]:
RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[WARNING:2024-11-09 22:39:21,823]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[INFO:2024-11-09 22:39:21,824]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.7.1.metadata_quorum=COMBINED_KRAFT:
Data: None
[INFO:2024-11-09 22:39:21,931]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[INFO:2024-11-09 22:39:21,931]: Triggering test 14 of 19...
[INFO:2024-11-09 22:39:21,940]: RunnerClient: Loading test {'directory':
'/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name':
'streams_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args': {'from_version':
'3.8.1', 'metadata_quorum': 'COMBINED_KRAFT'}}
[INFO:2024-11-09 22:39:21,948]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
on run 1/1
[INFO:2024-11-09 22:39:21,950]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Setting up...
[INFO:2024-11-09 22:39:21,952]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Running...
[INFO:2024-11-09 22:41:49,029]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Tearing down...
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
351, in _do_run
data = self.run_test()
File
"/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line
411, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line
438, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 137, in test_rolling_upgrade_with_2_bounces
self.do_stop_start_bounce(p, from_version[:-2], to_version, counter,
extra_properties)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py",
line 402, in do_stop_start_bounce
err_msg="Never saw output '%s' on " % self.processed_data_msg +
str(node.account))
File
"/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 754, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line
58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw output 'processed [0-9]* records from
topic=data' on ducker@ducker04
[WARNING - 2024-11-09 22:43:10,413 - runner_client - log - lineno:459]:
RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[WARNING:2024-11-09 22:43:10,414]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Test requested 6 nodes, used only 5
[INFO:2024-11-09 22:43:10,414]: RunnerClient:
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_rolling_upgrade_with_2_bounces.from_version=3.8.1.metadata_quorum=COMBINED_KRAFT:
Data: None
[INFO:2024-11-09 22:43:10,521]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
