[jira] [Updated] (KAFKA-16579) Revert changes to consumer_rolling_upgrade_test.py for the new async Consumer
[
https://issues.apache.org/jira/browse/KAFKA-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-16579:
--
Description:
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a
slew of system tests to run both the "old" and "new" implementations.
KAFKA-16271 updated the system tests in {{consumer_rolling_upgrade_test.py}} so
it could test the new consumer. However, the test is tailored specifically to
the "old" Consumer's protocol and assignment strategy upgrade.
Unsurprisingly, when we run those system tests with the new
{{AsyncKafkaConsumer}}, we get errors like the following:
{code}
test_id:
kafkatest.tests.client.consumer_rolling_upgrade_test.ConsumerRollingUpgradeTest.rolling_update_test.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status: FAIL
run time: 29.634 seconds
AssertionError("Mismatched assignment: {frozenset(),
frozenset({TopicPartition(topic='test_topic', partition=0),
TopicPartition(topic='test_topic', partition=1)})}")
Traceback (most recent call last):
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/consumer_rolling_upgrade_test.py",
line 77, in rolling_update_test
self._verify_range_assignment(consumer)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/consumer_rolling_upgrade_test.py",
line 41, in _verify_range_assignment
"Mismatched assignment: %s" % assignment
AssertionError: Mismatched assignment: {frozenset(),
frozenset({TopicPartition(topic='test_topic', partition=0),
TopicPartition(topic='test_topic', partition=1)})}
{code}
The task here is to revert the changes made in KAFKA-16271.
was:
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a
slew of system tests to run both the "old" and "new" implementations.
KAFKA-16271 updated the system tests in {{consumer_rolling_upgrade_test.py}} so
it could test the new consumer. However, the test is tailored specifically to
the "old" Consumer's protocol and assignment strategy upgrade.
Unsurprisingly, when we run those system tests with the new
{{AsyncKafkaConsumer}}, we get errors like the following:
{code:java}
test_id:
kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status: FAIL
run time: 6 minutes 3.899 seconds
InsufficientResourcesError('Not enough nodes available to allocate. linux
nodes requested: 1. linux nodes available: 0')
Traceback (most recent call last):
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
line 919, in test_exactly_once_source
consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka,
self.source.topic, consumer_timeout_ms=1000, print_key=True)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py",
line 97, in __init__
BackgroundThreadService.__init__(self, context, num_nodes)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py",
line 26, in __init__
super(BackgroundThreadService, self).__init__(context, num_nodes,
cluster_spec, *args, **kwargs)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py",
line 107, in __init__
self.allocate_nodes()
File
"/
[jira] [Updated] (KAFKA-16579) Revert changes to consumer_rolling_upgrade_test.py for the new async Consumer
[
https://issues.apache.org/jira/browse/KAFKA-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-16579:
--
Description:
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a
slew of system tests to run both the "old" and "new" implementations.
KAFKA-16271 updated the system tests in {{consumer_rolling_upgrade_test.py}} so
it could test the new consumer. However, the test is tailored specifically to
the "old" Consumer's protocol and assignment strategy upgrade.
Unsurprisingly, when we run those system tests with the new
{{AsyncKafkaConsumer}}, we get errors like the following:
{code:java}
test_id:
kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status: FAIL
run time: 6 minutes 3.899 seconds
InsufficientResourcesError('Not enough nodes available to allocate. linux
nodes requested: 1. linux nodes available: 0')
Traceback (most recent call last):
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
line 919, in test_exactly_once_source
consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka,
self.source.topic, consumer_timeout_ms=1000, print_key=True)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py",
line 97, in __init__
BackgroundThreadService.__init__(self, context, num_nodes)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py",
line 26, in __init__
super(BackgroundThreadService, self).__init__(context, num_nodes,
cluster_spec, *args, **kwargs)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py",
line 107, in __init__
self.allocate_nodes()
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py",
line 217, in allocate_nodes
self.nodes = self.cluster.alloc(self.cluster_spec)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py",
line 54, in alloc
allocated = self.do_alloc(cluster_spec)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py",
line 31, in do_alloc
allocated = self._available_nodes.remove_spec(cluster_spec)
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py",
line 117, in remove_spec
raise InsufficientResourcesError("Not enough nodes available to allocate. "
+ msg)
ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes
available to allocate. linux nodes requested: 1. linux nodes available: 0
{code}
The task here is to revert the changes made in KAFKA-16272 [PR
15576|https://github.com/apache/kafka/pull/15576].
was:
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a
slew of system tests to run both the "old" and "new" implementations.
KAFKA-16272 updated the system tests in {{connect_distributed_test.py}} so it
could test the new consumer with Connect. However, we are not supporting
Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect
system tests with the new {{{}AsyncKafkaConsumer{}}}, we get errors like the
following:
{code:java}
test_id:
kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status: FAIL
run time: 6 minutes 3.899 seconds
InsufficientResourcesError('Not enough nodes available to allocate. linux
nodes requested: 1. linux nodes available: 0')
Traceback (most recent call last):
File
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py",
