danielwardega7777 opened a new pull request, #77387:
URL: https://github.com/apache/airflow/pull/77387
ManagedKafkaTokenProvider._valid_credentials() previously returned None when
credentials were already valid, and the managed_kafka_consumer_group_operations
system test failed on newly created clusters due to hardcoded bootstrap
hostnames, missing offset reset configuration, and clearing all topic offsets
during consumer group updates.
This change:
- Fixes ManagedKafkaTokenProvider._valid_credentials() to return
self._credentials even when credentials do not need refreshing, and adds unit
test coverage (test_get_confluent_token_with_valid_credentials).
- Resolves the cluster bootstrapAddress dynamically from the Managed Kafka
Cluster API in example_managed_kafka_consumer_group.py (falling back to
BOOTSTRAP_URL), since newly created clusters use
bootstrap-<hash>.<hash>.<location>.managedkafka.s.cloud.goog:9092.
- Adds "auto.offset.reset": "earliest" to the Kafka connection extras so
consume_from_topic consumes produced messages and commits offsets for the
consumer group.
- Updates update_consumer_group to set partition 0 offset on the test topic
instead of "topics": {} (which cleared all committed offsets and caused the
empty consumer group to be removed before get_consumer_group and
delete_consumer_group).
- Adds a delete_connection teardown task (TriggerRule.ALL_DONE) to clean up
the Kafka Airflow connection in example_managed_kafka_consumer_group.py.
- Defaults NETWORK to "default" when GOOGLE_PROVIDER_NETWORK is unset or
empty across example_managed_kafka_cluster.py, example_managed_kafka_topic.py,
and example_managed_kafka_consumer_group.py.
Verification:
- Unit tests (test_managed_kafka.py for hooks and operators): 29 passed.
- Linting and formatting (ruff check and ruff format --check): passed.
- System tests verified end-to-end on GCP (us-central1):
managed_kafka_cluster_operations, managed_kafka_topic_operations, and
managed_kafka_consumer_group_operations all passed.
---
Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Jetski (Gemini) following the guidelines
(https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]