danielwardega7777 opened a new pull request, #77387:
URL: https://github.com/apache/airflow/pull/77387

   ManagedKafkaTokenProvider._valid_credentials() previously returned None when 
credentials were already valid, and the managed_kafka_consumer_group_operations 
system test failed on newly created clusters due to hardcoded bootstrap 
hostnames, missing offset reset configuration, and clearing all topic offsets 
during consumer group updates.
   
   This change:
   - Fixes ManagedKafkaTokenProvider._valid_credentials() to return 
self._credentials even when credentials do not need refreshing, and adds unit 
test coverage (test_get_confluent_token_with_valid_credentials).
   - Resolves the cluster bootstrapAddress dynamically from the Managed Kafka 
Cluster API in example_managed_kafka_consumer_group.py (falling back to 
BOOTSTRAP_URL), since newly created clusters use 
bootstrap-<hash>.<hash>.<location>.managedkafka.s.cloud.goog:9092.
   - Adds "auto.offset.reset": "earliest" to the Kafka connection extras so 
consume_from_topic consumes produced messages and commits offsets for the 
consumer group.
   - Updates update_consumer_group to set partition 0 offset on the test topic 
instead of "topics": {} (which cleared all committed offsets and caused the 
empty consumer group to be removed before get_consumer_group and 
delete_consumer_group).
   - Adds a delete_connection teardown task (TriggerRule.ALL_DONE) to clean up 
the Kafka Airflow connection in example_managed_kafka_consumer_group.py.
   - Defaults NETWORK to "default" when GOOGLE_PROVIDER_NETWORK is unset or 
empty across example_managed_kafka_cluster.py, example_managed_kafka_topic.py, 
and example_managed_kafka_consumer_group.py.
   
   Verification:
   - Unit tests (test_managed_kafka.py for hooks and operators): 29 passed.
   - Linting and formatting (ruff check and ruff format --check): passed.
   - System tests verified end-to-end on GCP (us-central1): 
managed_kafka_cluster_operations, managed_kafka_topic_operations, and 
managed_kafka_consumer_group_operations all passed.
   
   ---
   
   Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Jetski (Gemini) following the guidelines 
(https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to