dejii opened a new pull request, #54101: URL: https://github.com/apache/airflow/pull/54101
This PR makes the `CloudStorageTransferServiceCreateJobsTrigger` more robust by allowing it to wait for transfer job operations to become available instead of failing immediately if they don't exist on the first check. Problem: The `CloudStorageTransferServiceCreateJobsTrigger` would error and fail the task if a transfer job did not have an associated operation on the first poll. This could lead to premature failures in cases where the operation's creation is slightly delayed by the GCS Transfer API (typically observed when many transfer service jobs are created simultaneously), causing an undesirable state where the task fails but the operation continues and retrying the task creates a new transfer operation. Solution: This PR modifies the trigger to gracefully handle this scenario. Instead of erroring immediately, the trigger now checks if all jobs have a corresponding operation. If any are missing, it logs a waiting message and retries after the configured polling interval. Unit tests have been updated to cover the new behavior. The new tests simulate the condition where operations are not initially present and verify that the trigger waits and eventually succeeds once the operations become available. <!-- Please keep an empty line above the dashes. --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
