sunank200 commented on issue #36090: URL: https://github.com/apache/airflow/issues/36090#issuecomment-2167393343
> @sunank200 @akaul my original solution proposed in https://github.com/apache/airflow/issues/36090#issuecomment-2094972855 had a critical part missing, which means that the solution you implemented (in the BigQuery and DataProc operators) needs to be updated, along with the unmerged update to the Databricks operator (PR https://github.com/apache/airflow/pull/39373). > > The problem was that we would not correctly cancel the external job if the task was CLEARED, rather than being set explicitly to SUCCESS or FAILED. This is because if the task is cleared, the new job will likely end up DEFERRED before the asyncio.CancelledError is even thrown. @thesuperzapper I tried clearing the task for `DataprocCreateClusterOperator` and it deleted the provisioning cluster for me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
