TakawaAkirayo commented on PR #41232: URL: https://github.com/apache/airflow/pull/41232#issuecomment-2268150420
CC @potiuk If there are any other suggested solutions for this, our use case is as follows: 1.Submit a job to Apache Kyuubi in the worker, which will return a remote job ID and the remote job executed in Kyuubi side. 2.Defer the task by creating a trigger and pass the remote job ID to the trigger. 3.The trigger will continuously check the status of the remote job in the run method and yield an event once the job succeeds or fails. 4.In the trigger's cleanup method, we terminate the remote job if the remote job ID exists. The problem is that the cleanup method could called when a task is canceled or the trigger is reassigned to another triggerer. The issue arises when the cleanup method is called during reassignment, which terminates the remote job. As a result, once reassigned to another triggerer, the remote job fails. To avoid making the trigger heavier, we chose not to move step 1 (submitting the job and receiving the remote job ID) into the trigger's run method. This decision is based on the need to handle many preparations, such as authentication setup and local file propagation, which could burden the trigger process. Instead, we perform these preparations in the worker and delegate only the status check to the trigger. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
