luc-pimentel commented on issue #74063: URL: https://github.com/apache/airflow/issues/74063#issuecomment-6550017962
I reproduced this on 3.4.0rc2 with the TypeScript runtime and want to send a PR. Before I open it, I want to confirm the approach, because you asked on #74578 to fix this in the supervisor or the SDK runtime rather than in the Execution API, and to handle language-SDK Dags and Python Dags with stub tasks separately. For a Python Dag with `@task.stub` tasks, I see two options in the supervisor: 1. When the runtime finishes, the supervisor tells the server to run the callbacks for this task. The server queues the same callback request it queues today for a task killed externally, and the Dag processor runs it. Same path #60858 picked for a killed task process. The Dag processor also needs to run `on_success_callback`, which it skips today. 2. The supervisor starts a Python process on the worker, loads the Dag file from the bundle and runs the callbacks there. No server change, but the worker has to load the Dag file, and the Dag processor's callback code gets copied. I built option 1 locally to test it. The stub tasks' success, failure and retry callbacks ran in the Dag processor, same lines as the Python tasks. The flag goes in the URL of the state update, not in the body, because the success and retry bodies are also what a runtime sends to the supervisor, and a new field there would mean a new supervisor schema version. Dags written in a language SDK stay out, since they have no task callbacks. Is this what you had in mind? Happy to send the PR. Tested with one Dag that has a Python task and a stub task for each outcome, all with the same success, failure and retry callbacks. Before: the Python tasks wrote their callback lines, the stub tasks wrote none, the `callback` table stayed empty. With option 1: every task wrote its lines, retry included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
