pmalafosse opened a new pull request #16685:
URL: https://github.com/apache/airflow/pull/16685
FYI @darwinyip
This PR adds a parameter reattach_prev_task to ECSOperator.
**Before:**
Until now we could use 'reattach' which was reattaching a running ECS Task
(if there was one running) of the same 'family' instead of creating a new one.
The problem was that if we had workflows using the same ECS Task Definition in
several tasks, it didn't know which one to reattach.
**Now:**
Now with reattach_prev_task instead, when we launch a new ECS task, it will
store temporarily the ECS Task ARN in XCOM. If there is an issue during the run
(typically connection problem between Airflow and ECS for long-running tasks):
- self._start_task will store the ECS task ARN in XCOM (in a 'fake' task_id
equal to f"{self.task_id}_task_arn"
- in the next execution, it will check if this task ARN is still running and
if so it will reattach it to the operator, otherwise it will create a new one
- when the operator runs succesfully it will delete the XCOM value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
