o-nikolas commented on code in PR #36803:
URL: https://github.com/apache/airflow/pull/36803#discussion_r1462245406
##########
airflow/jobs/scheduler_job_runner.py:
##########
@@ -721,6 +721,7 @@ def _process_executor_events(self, session: Session) -> int:
if state == TaskInstanceState.QUEUED:
ti.external_executor_id = info
self.log.info("Setting external_id for %s to %s", ti, info)
+ session.commit()
Review Comment:
I'm not a DB/SQLA expert, but do we want to commit here, our outside of the
for loop? Here is going to do a commit for each queued task individually that
we got from the query. Which could be many? Interested to hear others weigh in.
##########
airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:
##########
@@ -377,6 +383,9 @@ def attempt_task_runs(self):
else:
task = run_task_response["tasks"][0]
self.active_workers.add_task(task, task_key, queue, cmd,
exec_config, attempt_number)
+ # Add Fargate task ARN to executor event buffer, which gets
saved
+ # in TaskInstance.external_executor_id.
+ self.event_buffer[task_key] = (State.QUEUED, task.task_arn)
Review Comment:
We have `self.success` and `self.fail` (e.g.
[link](https://github.com/aws-mwaa/upstream-to-airflow/blob/d18c01a7a2f04449262606d79b4f663bdb200414/airflow/executors/base_executor.py#L316))
which handles the other state changes. These are provided by the base
executor. I wonder if we should add a new method for putting a task in queued
state :thinking: but this might cause weird issues if a provider which contains
an executor is installed alongside an older version of airflow... (also, not
required for this PR, just thinking out loud)
##########
airflow/providers/amazon/provider.yaml:
##########
@@ -936,6 +936,17 @@ config:
type: boolean
example: "True"
default: "True"
+ adopt_task_instances:
+ description: |
+ If True, the executor will try to adopt orphaned task instances from
a SchedulerJob
+ shutdown event, for example when a scheduler container is
re-deployed or terminated.
+ If False, the executor will terminate all active AWS Batch Jobs when
the scheduler shuts down.
Review Comment:
```suggestion
If False, the executor will terminate all active AWS ECS Tasks
when the scheduler shuts down.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]