o-nikolas commented on code in PR #36803:
URL: https://github.com/apache/airflow/pull/36803#discussion_r1462245406


##########
airflow/jobs/scheduler_job_runner.py:
##########
@@ -721,6 +721,7 @@ def _process_executor_events(self, session: Session) -> int:
             if state == TaskInstanceState.QUEUED:
                 ti.external_executor_id = info
                 self.log.info("Setting external_id for %s to %s", ti, info)
+                session.commit()

Review Comment:
   I'm not a DB/SQLA expert, but do we want to commit here, our outside of the 
for loop? Here is going to do a commit for each queued task individually that 
we got from the query. Which could be many? Interested to hear others weigh in. 



##########
airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:
##########
@@ -377,6 +383,9 @@ def attempt_task_runs(self):
             else:
                 task = run_task_response["tasks"][0]
                 self.active_workers.add_task(task, task_key, queue, cmd, 
exec_config, attempt_number)
+                # Add Fargate task ARN to executor event buffer, which gets 
saved
+                # in TaskInstance.external_executor_id.
+                self.event_buffer[task_key] = (State.QUEUED, task.task_arn)

Review Comment:
   We have `self.success` and `self.fail` (e.g. 
[link](https://github.com/aws-mwaa/upstream-to-airflow/blob/d18c01a7a2f04449262606d79b4f663bdb200414/airflow/executors/base_executor.py#L316))
 which handles the other state changes. These are provided by the base 
executor. I wonder if we should add a new method for putting a task in queued 
state :thinking: but this might cause weird issues if a provider which contains 
an executor is installed alongside an older version of airflow... (also, not 
required for this PR, just thinking out loud)
   



##########
airflow/providers/amazon/provider.yaml:
##########
@@ -936,6 +936,17 @@ config:
         type: boolean
         example: "True"
         default: "True"
+      adopt_task_instances:
+        description: |
+          If True, the executor will try to adopt orphaned task instances from 
a SchedulerJob
+          shutdown event, for example when a scheduler container is 
re-deployed or terminated.
+          If False, the executor will terminate all active AWS Batch Jobs when 
the scheduler shuts down.

Review Comment:
   ```suggestion
             If False, the executor will terminate all active AWS ECS Tasks 
when the scheduler shuts down.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to