kaxil commented on code in PR #76964:
URL: https://github.com/apache/airflow/pull/76964#discussion_r4607058932


##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp 
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job 
still holds its name.

Review Comment:
   This rename only happens when `check_if_job_exists` is on, which it is by 
default. With it off, `submit_job` sends the stored name again and SageMaker 
rejects the create, so I'd mention that flag here too.



##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state

Review Comment:
   AWS gives the algorithm 120 seconds after `StopTrainingJob` to save model 
artifacts, so a stopped job can have a model. The reason in the code comment on 
`is_job_active` is the real one: with `print_log=False`, 
`wait_for_training_job` treats the resulting `Stopped` as done. Could the docs 
use that?



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:
##########
@@ -1328,7 +1328,7 @@ def execute(self, context: Context) -> dict:
         self.execute_resumable(context)
         # On reconnect the mixin returns poll_until_complete's value and never 
calls
         # get_job_result, so the result is read off the operator instead.
-        return cast("dict[str, dict]", self._result)
+        return {"Training": self.serialized_training_data}

Review Comment:
   A subclass that overrides `serialize_result` (to trim the XCom, say) now has 
its return value ignored on the sync path, since this reads the attribute the 
base method sets as a side effect. 9.40.0 returned `self.serialize_result(...)` 
here. Would `return self.get_job_result(self._training_job_name, context)` 
work? It returns the cached `_result` on every path and needs no cast.



##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp 
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job 
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the 
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt 
is treated the same
+as a retry. A task whose job already completed returns the existing result 
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or 
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set 
``keep_task_state`` when

Review Comment:
   With the default `cancel_on_kill=False`, clearing a running task on 3.4+ 
leaves the first job running, and the next attempt submits a second, renamed 
one, so both are billed. The core `resumable-tasks` page sends readers to the 
operator docs for exactly this case. Could this add a sentence: 
`keep_task_state` to reconnect, or `cancel_on_kill=True` to stop the old job?



##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp 
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job 
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the 
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt 
is treated the same
+as a retry. A task whose job already completed returns the existing result 
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or 
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set 
``keep_task_state`` when
+clearing to keep it, and the next attempt then reconnects to the job or 
returns its result. Clearing
+a whole Dag run, marking a task as failed or success, and ``airflow tasks 
clear`` still keep the job
+name before Airflow 3.5. From Airflow 3.5, every way of clearing deletes it by 
default.
+
+See :doc:`apache-airflow:core-concepts/resumable-tasks` for the ways of 
clearing, and for the
+``[state_store] clear_on_success`` setting.
+
+To opt out and always start a new job on retry, set ``durable=False``:
+
+.. code-block:: python
+
+  training = SageMakerTrainingOperator(
+      task_id="train_model",
+      config=training_config,
+      durable=False,
+  )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is 
set, ``durable`` has

Review Comment:
   This only holds with `wait_for_completion=True` too. `execute()` defers only 
when both are set, so `deferrable=True, wait_for_completion=False` falls 
through to `execute_resumable` and persists the name.



##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp 
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job 
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the 
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt 
is treated the same
+as a retry. A task whose job already completed returns the existing result 
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or 
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set 
``keep_task_state`` when
+clearing to keep it, and the next attempt then reconnects to the job or 
returns its result. Clearing
+a whole Dag run, marking a task as failed or success, and ``airflow tasks 
clear`` still keep the job

Review Comment:
   `airflow dags clear` also keeps it before 3.5, so it belongs in this list 
next to `airflow tasks clear`.



##########
providers/amazon/tests/unit/amazon/aws/operators/test_sagemaker_training.py:
##########
@@ -185,13 +185,18 @@ def test_durable_persists_the_submitted_job_name(self, _, 
mock_training, mock_de
     @mock.patch.object(SageMakerHook, "create_training_job")
     @mock.patch.object(sagemaker, "serialize", return_value="")
     @mock.patch.object(SageMakerHook, "configure_s3_resources")
+    @mock.patch.object(SageMakerHook, "stop_training_job")
     def test_durable_reconnects_to_a_running_job(
-        self, mock_configure_s3, _, mock_training, mock_desc, mock_poll
+        self, mock_stop, mock_configure_s3, _, mock_training, mock_desc, 
mock_poll
     ):
         mock_desc.return_value = {"TrainingJobStatus": "InProgress"}
+        self.sagemaker.cancel_on_kill = True
         store = self._task_state_store("previously_running_job")
 
         result = self.sagemaker.execute({"task_state_store": store})
+        self.sagemaker.on_kill()

Review Comment:
   This calls `on_kill()` after `execute()` returns, so it still passes if 
`_training_job_name` is only set after the wait finishes, when a real kill 
during polling would stop nothing. Firing it from the poll, the way 
`test_on_kill_stops_a_job_killed_during_submit` does from the submit, would pin 
that: `mock_poll.side_effect = lambda *a, **k: self.sagemaker.on_kill()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to