amoghrajesh commented on code in PR #76964:
URL: https://github.com/apache/airflow/pull/76964#discussion_r4611490568
##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
:start-after: [START howto_operator_sagemaker_training]
:end-before: [END howto_operator_sagemaker_training]
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to
completion on the
+worker. By default the operator runs in a *durable* mode that makes this
crash-safe: the training
+job name is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer
found, the operator
+ submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt
is treated the same
+as a retry. A task whose job already completed returns the existing result
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set
``keep_task_state`` when
Review Comment:
Added a paragraph on the second billed job, pointing at `keep_task_state` or
`cancel_on_kill=True`.
##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
:start-after: [START howto_operator_sagemaker_training]
:end-before: [END howto_operator_sagemaker_training]
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to
completion on the
+worker. By default the operator runs in a *durable* mode that makes this
crash-safe: the training
+job name is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer
found, the operator
+ submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt
is treated the same
+as a retry. A task whose job already completed returns the existing result
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set
``keep_task_state`` when
+clearing to keep it, and the next attempt then reconnects to the job or
returns its result. Clearing
+a whole Dag run, marking a task as failed or success, and ``airflow tasks
clear`` still keep the job
Review Comment:
Added
##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,59 @@ To create an Amazon Sagemaker training job you can use
:start-after: [START howto_operator_sagemaker_training]
:end-before: [END howto_operator_sagemaker_training]
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to
completion on the
+worker. By default the operator runs in a *durable* mode that makes this
crash-safe: the training
+job name is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer
found, the operator
+ submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the
checkpoint
+``S3Uri`` does not depend on the job name.
+
+On Airflow 3.3, clearing a task keeps the stored job name, so the next attempt
is treated the same
+as a retry. A task whose job already completed returns the existing result
without starting a new
+job.
+
+From Airflow 3.4, clearing a single task instance (UI, REST API or
``airflowctl``) deletes the
+stored job name by default, so the next attempt submits a new job. Set
``keep_task_state`` when
+clearing to keep it, and the next attempt then reconnects to the job or
returns its result. Clearing
+a whole Dag run, marking a task as failed or success, and ``airflow tasks
clear`` still keep the job
+name before Airflow 3.5. From Airflow 3.5, every way of clearing deletes it by
default.
+
+See :doc:`apache-airflow:core-concepts/resumable-tasks` for the ways of
clearing, and for the
+``[state_store] clear_on_success`` setting.
+
+To opt out and always start a new job on retry, set ``durable=False``:
+
+.. code-block:: python
+
+ training = SageMakerTrainingOperator(
+ task_id="train_model",
+ config=training_config,
+ durable=False,
+ )
+
+Durable execution applies to the synchronous path. When ``deferrable=True`` is
set, ``durable`` has
Review Comment:
Now scoped to `deferrable=True` with `wait_for_completion=True`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]