amoghrajesh commented on code in PR #71211: URL: https://github.com/apache/airflow/pull/71211#discussion_r3765751815
########## providers/amazon/docs/operators/glue.rst: ########## @@ -139,6 +139,86 @@ To submit a new AWS Glue job you can use :class:`~airflow.providers.amazon.aws.o The same AWS IAM role used for the crawler can be used here as well, but it will need policies to provide access to the output location for result data. +Durable execution +================== + +``GlueJobOperator`` submits a job run and then polls it to completion on the worker. By default +the operator runs in a *durable* mode that makes this crash-safe: the Glue job run id is +persisted to :doc:`task state store <apache-airflow:core-concepts/task-state-store>` before +polling begins, so if the worker crashes or is preempted and the task is retried, the operator +reconnects to the run that is already executing in Glue instead of starting a new one. + +This matters more for Glue because a Glue job's ``concurrent_run_limit`` defaults to ``1``, so +submitting a second run while the first is still active does not create a harmless duplicate, it +fails outright with ``ConcurrentRunsExceededException`` and the task keeps retrying against a run +it can never see. Durable execution turns that retry into a normal reconnect. + +On retry the operator checks the prior run's state: + +* if it is still starting, running, waiting for capacity, or being stopped, the operator + reconnects and continues polling +* if it already succeeded, the operator returns immediately without resubmitting +* if it failed terminally, or its id has expired and is no longer found, the operator submits the + job fresh +* if it already stopped, the operator submits the job fresh + +A stopped run is treated as a failure, not a success. Glue's API has no way to tell a run +cancelled manually (for example, in the AWS console) apart from one this operator's own +:meth:`~airflow.providers.amazon.aws.operators.glue.GlueJobOperator.on_kill` stopped, which happens +whenever ``stop_job_run_on_kill=True`` and the task is killed -- on SIGTERM, on +``execution_timeout``, or when the task is cleared while running. If the stored state is already +``STOPPED``, the operator submits fresh. If a reconnect finds the run still stopping and it +settles into ``STOPPED`` while polling, the operator raises instead of returning a result -- the +task fails and a normal retry resubmits. Either way, a self-inflicted stop never gets silently +reported as a false success. + +This protection also applies when ``wait_for_completion=False`` -- even though that task attempt +never polls at all, a retry after a successful submission still reconnects rather than +resubmitting, since the run id is persisted immediately after submission regardless of whether the +task waits for it to finish. + +Durable execution requires Airflow 3.3 or newer for the task state store lookup above. On earlier +Airflow versions, or if a prior run was never recorded to the task state store, ``durable=True`` +still recovers a prior run via an older mechanism: the operator checks XCom for a cached run id +first, then falls back to scanning the job's run history for a run tagged with this task +instance's identity, and reconnects if it finds one that is still active. + +Like the persisted state itself, the stored run id isn't deleted automatically, that only happens +when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is longer than +``[state_store] default_retention_days`` (30 days by default) and cleanup runs in between, the run +id won't be there for the next retry, and the operator falls back to the XCom/scan mechanism +above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter +than your longest ``retry_delay``. + +Clearing a task is treated the same as a retry, which matters specifically for a task whose job +already succeeded: clearing does not delete the stored run id, so the next attempt reads it back +and returns immediately without submitting anything to Glue. See +:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the +``[state_store] clear_on_success`` setting that restores "clearing always resubmits." + +To opt out and always start a fresh run on retry, set ``durable=False``: + +.. code-block:: python + + glue_job = GlueJobOperator( + task_id="glue_job", + job_name="my_glue_job", + script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py", + durable=False, + ) + +The task state store lookup above is only used on the synchronous path -- when ``deferrable=True`` +is set, the Triggerer already tracks the run across the wait, so a run id is never persisted there. +``durable`` still has an effect on retry, though: a retry of a deferrable task resubmits by Review Comment: Replaced "resubmits by default" to "would otherwise resubmit." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
