kaxil commented on PR #50300: URL: https://github.com/apache/airflow/pull/50300#issuecomment-2917798170
> I think this PR introduces a bug for deferrable operators. If you look at [AWS dashboard](https://aws-mwaa.github.io/#/open-source/system-tests/dashboard.html), all system tests in deferrable mode are failing. And they are failing since this PR has been merged. > > ## More details > Here is the DAG I am using: > > ```python > with DAG( > dag_id=DAG_ID, > schedule="@once", > start_date=datetime(2021, 1, 1), > tags=["example"], > catchup=False, > ) as dag: > # This context contains the ENV_ID and any env variables requested when the > # task was built above. Access the info as you would any other TaskFlow task. > env_id = "722404908466step" > role_arn = "arn:aws:iam::722404908466:role/RoleSysTest_example_step_functions" > > state_machine_arn = create_state_machine(env_id, role_arn) > > # [START howto_operator_step_function_start_execution] > start_execution = StepFunctionStartExecutionOperator( > task_id="start_execution", state_machine_arn=state_machine_arn, deferrable=True > ) > # [END howto_operator_step_function_start_execution] > > execution_arn = start_execution.output > > # [START howto_sensor_step_function_execution] > wait_for_execution = StepFunctionExecutionSensor( > task_id="wait_for_execution", execution_arn=execution_arn > ) > # [END howto_sensor_step_function_execution] > wait_for_execution.poke_interval = 1 > > chain( > # TEST SETUP > state_machine_arn, > # TEST BODY > start_execution, > wait_for_execution, > ) > ``` > > When I run this DAG with `dag.test`, `execution_arn` is `None`. > > I can see this in logs: > > ``` > [2025-05-23T15:57:34.992+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://in-process.invalid./task-instances/0196fddc-3987-7d7f-9e3b-87db92c63c52/run "HTTP/1.1 200 OK" > 2025-05-23 15:57:34 [debug ] Sending request [task] msg=GetXCom(key='return_value', dag_id='example_step_functions', run_id='manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR', task_id='start_execution', map_index=None, include_prior_dates=False, type='GetXCom') > 2025-05-23 15:57:34 [debug ] Received message from task runner [task] msg=GetXCom(key='return_value', dag_id='example_step_functions', run_id='manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR', task_id='start_execution', map_index=None, include_prior_dates=False, type='GetXCom') > [2025-05-23T15:57:34.999+0000] {_client.py:1026} INFO - HTTP Request: GET http://in-process.invalid./xcoms/example_step_functions/manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR/start_execution/return_value "HTTP/1.1 404 Not Found" > 2025-05-23 15:57:34 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "XCom with key='return_value' map_index=-1 not found for task 'start_execution' in DAG run 'manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR' of 'example_step_functions'"}} > 2025-05-23 15:57:34 [error ] XCom not found [airflow.sdk.api.client] dag_id=example_step_functions detail={'detail': {'reason': 'not_found', 'message': "XCom with key='return_value' map_index=-1 not found for task 'start_execution' in DAG run 'manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR' of 'example_step_functions'"}} key=return_value map_index=None run_id=manual__2025-05-23T15:56:29.405558+00:00_NkzldSVR status_code=404 task_id=start_execution > ``` > > When executed in deferrable mode, the task `start_execution` does not return any value (`None`), which then makes the following task fail (because it needs the return value of `start_execution` to proceed). I looked at several system tests and they are all failing for the same reason. > > Any chance someone knows why this happens? I'll look into the code but somebody has something in top of his mind, that will be faster :) Fixed in https://github.com/apache/airflow/pull/51182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
