umangmystery opened a new issue, #34929: URL: https://github.com/apache/airflow/issues/34929
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Version: [v2.5.1](https://pypi.python.org/pypi/apache-airflow/2.5.1) Git Version: .release:2.5.1+49867b660b6231c1319969217bc61917f7cf9829 We have a self managed Airflow server running on a Medium EC2 instance (Ubuntu 22.04.3 LTS). I have a dag that runs a set of PythonOperator task that connects to Snowflake using Snowflake provider and executes a series of stored procedures on Snowflake. The order of task execution is very important for the ETL and lately several of our DAGs have caused issues due to retries. Redacted DAG snippet: ` from __future__ import annotations import os from datetime import datetime import logging from airflow import DAG from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook from airflow.models import Variable from airflow.operators.empty import EmptyOperator from airflow.utils.trigger_rule import TriggerRule from airflow.exceptions import AirflowFailException from airflow.operators.python import PythonOperator logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) SNOWFLAKE_CONN_ID = "" DAG_ID = "" DATABASE="" SID=f"select max(sid) from {DATABASE}.stg.STAGING_TABLE " # Trigger rules ALL_SUCCESS = 'all_success' ALL_FAILED = 'all_failed' ALL_DONE = 'all_done' ONE_SUCCESS = 'one_success' ONE_FAILED = 'one_failed' dag = DAG( DAG_ID, start_date=datetime(2023, 1, 1), default_args={"snowflake_conn_id": SNOWFLAKE_CONN_ID, "retries":0}, catchup=False, schedule=None ) def call_stored_proc(proc_name): dwh_hook = SnowflakeHook(snowflake_conn_id=SNOWFLAKE_CONN_ID) with dwh_hook.get_conn() as conn: cur = conn.cursor() cur.execute(f'CALL {DATABASE}.STG.{proc_name}({SID});') result = cur.fetchone() if result[0] == 'Procedure executed successfully': logging.info('PROCEDURE EXECUTED SUCCESSFULLY') else: raise AirflowFailException("Check the DAG logs for more information. ERROR FROM SNOWFLAKE: ", result) logging.info(f"QUERY EXECUTION RESULT: {str(result)}") begin_job = EmptyOperator(task_id='Begin') task1 = PythonOperator( task_id = "task1", python_callable=call_stored_proc, dag=dag, retries=0, op_kwargs={ 'proc_name':'task1'} ) task2 = PythonOperator( task_id = "task2", python_callable=call_stored_proc, dag=dag, retries=0, op_kwargs={ 'proc_name':'task2'} ) task3 = PythonOperator( task_id = "task3", python_callable=call_stored_proc, dag=dag, retries=0, op_kwargs={ 'proc_name':'task3'} ) end_job = EmptyOperator(task_id='End') # EXECUTING TASKS begin_job >> task1 >> task2 >> task3 >> end_job` What happens is for some reason task 1 fails and then marks task 2 & task 3 as "upstream failed" and then despite overriding task retrues as retries = 0 at DAG level and task level, the task retries occurs and the execution is successful but the dag stops at task 1 and does not execute the remaining task since the first try failed . Snippet from the Gant chart:  When investigating the logs, something usual popped up when the task went into retry: [2023-10-12, 23:28:44 EDT] {taskinstance.py:1280} INFO - **Starting attempt 2 of 1** Airflow recognized that there is supposed to be only one try but it attempts it again. Following are the redacted logs: **TRY 1: ** `*** Found logs in s3: *** * s3://airflow-logs-aarsvrairflow/logs/dag_id=DATABASE_prod_etl/run_id=manual__2023-10-13T01:39:39.122125+00:00/task_id=task1/attempt=1.log [2023-10-12, 23:16:55 EDT] {taskinstance.py:1083} INFO - Dependencies all met for <TaskInstance: DATABASE_prod_etl.task1 manual__2023-10-13T01:39:39.122125+00:00 [queued]> [2023-10-12, 23:16:55 EDT] {taskinstance.py:1083} INFO - Dependencies all met for <TaskInstance: DATABASE_prod_etl.task1 manual__2023-10-13T01:39:39.122125+00:00 [queued]> [2023-10-12, 23:16:55 EDT] {taskinstance.py:1279} INFO - -------------------------------------------------------------------------------- [2023-10-12, 23:16:55 EDT] {taskinstance.py:1280} INFO - Starting attempt 1 of 1 [2023-10-12, 23:16:55 EDT] {taskinstance.py:1281} INFO - -------------------------------------------------------------------------------- [2023-10-12, 23:16:56 EDT] {taskinstance.py:1300} INFO - Executing <Task(PythonOperator): task1> on 2023-10-13 01:39:39.122125+00:00 [2023-10-12, 23:16:56 EDT] {base_task_runner.py:129} INFO - Running on host: ip-000-000-000-000 [2023-10-12, 23:16:56 EDT] {base_task_runner.py:130} INFO - Running: ['airflow', 'tasks', 'run', 'DATABASE_prod_etl', 'task1', 'manual__2023-10-13T01:39:39.122125+00:00', '--job-id', '3467', '--raw', '--subdir', 'DAGS_FOLDER/DATABASE/DATABASE_prod_etl_dag.py', '--cfg-path', '/tmp/tmpwuazz5fl'] [2023-10-12, 23:17:05 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 /usr/local/lib/python3.10/dist-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) [2023-10-12, 23:17:12 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 [[34m2023-10-12, 23:17:12 EDT[0m] {[34mdagbag.py:[0m538} INFO[0m - Filling up the DagBag from /home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py[0m [2023-10-12, 23:17:18 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 /home/airflow/.local/lib/python3.10/site-packages/snowflake/connector/options.py:107 UserWarning: You have an incompatible version of 'pyarrow' installed (12.0.1), please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == "pandas"' [2023-10-12, 23:17:26 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 [[34m2023-10-12, 23:17:26 EDT[0m] {[34mtask_command.py:[0m388} INFO[0m - Running <TaskInstance: DATABASE_prod_etl.task1 manual__2023-10-13T01:39:39.122125+00:00 [running]> on host ip-000-000-000-000[0m [2023-10-12, 23:17:27 EDT] {taskinstance.py:1507} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=DATABASE_prod_etl AIRFLOW_CTX_TASK_ID=task1 AIRFLOW_CTX_EXECUTION_DATE=2023-10-13T01:39:39.122125+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-10-13T01:39:39.122125+00:00 [2023-10-12, 23:17:27 EDT] {base.py:73} INFO - Using connection ID 'DB_FOLDER_prod' for task execution. [2023-10-12, 23:17:27 EDT] {connection.py:282} INFO - Snowflake Connector for Python Version: 3.0.2, Python Version: 3.10.12, Platform: Linux-6.2.0-1012-aws-x86_64-with-glibc2.35 [2023-10-12, 23:17:27 EDT] {connection.py:989} INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. [2023-10-12, 23:17:27 EDT] {connection.py:1007} INFO - Setting use_openssl_only mode to False [2023-10-12, 23:17:29 EDT] {cursor.py:738} INFO - query: [CALL DATABASE.STG.task1();] [2023-10-12, 23:32:46 EDT] {cursor.py:751} INFO - query execution done [2023-10-12, 23:32:46 EDT] {cursor.py:890} INFO - Number of results in first chunk: 1 [2023-10-12, 23:32:46 EDT] {connection.py:586} INFO - closed [2023-10-12, 23:32:47 EDT] {connection.py:589} INFO - No async queries seem to be running, deleting session [2023-10-12, 23:32:47 EDT] {taskinstance.py:1768} ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 175, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 192, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py", line 84, in load_file_to_stage raise AirflowFailException("Check the DAG logs for more information. ERROR FROM SNOWFLAKE: ", result) airflow.exceptions.AirflowFailException: ('Check the DAG logs for more information. ERROR FROM SNOWFLAKE: ', ('{\n "Error type": "STATEMENT_ERROR",\n "SQLCODE": 0,\n "SQLERRM": "Sequence SID has been dropped or cannot be accessed.",\n "SQLSTATE": "22000"\n}',)) [2023-10-12, 23:32:47 EDT] {taskinstance.py:1318} INFO - Immediate failure requested. Marking task as FAILED. dag_id=DATABASE_prod_etl, task_id=task1, execution_date=20231013T013939, start_date=20231013T032844, end_date=20231013T033247 [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 Traceback (most recent call last): [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/bin/airflow", line 8, in <module> [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 sys.exit(main()) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/__main__.py", line 39, in main [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 args.func(args) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/cli_parser.py", line 52, in command [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 return func(*args, **kwargs) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/utils/cli.py", line 108, in wrapper [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 return f(*args, **kwargs) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 395, in task_run [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 _run_task_by_selected_method(args, dag, ti) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 195, in _run_task_by_selected_method [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 _run_raw_task(args, ti) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 269, in _run_raw_task [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 ti._run_raw_task( [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/utils/session.py", line 75, in wrapper [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 return func(*args, session=session, **kwargs) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 1374, in _run_raw_task [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 self._execute_task_with_callbacks(context, test_mode) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 1520, in _execute_task_with_callbacks [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 result = self._execute_task(context, task_orig) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 1581, in _execute_task [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 result = execute_callable(context=context) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 175, in execute [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 return_value = self.execute_callable() [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 192, in execute_callable [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 return self.python_callable(*self.op_args, **self.op_kwargs) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 File "/home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py", line 84, in load_file_to_stage [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 raise AirflowFailException("Check the DAG logs for more information. ERROR FROM SNOWFLAKE: ", result) [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: Subtask task1 airflow.exceptions.AirflowFailException: ('Check the DAG logs for more information. ERROR FROM SNOWFLAKE: ', ('{\n "Error type": "STATEMENT_ERROR",\n "SQLCODE": 0,\n "SQLERRM": "Sequence AUDIT_SID has been dropped or cannot be accessed.",\n "SQLSTATE": "22000"\n}',)) [2023-10-12, 23:32:52 EDT] {local_task_job.py:208} INFO - Task exited with return code 1 [2023-10-12, 23:32:52 EDT] {taskinstance.py:2578} INFO - 0 downstream tasks scheduled from follow-on schedule check` **TRY 2: ** `*** Found logs in s3: *** * s3://airflow-logs-aarsvrairflow/logs/dag_id=DATABSE/run_id=manual__2023-10-13T01:39:39.122125+00:00/task_id=task1/attempt=2.log [2023-10-12, 23:28:44 EDT] {taskinstance.py:1083} INFO - Dependencies all met for <TaskInstance: DATABSE.task1 manual__2023-10-13T01:39:39.122125+00:00 [queued]> [2023-10-12, 23:28:44 EDT] {taskinstance.py:1083} INFO - Dependencies all met for <TaskInstance: DATABSE.task1 manual__2023-10-13T01:39:39.122125+00:00 [queued]> [2023-10-12, 23:28:44 EDT] {taskinstance.py:1279} INFO - -------------------------------------------------------------------------------- [2023-10-12, 23:28:44 EDT] {taskinstance.py:1280} INFO - Starting attempt 2 of 1 [2023-10-12, 23:28:44 EDT] {taskinstance.py:1281} INFO - -------------------------------------------------------------------------------- [2023-10-12, 23:28:45 EDT] {taskinstance.py:1300} INFO - Executing <Task(PythonOperator): task1> on 2023-10-13 01:39:39.122125+00:00 [2023-10-12, 23:28:45 EDT] {base_task_runner.py:129} INFO - Running on host: ip-000-000-000-000 [2023-10-12, 23:28:45 EDT] {base_task_runner.py:130} INFO - Running: ['airflow', 'tasks', 'run', 'DATABSE', 'task1', 'manual__2023-10-13T01:39:39.122125+00:00', '--job-id', '3468', '--raw', '--subdir', 'DAGS_FOLDER/DB_FOLDER/DATABSE_dag.py', '--cfg-path', '/tmp/tmpg03mtrz3'] [2023-10-12, 23:28:52 EDT] {base_task_runner.py:112} INFO - Job 3468: Subtask task1 /usr/local/lib/python3.10/dist-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) [2023-10-12, 23:28:59 EDT] {base_task_runner.py:112} INFO - Job 3468: Subtask task1 [[34m2023-10-12, 23:28:59 EDT[0m] {[34mdagbag.py:[0m538} INFO[0m - Filling up the DagBag from /home/airflow/airflow/dags/DB_FOLDER/DATABSE_dag.py[0m [2023-10-12, 23:29:05 EDT] {base_task_runner.py:112} INFO - Job 3468: Subtask task1 /home/airflow/.local/lib/python3.10/site-packages/snowflake/connector/options.py:107 UserWarning: You have an incompatible version of 'pyarrow' installed (12.0.1), please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == "pandas"' [2023-10-12, 23:29:13 EDT] {base_task_runner.py:112} INFO - Job 3468: Subtask task1 [[34m2023-10-12, 23:29:13 EDT[0m] {[34mtask_command.py:[0m388} INFO[0m - Running <TaskInstance: DATABSE.task1 manual__2023-10-13T01:39:39.122125+00:00 [running]> on host ip-000-000-000-000 AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=DATABSE AIRFLOW_CTX_TASK_ID=task1 AIRFLOW_CTX_EXECUTION_DATE=2023-10-13T01:39:39.122125+00:00 AIRFLOW_CTX_TRY_NUMBER=2 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-10-13T01:39:39.122125+00:00 [2023-10-12, 23:29:13 EDT] {base.py:73} INFO - Using connection ID 'DB_FOLDER_prod' for task execution. [2023-10-12, 23:29:13 EDT] {connection.py:282} INFO - Snowflake Connector for Python Version: 3.0.2, Python Version: 3.10.12, Platform: Linux-6.2.0-1012-aws-x86_64-with-glibc2.35 [2023-10-12, 23:29:13 EDT] {connection.py:989} INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity. [2023-10-12, 23:29:13 EDT] {connection.py:1007} INFO - Setting use_openssl_only mode to False [2023-10-12, 23:29:16 EDT] {cursor.py:738} INFO - query: [CALL DB.STG.task1();] [2023-10-12, 23:44:14 EDT] {cursor.py:751} INFO - query execution done [2023-10-12, 23:44:14 EDT] {cursor.py:890} INFO - Number of results in first chunk: 1 [2023-10-12, 23:44:14 EDT] {DATABSE_dag.py:82} INFO - PROCEDURE EXECUTED SUCCESSFULLY [2023-10-12, 23:44:14 EDT] {DATABSE_dag.py:85} INFO - QUERY EXECUTION RESULT: ('Procedure executed successfully',) [2023-10-12, 23:44:14 EDT] {connection.py:586} INFO - closed [2023-10-12, 23:44:14 EDT] {connection.py:589} INFO - No async queries seem to be running, deleting session [2023-10-12, 23:44:14 EDT] {python.py:177} INFO - Done. Returned value was: None [2023-10-12, 23:44:14 EDT] {taskinstance.py:1318} INFO - Marking task as SUCCESS. dag_id=DATABSE, task_id=task1, execution_date=20231013T013939, start_date=20231013T032844, end_date=20231013T034414 [2023-10-12, 23:44:18 EDT] {local_task_job.py:208} INFO - Task exited with return code 0 [2023-10-12, 23:44:18 EDT] {taskinstance.py:2578} INFO - 0 downstream tasks scheduled from follow-on schedule check` ### What you think should happen instead Instead of task 1 going into retry, it is supposed to fail the task and the dag. The retries config either at the server config level, DAG level or Task level should be followed but the task retries despite setting retries = 0 at all levels ### How to reproduce Setup an EC2 instance on AWS using Ubuntu 22.04.3. Install Airflow 2.5.1, Snowflake provider 4.0.4, Postgres backend db and all other python libraries specified below. Setup Snowflake connection on Airflow with the database. Create a Stored procedure that performs CRUD operations on Snowflake. Call the stored procedure from a DAG using the Snowflake Hook (Code shared above). Set retries =0 for dags and tasks. Airflow Config (parameters that were changed from default vaues): ` executor = LocalExecutor parallelism = 32 max_active_tasks_per_dag = 16 max_active_runs_per_dag = 16 default_task_retries = 0 auth_backends = airflow.api.auth.backend.basic_auth worker_refresh_batch_size = 1 worker_refresh_interval = 6000 workers = 4 job_heartbeat_sec = 10800 scheduler_heartbeat_sec = 120 num_runs = -1 scheduler_idle_sleep_time = 1 min_file_process_interval = 30 parsing_cleanup_interval = 60 scheduler_health_check_threshold = 240 orphaned_tasks_check_interval = 300.0 scheduler_zombie_task_threshold = 10800 zombie_detection_interval = 10.0 ` ### Operating System Ubuntu 22.04.3 ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==7.2.0 apache-airflow-providers-apache-druid==3.3.1 apache-airflow-providers-apache-hdfs==3.2.0 apache-airflow-providers-apache-hive==5.1.2 apache-airflow-providers-celery==3.1.0 apache-airflow-providers-common-sql==1.3.3 apache-airflow-providers-ftp==3.3.1 apache-airflow-providers-google==8.9.0 apache-airflow-providers-http==4.1.1 apache-airflow-providers-imap==3.1.1 apache-airflow-providers-jdbc==3.3.0 apache-airflow-providers-qubole==3.3.1 apache-airflow-providers-samba==4.1.0 apache-airflow-providers-slack==7.2.0 apache-airflow-providers-snowflake==4.0.4 apache-airflow-providers-sqlite==3.3.1 ### Deployment Virtualenv installation ### Deployment details _No response_ ### Anything else There is no pattern to the occurence. It is random, certain dag runs will not let the task run into retries. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
