umangmystery opened a new issue, #34929:
URL: https://github.com/apache/airflow/issues/34929

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Version: [v2.5.1](https://pypi.python.org/pypi/apache-airflow/2.5.1)
   Git Version: .release:2.5.1+49867b660b6231c1319969217bc61917f7cf9829
   
   We have a self managed Airflow server running on a Medium EC2 instance 
(Ubuntu 22.04.3 LTS).
   
   I have a dag that runs a set of PythonOperator task that connects to 
Snowflake using Snowflake provider and executes a series of stored procedures 
on Snowflake.  The order of task execution is very important for the ETL and 
lately several of our DAGs have caused issues due to retries. 
   
   Redacted DAG snippet: 
   `
   
   from __future__ import annotations
   
   import os
   from datetime import datetime
   import logging
   
   from airflow import DAG
   from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
   from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
   from airflow.models import Variable
   from airflow.operators.empty import EmptyOperator
   from airflow.utils.trigger_rule import TriggerRule
   from airflow.exceptions import AirflowFailException
   from airflow.operators.python import PythonOperator
   
   
   logging.basicConfig(level=logging.INFO)
   logger = logging.getLogger(__name__)
   
   
   
   SNOWFLAKE_CONN_ID = ""
   DAG_ID = ""
   DATABASE=""
   SID=f"select max(sid) from {DATABASE}.stg.STAGING_TABLE "
   
   # Trigger rules
   ALL_SUCCESS = 'all_success'
   ALL_FAILED = 'all_failed'
   ALL_DONE = 'all_done'
   ONE_SUCCESS = 'one_success'
   ONE_FAILED = 'one_failed'
   
   
   dag = DAG(
       DAG_ID,
       start_date=datetime(2023, 1, 1),
       default_args={"snowflake_conn_id": SNOWFLAKE_CONN_ID, "retries":0},
       catchup=False,
       schedule=None
   )
   
   def call_stored_proc(proc_name):
       dwh_hook =  SnowflakeHook(snowflake_conn_id=SNOWFLAKE_CONN_ID)
       with dwh_hook.get_conn() as conn:
           cur = conn.cursor()
   
           cur.execute(f'CALL {DATABASE}.STG.{proc_name}({SID});')
           result = cur.fetchone()
           if result[0] == 'Procedure executed successfully':
               logging.info('PROCEDURE EXECUTED SUCCESSFULLY')
           else:
               raise AirflowFailException("Check the DAG logs for more 
information. ERROR FROM SNOWFLAKE: ", result)
           logging.info(f"QUERY EXECUTION RESULT: {str(result)}")
   
       
   begin_job = EmptyOperator(task_id='Begin')
   
   task1 = PythonOperator(
        task_id = "task1",
        python_callable=call_stored_proc,
        dag=dag,
        retries=0,
        op_kwargs={ 'proc_name':'task1'} 
   )
   
   
   task2 = PythonOperator(
        task_id = "task2",
        python_callable=call_stored_proc,
        dag=dag,
        retries=0,
        op_kwargs={ 'proc_name':'task2'} 
   )
   
   task3 = PythonOperator(
        task_id = "task3",
        python_callable=call_stored_proc,
        dag=dag,
        retries=0,
        op_kwargs={ 'proc_name':'task3'} 
   )
   
   
   end_job = EmptyOperator(task_id='End')
   
   # EXECUTING TASKS
   begin_job >> task1 >> task2 >> task3 >> end_job`
   
   
   What happens is for some reason task 1 fails and then marks task 2 & task 3 
as "upstream failed" and then despite overriding task retrues as retries = 0 at 
DAG level and task level, the task retries occurs and the execution is 
successful but the dag stops at task 1 and does not execute the remaining task 
since the first try failed . 
   
   Snippet from the Gant chart: 
   
![image](https://github.com/apache/airflow/assets/26141535/084d03f9-4945-414d-bc73-db5f8718fe8b)
   
   When investigating the logs, something usual popped up when the task went 
into retry: 
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1280} INFO - **Starting attempt 
2 of 1**
   
   Airflow recognized that there is supposed to be only one try but it attempts 
it again.
   
   Following are the redacted logs: 
   **TRY 1: **
   `*** Found logs in s3:
   ***   * 
s3://airflow-logs-aarsvrairflow/logs/dag_id=DATABASE_prod_etl/run_id=manual__2023-10-13T01:39:39.122125+00:00/task_id=task1/attempt=1.log
   [2023-10-12, 23:16:55 EDT] {taskinstance.py:1083} INFO - Dependencies all 
met for <TaskInstance: DATABASE_prod_etl.task1 
manual__2023-10-13T01:39:39.122125+00:00 [queued]>
   [2023-10-12, 23:16:55 EDT] {taskinstance.py:1083} INFO - Dependencies all 
met for <TaskInstance: DATABASE_prod_etl.task1 
manual__2023-10-13T01:39:39.122125+00:00 [queued]>
   [2023-10-12, 23:16:55 EDT] {taskinstance.py:1279} INFO - 
   
--------------------------------------------------------------------------------
   [2023-10-12, 23:16:55 EDT] {taskinstance.py:1280} INFO - Starting attempt 1 
of 1
   [2023-10-12, 23:16:55 EDT] {taskinstance.py:1281} INFO - 
   
--------------------------------------------------------------------------------
   [2023-10-12, 23:16:56 EDT] {taskinstance.py:1300} INFO - Executing 
<Task(PythonOperator): task1> on 2023-10-13 01:39:39.122125+00:00
   [2023-10-12, 23:16:56 EDT] {base_task_runner.py:129} INFO - Running on host: 
ip-000-000-000-000
   [2023-10-12, 23:16:56 EDT] {base_task_runner.py:130} INFO - Running: 
['airflow', 'tasks', 'run', 'DATABASE_prod_etl', 'task1', 
'manual__2023-10-13T01:39:39.122125+00:00', '--job-id', '3467', '--raw', 
'--subdir', 'DAGS_FOLDER/DATABASE/DATABASE_prod_etl_dag.py', '--cfg-path', 
'/tmp/tmpwuazz5fl']
   [2023-10-12, 23:17:05 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 /usr/local/lib/python3.10/dist-packages/airflow/models/base.py:49 
MovedIn20Warning: Deprecated API features detected! These feature(s) are not 
compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to 
updating applications, ensure requirements files are pinned to 
"sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all 
deprecation warnings.  Set environment variable 
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on 
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
   [2023-10-12, 23:17:12 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 [2023-10-12, 23:17:12 EDT] {dagbag.py:538} 
INFO - Filling up the DagBag from 
/home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py
   [2023-10-12, 23:17:18 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 
/home/airflow/.local/lib/python3.10/site-packages/snowflake/connector/options.py:107
 UserWarning: You have an incompatible version of 'pyarrow' installed (12.0.1), 
please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == 
"pandas"'
   [2023-10-12, 23:17:26 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 [2023-10-12, 23:17:26 EDT] 
{task_command.py:388} INFO - Running <TaskInstance: 
DATABASE_prod_etl.task1 manual__2023-10-13T01:39:39.122125+00:00 [running]> on 
host ip-000-000-000-000
   [2023-10-12, 23:17:27 EDT] {taskinstance.py:1507} INFO - Exporting the 
following env vars:
   AIRFLOW_CTX_DAG_OWNER=airflow
   AIRFLOW_CTX_DAG_ID=DATABASE_prod_etl
   AIRFLOW_CTX_TASK_ID=task1
   AIRFLOW_CTX_EXECUTION_DATE=2023-10-13T01:39:39.122125+00:00
   AIRFLOW_CTX_TRY_NUMBER=1
   AIRFLOW_CTX_DAG_RUN_ID=manual__2023-10-13T01:39:39.122125+00:00
   [2023-10-12, 23:17:27 EDT] {base.py:73} INFO - Using connection ID 
'DB_FOLDER_prod' for task execution.
   [2023-10-12, 23:17:27 EDT] {connection.py:282} INFO - Snowflake Connector 
for Python Version: 3.0.2, Python Version: 3.10.12, Platform: 
Linux-6.2.0-1012-aws-x86_64-with-glibc2.35
   [2023-10-12, 23:17:27 EDT] {connection.py:989} INFO - This connection is in 
OCSP Fail Open Mode. TLS Certificates would be checked for validity and 
revocation status. Any other Certificate Revocation related exceptions or OCSP 
Responder failures would be disregarded in favor of connectivity.
   [2023-10-12, 23:17:27 EDT] {connection.py:1007} INFO - Setting 
use_openssl_only mode to False
   [2023-10-12, 23:17:29 EDT] {cursor.py:738} INFO - query: [CALL 
DATABASE.STG.task1();]
   [2023-10-12, 23:32:46 EDT] {cursor.py:751} INFO - query execution done
   [2023-10-12, 23:32:46 EDT] {cursor.py:890} INFO - Number of results in first 
chunk: 1
   [2023-10-12, 23:32:46 EDT] {connection.py:586} INFO - closed
   [2023-10-12, 23:32:47 EDT] {connection.py:589} INFO - No async queries seem 
to be running, deleting session
   [2023-10-12, 23:32:47 EDT] {taskinstance.py:1768} ERROR - Task failed with 
exception
   Traceback (most recent call last):
     File 
"/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 
175, in execute
       return_value = self.execute_callable()
     File 
"/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 
192, in execute_callable
       return self.python_callable(*self.op_args, **self.op_kwargs)
     File "/home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py", line 
84, in load_file_to_stage
       raise AirflowFailException("Check the DAG logs for more information. 
ERROR FROM SNOWFLAKE: ", result)
   airflow.exceptions.AirflowFailException: ('Check the DAG logs for more 
information. ERROR FROM SNOWFLAKE: ', ('{\n  "Error type": "STATEMENT_ERROR",\n 
 "SQLCODE": 0,\n  "SQLERRM": "Sequence SID has been dropped or cannot be 
accessed.",\n  "SQLSTATE": "22000"\n}',))
   [2023-10-12, 23:32:47 EDT] {taskinstance.py:1318} INFO - Immediate failure 
requested. Marking task as FAILED. dag_id=DATABASE_prod_etl, task_id=task1, 
execution_date=20231013T013939, start_date=20231013T032844, 
end_date=20231013T033247
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 Traceback (most recent call last):
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File "/usr/local/bin/airflow", line 8, in <module>
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     sys.exit(main())
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/__main__.py", line 39, in main
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     args.func(args)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/cli/cli_parser.py", line 52, 
in command
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     return func(*args, **kwargs)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/utils/cli.py", line 108, in 
wrapper
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     return f(*args, **kwargs)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", 
line 395, in task_run
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     _run_task_by_selected_method(args, dag, ti)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", 
line 195, in _run_task_by_selected_method
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     _run_raw_task(args, ti)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", 
line 269, in _run_raw_task
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     ti._run_raw_task(
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/utils/session.py", line 75, in 
wrapper
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     return func(*args, session=session, **kwargs)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 
1374, in _run_raw_task
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     self._execute_task_with_callbacks(context, test_mode)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 
1520, in _execute_task_with_callbacks
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     result = self._execute_task(context, task_orig)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 
1581, in _execute_task
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     result = execute_callable(context=context)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 
175, in execute
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     return_value = self.execute_callable()
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/usr/local/lib/python3.10/dist-packages/airflow/operators/python.py", line 
192, in execute_callable
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     return self.python_callable(*self.op_args, **self.op_kwargs)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1   File 
"/home/airflow/airflow/dags/DATABASE/DATABASE_prod_etl_dag.py", line 84, in 
load_file_to_stage
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1     raise AirflowFailException("Check the DAG logs for more 
information. ERROR FROM SNOWFLAKE: ", result)
   [2023-10-12, 23:32:47 EDT] {base_task_runner.py:112} INFO - Job 3467: 
Subtask task1 airflow.exceptions.AirflowFailException: ('Check the DAG logs for 
more information. ERROR FROM SNOWFLAKE: ', ('{\n  "Error type": 
"STATEMENT_ERROR",\n  "SQLCODE": 0,\n  "SQLERRM": "Sequence AUDIT_SID has been 
dropped or cannot be accessed.",\n  "SQLSTATE": "22000"\n}',))
   [2023-10-12, 23:32:52 EDT] {local_task_job.py:208} INFO - Task exited with 
return code 1
   [2023-10-12, 23:32:52 EDT] {taskinstance.py:2578} INFO - 0 downstream tasks 
scheduled from follow-on schedule check`
   
   **TRY 2: **
   
   `*** Found logs in s3:
   ***   * 
s3://airflow-logs-aarsvrairflow/logs/dag_id=DATABSE/run_id=manual__2023-10-13T01:39:39.122125+00:00/task_id=task1/attempt=2.log
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1083} INFO - Dependencies all 
met for <TaskInstance: DATABSE.task1 manual__2023-10-13T01:39:39.122125+00:00 
[queued]>
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1083} INFO - Dependencies all 
met for <TaskInstance: DATABSE.task1 manual__2023-10-13T01:39:39.122125+00:00 
[queued]>
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1279} INFO - 
   
--------------------------------------------------------------------------------
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1280} INFO - Starting attempt 2 
of 1
   [2023-10-12, 23:28:44 EDT] {taskinstance.py:1281} INFO - 
   
--------------------------------------------------------------------------------
   [2023-10-12, 23:28:45 EDT] {taskinstance.py:1300} INFO - Executing 
<Task(PythonOperator): task1> on 2023-10-13 01:39:39.122125+00:00
   [2023-10-12, 23:28:45 EDT] {base_task_runner.py:129} INFO - Running on host: 
ip-000-000-000-000
   [2023-10-12, 23:28:45 EDT] {base_task_runner.py:130} INFO - Running: 
['airflow', 'tasks', 'run', 'DATABSE', 'task1', 
'manual__2023-10-13T01:39:39.122125+00:00', '--job-id', '3468', '--raw', 
'--subdir', 'DAGS_FOLDER/DB_FOLDER/DATABSE_dag.py', '--cfg-path', 
'/tmp/tmpg03mtrz3']
   [2023-10-12, 23:28:52 EDT] {base_task_runner.py:112} INFO - Job 3468: 
Subtask task1 /usr/local/lib/python3.10/dist-packages/airflow/models/base.py:49 
MovedIn20Warning: Deprecated API features detected! These feature(s) are not 
compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to 
updating applications, ensure requirements files are pinned to 
"sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all 
deprecation warnings.  Set environment variable 
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on 
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
   [2023-10-12, 23:28:59 EDT] {base_task_runner.py:112} INFO - Job 3468: 
Subtask task1 [2023-10-12, 23:28:59 EDT] {dagbag.py:538} 
INFO - Filling up the DagBag from 
/home/airflow/airflow/dags/DB_FOLDER/DATABSE_dag.py
   [2023-10-12, 23:29:05 EDT] {base_task_runner.py:112} INFO - Job 3468: 
Subtask task1 
/home/airflow/.local/lib/python3.10/site-packages/snowflake/connector/options.py:107
 UserWarning: You have an incompatible version of 'pyarrow' installed (12.0.1), 
please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == 
"pandas"'
   [2023-10-12, 23:29:13 EDT] {base_task_runner.py:112} INFO - Job 3468: 
Subtask task1 [2023-10-12, 23:29:13 EDT] 
{task_command.py:388} INFO - Running <TaskInstance: DATABSE.task1 
manual__2023-10-13T01:39:39.122125+00:00 [running]> on host ip-000-000-000-000
   AIRFLOW_CTX_DAG_OWNER=airflow
   AIRFLOW_CTX_DAG_ID=DATABSE
   AIRFLOW_CTX_TASK_ID=task1
   AIRFLOW_CTX_EXECUTION_DATE=2023-10-13T01:39:39.122125+00:00
   AIRFLOW_CTX_TRY_NUMBER=2
   AIRFLOW_CTX_DAG_RUN_ID=manual__2023-10-13T01:39:39.122125+00:00
   [2023-10-12, 23:29:13 EDT] {base.py:73} INFO - Using connection ID 
'DB_FOLDER_prod' for task execution.
   [2023-10-12, 23:29:13 EDT] {connection.py:282} INFO - Snowflake Connector 
for Python Version: 3.0.2, Python Version: 3.10.12, Platform: 
Linux-6.2.0-1012-aws-x86_64-with-glibc2.35
   [2023-10-12, 23:29:13 EDT] {connection.py:989} INFO - This connection is in 
OCSP Fail Open Mode. TLS Certificates would be checked for validity and 
revocation status. Any other Certificate Revocation related exceptions or OCSP 
Responder failures would be disregarded in favor of connectivity.
   [2023-10-12, 23:29:13 EDT] {connection.py:1007} INFO - Setting 
use_openssl_only mode to False
   [2023-10-12, 23:29:16 EDT] {cursor.py:738} INFO - query: [CALL 
DB.STG.task1();]
   [2023-10-12, 23:44:14 EDT] {cursor.py:751} INFO - query execution done
   [2023-10-12, 23:44:14 EDT] {cursor.py:890} INFO - Number of results in first 
chunk: 1
   [2023-10-12, 23:44:14 EDT] {DATABSE_dag.py:82} INFO - PROCEDURE EXECUTED 
SUCCESSFULLY
   [2023-10-12, 23:44:14 EDT] {DATABSE_dag.py:85} INFO - QUERY EXECUTION 
RESULT: ('Procedure executed successfully',)
   [2023-10-12, 23:44:14 EDT] {connection.py:586} INFO - closed
   [2023-10-12, 23:44:14 EDT] {connection.py:589} INFO - No async queries seem 
to be running, deleting session
   [2023-10-12, 23:44:14 EDT] {python.py:177} INFO - Done. Returned value was: 
None
   [2023-10-12, 23:44:14 EDT] {taskinstance.py:1318} INFO - Marking task as 
SUCCESS. dag_id=DATABSE, task_id=task1, execution_date=20231013T013939, 
start_date=20231013T032844, end_date=20231013T034414
   [2023-10-12, 23:44:18 EDT] {local_task_job.py:208} INFO - Task exited with 
return code 0
   [2023-10-12, 23:44:18 EDT] {taskinstance.py:2578} INFO - 0 downstream tasks 
scheduled from follow-on schedule check`
   
   
   
   
   
   ### What you think should happen instead
   
   Instead of task 1 going into retry, it is supposed to fail the task and the 
dag. The retries config either at the server config level, DAG level or Task 
level should be followed but the task retries despite setting retries = 0 at 
all levels 
   
   ### How to reproduce
   
   Setup an EC2 instance on AWS using Ubuntu 22.04.3. 
   Install Airflow 2.5.1, Snowflake provider 4.0.4, Postgres backend db  and 
all other python libraries specified below. 
   Setup Snowflake connection on Airflow with the database. 
   Create a Stored procedure that performs CRUD operations on Snowflake. 
   Call the stored procedure from a DAG using the Snowflake Hook (Code shared 
above). 
   
   Set retries =0 for dags and tasks. 
   
   Airflow Config (parameters that were changed from default vaues): 
   `
   executor = LocalExecutor
   parallelism = 32
   max_active_tasks_per_dag = 16
   max_active_runs_per_dag = 16
   default_task_retries = 0
   auth_backends = airflow.api.auth.backend.basic_auth
   worker_refresh_batch_size = 1
   worker_refresh_interval = 6000
   workers = 4
   job_heartbeat_sec = 10800
   scheduler_heartbeat_sec = 120
   num_runs = -1
   scheduler_idle_sleep_time = 1
   min_file_process_interval = 30
   parsing_cleanup_interval = 60
   scheduler_health_check_threshold = 240
   orphaned_tasks_check_interval = 300.0
   scheduler_zombie_task_threshold = 10800
   zombie_detection_interval = 10.0
   `
   
   
   ### Operating System
   
   Ubuntu 22.04.3
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==7.2.0
   apache-airflow-providers-apache-druid==3.3.1
   apache-airflow-providers-apache-hdfs==3.2.0
   apache-airflow-providers-apache-hive==5.1.2
   apache-airflow-providers-celery==3.1.0
   apache-airflow-providers-common-sql==1.3.3
   apache-airflow-providers-ftp==3.3.1
   apache-airflow-providers-google==8.9.0
   apache-airflow-providers-http==4.1.1
   apache-airflow-providers-imap==3.1.1
   apache-airflow-providers-jdbc==3.3.0
   apache-airflow-providers-qubole==3.3.1
   apache-airflow-providers-samba==4.1.0
   apache-airflow-providers-slack==7.2.0
   apache-airflow-providers-snowflake==4.0.4
   apache-airflow-providers-sqlite==3.3.1
   
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   There is no pattern to the occurence. It is random, certain dag runs will 
not let the task run into retries. 
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to