dondaum commented on issue #38532:
URL: https://github.com/apache/airflow/issues/38532#issuecomment-2049580920
I tried to reproduce the exact error but with no success. I tried to
reproduce it with the following DAG:
```Python
import datetime
import os
from airflow.providers.google.cloud.operators.bigquery import
BigQueryInsertJobOperator
from airflow import DAG
WAIT_QUERY = """
DECLARE retry_count INT64;
DECLARE success BOOL;
DECLARE size_bytes INT64;
DECLARE row_count INT64;
DECLARE DELAY_TIME DATETIME;
DECLARE WAIT STRING;
SET retry_count = 2;
SET success = FALSE;
WHILE retry_count <= 3 AND success = FALSE DO
BEGIN
SET row_count = (with a as (SELECT 1 as b) SELECT * FROM a WHERE 1 = 2);
IF row_count > 0 THEN
SELECT 'Table Exists!' as message, retry_count as retries;
SET success = TRUE;
ELSE
SELECT 'Table does not exist' as message, retry_count as retries,
row_count;
SET retry_count = retry_count + 1;
-- WAITFOR DELAY '00:00:10';
SET WAIT = 'TRUE';
SET DELAY_TIME = DATETIME_ADD(CURRENT_DATETIME,INTERVAL 90 SECOND);
WHILE WAIT = 'TRUE' DO
IF (DELAY_TIME < CURRENT_DATETIME) THEN
SET WAIT = 'FALSE';
END IF;
END WHILE;
END IF;
END;
END WHILE;
"""
with DAG(
dag_id=os.path.splitext(os.path.basename(__file__))[0],
schedule=None,
start_date=datetime.datetime(2024, 1, 1),
catchup=False,
tags=["testing"],
) as dag:
for i in range(10):
bq_task = BigQueryInsertJobOperator(
task_id=f"debug_query_{i}",
configuration={
"query": {
"query": WAIT_QUERY,
"useLegacySql": False,
"priority": "BATCH",
}
},
location="europe-west3",
deferrable=True,
)
```
Also, I set the retry option in the GCP connection to 0 so as not to
implicitly retry on failure.
Could you perhaps create a DAG that reproduces the error? And maybe you
could also check which apache-airflow-providers-google you are using?
My setup:
```
Apache Airflow
version | 2.7.3
executor | LocalExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn |
postgresql+psycopg2://airflow:airflow@postgres/airflow
dags_folder | /opt/airflow/dags
plugins_folder | /opt/airflow/plugins
base_log_folder | /opt/airflow/logs
remote_base_log_folder |
System info
OS | Linux
architecture | x86_64
uname | uname_result(system='Linux', node='42d8cf034cfc',
release='5.10.16.3-microsoft-standard-WSL2', version='#1 SMP Fri Apr 2 22:23:49
UTC 2021', machine='x86_64')
locale | ('en_US', 'UTF-8')
python_version | 3.11.6 (main, Nov 1 2023, 14:02:22) [GCC 10.2.1 20210110]
python_location | /usr/local/bin/python
Tools info
git | NOT AVAILABLE
ssh | OpenSSH_8.4p1 Debian-5+deb11u2, OpenSSL 1.1.1w 11 Sep
2023
kubectl | NOT AVAILABLE
gcloud | NOT AVAILABLE
cloud_sql_proxy | NOT AVAILABLE
mysql | mysql Ver 8.0.35 for Linux on x86_64 (MySQL Community
Server - GPL)
sqlite3 | 3.34.1 2021-01-20 14:10:07
10e20c0b43500cfb9bbc0eaa061c57514f715d87238f4d835880cd846b9ealt1
psql | psql (PostgreSQL) 16.0 (Debian 16.0-1.pgdg110+1)
Paths info
airflow_home | /opt/airflow
system_path |
/root/bin:/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
python_path |
/home/airflow/.local/bin:/usr/local/lib/python311.zip:/usr/local/lib/python3.11:/usr/local/lib/python3.11/lib-dynload:/home/airflow/.local/lib/python3.11/site-packages:
|
/usr/local/lib/python3.11/site-packages:/opt/airflow/dags:/opt/airflow/config:/opt/airflow/plugins
airflow_on_path | True
Providers info
apache-airflow-providers-amazon | 8.10.0
apache-airflow-providers-apache-beam | 5.6.2
apache-airflow-providers-celery | 3.6.0
apache-airflow-providers-cncf-kubernetes | 8.0.1
apache-airflow-providers-common-sql | 1.11.1
apache-airflow-providers-daskexecutor | 1.1.0
apache-airflow-providers-dbt-cloud | 3.7.0
apache-airflow-providers-docker | 3.8.0
apache-airflow-providers-elasticsearch | 5.1.0
apache-airflow-providers-ftp | 3.7.0
apache-airflow-providers-google | 10.16.0
apache-airflow-providers-grpc | 3.3.0
apache-airflow-providers-hashicorp | 3.6.4
apache-airflow-providers-http | 4.10.0
apache-airflow-providers-imap | 3.5.0
apache-airflow-providers-microsoft-azure | 8.1.0
apache-airflow-providers-mysql | 5.5.4
apache-airflow-providers-odbc | 4.1.0
apache-airflow-providers-openlineage | 1.2.0
apache-airflow-providers-postgres | 5.10.2
apache-airflow-providers-redis | 3.4.0
apache-airflow-providers-sendgrid | 3.4.0
apache-airflow-providers-sftp | 4.9.0
apache-airflow-providers-slack | 8.3.0
apache-airflow-providers-snowflake | 5.1.0
apache-airflow-providers-sqlite | 3.7.1
apache-airflow-providers-ssh | 3.10.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
