mrn-aglic opened a new issue, #31584:
URL: https://github.com/apache/airflow/issues/31584
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Using Apache Airflow 2.4.3 and apache airflow google provider 8.4 (also
tried with 10.1.0).
We have a query that in production should run for a long time, so we wanted
to make the BigQueryInsertJobOperator deferrable.
Making the operator deferrable runs the job, but the UI and triggerer
process don't seem to be notified that the operator has finished.
I have validated that the query is actually run as the data appears in the
table, but the operator gets stuck in a deferred state.
### What you think should happen instead
After the big query job is finished, the operator should exit it's deferred
state.
### How to reproduce
Skeleton of the code used
```
with DAG(
dag_id="some_dag_id",
schedule="@daily",
catchup=False,
start_date=pendulum.datetime(2023, 5, 8),
):
extract_data = BigQueryInsertJobOperator(
task_id="extract_data",
impersonation_chain=GCP_ASTRO_TEAM_SA.get()
params={"dst_table": _DST_TABLE, "lookback_days": _LOOKBACK_DAYS},
configuration={
"query": {
"query": "{% include 'sql/sql_file.sql' %}",
"useLegacySql": False,
}
},
outlets=DATASET,
execution_timeout=timedelta(hours=2, minutes=30),
deferrable=True,
)
```
### Operating System
Mac OS Ventura 13.3.1
### Versions of Apache Airflow Providers
apache airflow google provider 8.4 (also tried with 10.1.0).
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
We're using astro and on local the Airflow environment is started using
`astro dev start`. The issue appears when running the DAG locally.
An entire other issue (may be unrelated) appears on Sandbox deployment.
### Anything else
Every time the operator is marked as deferrable.
I noticed this a few days ago (week started with Monday May 22nd 2023).
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
