masterlittle opened a new issue, #29307:
URL: https://github.com/apache/airflow/issues/29307
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==8.8.0
### Apache Airflow version
2.5.1
### Operating System
Mac
### Deployment
Astronomer
### Deployment details
_No response_
### What happened
While using BigqueryInsertJobOperator in defferable mode, it fails.
Logs -
`airflow.exceptions.AirflowException: 404, message='Not Found: {\n "error":
{\n "code": 404,\n "message": "Not found: Job
dev-data-platform-294611:airflow_derived_tables_all_tasks_group_temp_user_seg_all_create_stage_table_2023_02_01T04_30_00_00_00_0f2853ad8762909d41067023ddb3c6d8",\n
"errors": [\n {\n "message": "Not found: Job
dev-data-platform-294611:airflow_derived_tables_all_tasks_group_temp_user_seg_all_create_stage_table_2023_02_01T04_30_00_00_00_0f2853ad8762909d41067023ddb3c6d8",\n
"domain": "global",\n "reason": "notFound"\n }\n ],\n
"status": "NOT_FOUND"\n }\n}\n',
url=URL('https://www.googleapis.com/bigquery/v2/projects/dev-data-platform-294611/jobs/airflow_derived_tables_all_tasks_group_temp_user_seg_all_create_stage_table_2023_02_01T04_30_00_00_00_0f2853ad8762909d41067023ddb3c6d8')`
### What you think should happen instead
The bigquery insert job should succeed.
Debugged and the error happens because the google-aio lib does not pass in
the location param when making the GET JOB api call to bigquery.
Acc. to docs if using any region besides us and europe, this location should
be passed.
As seen in the logs, it uses the domain global instead of the location that
is passed to it.
### How to reproduce
In the BigqueryInsertJobOperator, give a location which is not in US (eg -
asia-south1) and make deferrable=true
```
BigQueryInsertJobOperator(
task_id="create_stage_table",
project_id=configuration.task_info.bigquery_info.billing_project_id,
location='asia-south1',
deferrable=True,
configuration={
"query": {
"query": f"{{{{
ti.xcom_pull(task_ids='all_tasks.group_"
f"{configuration.task_info.table_name}.prepare_query') }}}}",
"useLegacySql": False,
"priority": "BATCH",
"writeDisposition": "WRITE_TRUNCATE",
"destinationTable": {
"projectId":
configuration.task_info.project_name,
"datasetId":
configuration.task_info.stage_dataset,
"tableId":
configuration.task_info.stage_table,
},
"clustering": {"fields":
get_clustering_key(configuration)},
"time_partitioning": {
'field':
configuration.task_info.table_info.partition_key,
'type': 'DAY',
},
},
"dryRun": "{{
var.value.get('derived_tables_dry_run', 0) }}",
"labels": get_query_labels(configuration),
},
)
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
