moiseenkov opened a new pull request, #37282: URL: https://github.com/apache/airflow/pull/37282
This PR provides a workaround solution for the issue 35833 (the link is broken intentionally in order to prevent connecting internal PR with the community issue). It is really necessary because users are unable to use `BigQueryInsertJobOperator` in deferrable mode in regions other than `EU` or `US`. This fix was provided now because there is no response neither from `gcloud-aio` nor `google-cloud-bigquery` regarding their plans for providing async [get_job](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get) method with the optional `location` parameter. The proposed solution: 1. The `BigQueryAsyncHook` wraps the call of the synchronous method `BigQueryAsyncHook.get_job()` in `asyncio.loop.run_in_executor()`. 2. The optional parameter `location` is passed from the `BigQueryInsertJobOperator` to the `BigQueryInsertJobTrigger`, where it is handed to the hook. 3. Because the `BigQueryInsertJobTrigger` is a parent class for other four triggers (see below), they were updated in order to comply with the base class attributes. ``` BigQueryCheckTrigger, BigQueryGetDataTrigger, BigQueryIntervalCheckTrigger, BigQueryValueCheckTrigger ``` 4. Despite the modification, triggers from (3) and all the operators that use these triggers and changed async method `BigQueryAsyncHook.get_job_status()` don;t change behavior because the optional parameter `location` is ignored there. However, it would be very nice to fix them too and test carefully in a separated PR: ``` BigQueryCheckOperator, BigQueryGetDataOperator, BigQueryIntervalCheckOperator, BigQueryValueCheckOperator, BigQueryTablePartitionExistenceSensor, BigQueryToGCSOperator, GCSToBigQueryOperator ``` closes: 35833 (the link is broken intentionally in order to prevent connecting internal PR with the community issue) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
