bhagany commented on code in PR #30053:
URL: https://github.com/apache/airflow/pull/30053#discussion_r1235500903
##########
airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -193,7 +193,7 @@ def _submit_job(
return hook.insert_job(
configuration=configuration,
- project_id=hook.project_id,
+ project_id=configuration["extract"]["sourceTable"]["projectId"],
Review Comment:
@sleepy-tiger I agree, we also have this issue. I think the original bug
report was based on a misunderstanding of the error, and while this fix does
incidentally support the reporter's gcp configuration, I think that is mostly
an accident, and it also breaks many other use cases.
see also
https://github.com/apache/airflow/pull/30053/files#diff-875bf3d1bfbba7067dc754732c0e416b8ebe7a5b722bc9ac428b98934f04a16fR512
and
https://github.com/apache/airflow/pull/30053/files#diff-875bf3d1bfbba7067dc754732c0e416b8ebe7a5b722bc9ac428b98934f04a16fR587,
which override the `project_id` that the user passes in, making it impossible
to use a `project_id` other than what is specified in the source or destination
tables. In general, more clarity is needed in distinguishing between which
projects are being used for storage, and which are being used for compute.
I plan on filing an issue about this later today if one doesn't already
exist, and I'll update here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
