yuseok89 commented on code in PR #70542:
URL: https://github.com/apache/airflow/pull/70542#discussion_r3699313174
##########
providers/google/src/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -359,7 +344,28 @@ def _handle_job_error(job: BigQueryJob | UnknownJob) ->
None:
if job.error_result:
raise AirflowException(f"BigQuery job {job.job_id} failed:
{job.error_result}")
+ def _warn_on_deprecated_template_fields(self) -> None:
+ if self.src_fmt_configs:
+ warnings.warn(
+ "The 'src_fmt_configs' parameter is deprecated. Use
'extra_config' instead. "
+ "Note: 'extra_config' uses the fully-nested API structure, so
format-specific "
+ "options must be nested under their parent key "
+ "(e.g., {'parquetOptions': {'enableListInference': True}}
rather than "
+ "{'enableListInference': True}).",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
+
def execute(self, context: Context):
+ # Template fields render after __init__, so defaults that depend on a
template field
+ # (schema_object_bucket falls back to bucket) and the src_fmt_configs
deprecation check
+ # must run here, against the rendered values.
+ if self.src_fmt_configs is None:
+ self.src_fmt_configs = {}
+ if self.schema_object_bucket is None:
Review Comment:
Added the log as suggested.
The observable change here is the RTIF None, which is fully restorable by
moving the fallback into render_template_fields, but it'll recur for every
operator in the campaign, so render-vs-log is probably worth deciding once
campaign-wide rather than per-PR.
##########
providers/google/src/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -359,7 +344,28 @@ def _handle_job_error(job: BigQueryJob | UnknownJob) ->
None:
if job.error_result:
raise AirflowException(f"BigQuery job {job.job_id} failed:
{job.error_result}")
+ def _warn_on_deprecated_template_fields(self) -> None:
+ if self.src_fmt_configs:
+ warnings.warn(
+ "The 'src_fmt_configs' parameter is deprecated. Use
'extra_config' instead. "
+ "Note: 'extra_config' uses the fully-nested API structure, so
format-specific "
+ "options must be nested under their parent key "
+ "(e.g., {'parquetOptions': {'enableListInference': True}}
rather than "
+ "{'enableListInference': True}).",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
Review Comment:
Agreed it's vestigial.
Happy to follow whatever the campaign settles on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
