[
https://issues.apache.org/jira/browse/AIRFLOW-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057747#comment-17057747
]
nexoriv edited comment on AIRFLOW-5334 at 3/12/20, 9:31 AM:
------------------------------------------------------------
If you have the schema defined in Google Deployment Manager why do you need to
set the schema_fields in the operator?
was (Author: nexoriv877):
If you have the schema defined why do you need to set the schema_fields?
> After Airflow upgrade to 1.10.3 in the gcs to bq operator it is mandatory to
> specify either autodetect or schema paramaters
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5334
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5334
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
> Affects Versions: 1.10.3
> Reporter: Benedetto Vitale
> Priority: Major
>
> In PR [https://github.com/apache/airflow/pull/3880] the following check was
> introduced when adding support for autodetect feature in the GCS To BQ
> operator:
> {code:java}
> if schema_fields is None and not autodetect:
> raise ValueError( 'You must either pass a schema or autodetect=True.')
> {code}
> But this behaviour is supported at the moment by google apis themselves, so
> it indeed added a functionality this PR but also removed one implicitly, was
> this expected ?
> In our case we have big query schemas already defined with Google Deployment
> Manager and we do not need to pass either of those fields, is this a
> behaviour you guys consider a bug or is there something wrong with defining
> the schema outside Airflow itself according to you ? At the moment the 1.10.3
> upgrade breaks many of our workflows because of that check (a similar check
> was added also to the Big Query hook)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
