Benedetto Vitale created AIRFLOW-5334:
-----------------------------------------
Summary: After Airflow upgrade to 1.10.3 in the gcs to bq operator
it is mandatory to specify either autodetect or schema paramaters
Key: AIRFLOW-5334
URL: https://issues.apache.org/jira/browse/AIRFLOW-5334
Project: Apache Airflow
Issue Type: Bug
Components: gcp
Affects Versions: 1.10.3
Reporter: Benedetto Vitale
In PR [https://github.com/apache/airflow/pull/3880] the following check was
introduced when adding support for autodetect feature in the GCS To BQ operator:
{code:java}
if schema_fields is None and not autodetect:
raise ValueError( 'You must either pass a schema or autodetect=True.')
{code}
But this behaviour is supported at the moment by google apis themselves, so it
indeed added a functionality this PR but also removed one implicitly, was this
expected ?
In our case we have big query schemas already defined with Google Deployment
Manager and we do not need to pass either of those fields, is this a behaviour
you guys consider a bug or is there something wrong with defining the schema
outside Airflow itself according to you ? At the moment the 1.10.3 upgrade
breaks many of our workflows because of that check (a similar check was added
also to the Big Query hook)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
