[
https://issues.apache.org/jira/browse/AIRFLOW-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799815#comment-16799815
]
ASF subversion and git services commented on AIRFLOW-3541:
----------------------------------------------------------
Commit fe8c9a513d468d230b5cca77535d60f18295f05d in airflow's branch
refs/heads/master from BrechtDeVlieger
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fe8c9a5 ]
[AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
> Include --useAvroLogicalTypes setting for the Airflow operator
> GoogleCloudStorageToBigQueryOperator
> ---------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3541
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3541
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Environment: Composer
> Reporter: Monica Parpi
> Priority: Major
> Labels: pull-request-available
>
> Getting an error [1] when loading data from GCS into BigQuery using the
> GoogleCloudStorageToBigQueryOperator for a Avro schema containing this column:
> {
> "name": "date_requested",
> "type": "int",
> "logicalType": "date",
> }
> This column contains the number of days since the epoch as indicated here [2].
> It is possible to load this data into a DATE column by using the
> "--useAvroLogicalTypes" parameter when initiating a load using the CLI [3]
> It would be very useful if this setting could be enabled via the Airflow
> operator that performs the equivalent data load [4].
> Thank you!
> [ [1]
> https://issuetracker.google.com/35905894|https://issuetracker.google.com/35905894]
>
> [[2]
> http://avro.apache.org/docs/current/spec.html#Date|http://avro.apache.org/docs/current/spec.html#Date]
> [3]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.useAvroLogicalTypes]
> [4]
> [https://airflow.apache.org/_modules/airflow/contrib/operators/gcs_to_bq.html#GoogleCloudStorageToBigQueryOperator]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
