[
https://issues.apache.org/jira/browse/AIRFLOW-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duan Shiqiang updated AIRFLOW-3488:
-----------------------------------
Description:
PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for
postgres data type ARRAY (it generates bigquery schema with that filed as
string).
The generated bigquery schema set mode for one field to 'REPEATED' if the
type_code of the postgres column is one of (1009, 1005, 1007, 1016). see:
[https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py#L185]
. This doesn't include all possible array types (see
[https://github.com/psycopg/psycopg2/blob/master/psycopg/typecast_builtins.c)]
was:
PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for
postgres data type ARRAY (it generates bigquery schema with that filed as
string).
> PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL ARRAYs
> properly
> -------------------------------------------------------------------------------
>
> Key: AIRFLOW-3488
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3488
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, operators
> Affects Versions: 1.10.1
> Reporter: Duan Shiqiang
> Priority: Major
>
> PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for
> postgres data type ARRAY (it generates bigquery schema with that filed as
> string).
> The generated bigquery schema set mode for one field to 'REPEATED' if the
> type_code of the postgres column is one of (1009, 1005, 1007, 1016). see:
> [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py#L185]
> . This doesn't include all possible array types (see
> [https://github.com/psycopg/psycopg2/blob/master/psycopg/typecast_builtins.c)]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
