[
https://issues.apache.org/jira/browse/AIRFLOW-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074257#comment-17074257
]
ASF GitHub Bot commented on AIRFLOW-7117:
-----------------------------------------
mik-laj commented on pull request #8049: [AIRFLOW-7117] Honor self.schema in
sql_to_gcs as schema to upload
URL: https://github.com/apache/airflow/pull/8049
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Honor self.schema in sql_to_gcs as schema to upload
> ---------------------------------------------------
>
> Key: AIRFLOW-7117
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7117
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Affects Versions: 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9
> Reporter: Hongyi Wang
> Assignee: Hongyi Wang
> Priority: Minor
>
> In
> [sql_to_gcs|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py],
> there is a
> [self.schema|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py#L61-L65],
> which was introduced in
> [pr-3091|https://github.com/apache/airflow/pull/3091/files] to allow user to
> provide their own schema while loading from sql to Google BigQuery.
> This is very useful when user want custom type mapping. For example,
> [mysql.date is default to map
> bigquery.TIMESTAMP|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/mysql_to_gcs.py#L49],
> but sometimes, people want it to be bigquery.DATE.
> However, during
> [mysql_to_gcs|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/mysql_to_gcs.py]
> re-organization, we lost that ability. (Only [use self.schema for
> pre-processing|https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/sql_to_gcs.py#L229-L232],
> while data type is no longer configurable).
> As a conclusion, we need to honor self.schema (if provided) to override
> default schema, like we did before.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
