[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949471#comment-16949471 ] Jarek Potiuk commented on AIRFLOW-3978: --- >From looking at the code - no matter what type field it is, if the field is >represented by "bytes", it will be automatically converted into base64 encoded >strings, which is exactly what BigQuery expects. I guess that should cover the >BINARY and VARBINARY case. [~RosterIn] - can you please verify if the case can be closed? If so - I will close the issue. > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: gcs > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945135#comment-16945135
]
jack commented on AIRFLOW-3978:
---
The question is what happens if your MySQL table has BINARY field and you use
MySqlToGoogleCloudStorageOperator and then load it to BigQuery
Will it be recognized as {{BYTES}} in BigQuery with auto detect?
> Add missing types in MySqlToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-3978
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3978
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.2
>Reporter: Roster
>Assignee: Roster
>Priority: Minor
> Labels: gcs
>
> There fields are missing and can not be mapped:
> TIME, BINARY , VARBINARY
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902979#comment-16902979
]
Tomasz Urbaszek commented on AIRFLOW-3978:
--
According to this comment
[https://github.com/apache/airflow/pull/4802#issuecomment-468859708] :
"MySQL FIELD_TYPE (types used in the protocol) is not directly mapped to column
type. For example, VARBINARY is returned as FIELD_TYPE.VAR_STRING."
In the version of MySQLdb used in Airflow one can see that indeed there is no
conversion from BINARY / VARBINARY but STRING / VAR_STRING could be str or
binary.
conversions = {
int: Thing2Str,
long: Thing2Str,
float: Float2Str,
NoneType: None2NULL,
ArrayType: array2Str,
bool: Bool2Str,
Date: Thing2Literal,
DateTimeType: DateTime2literal,
DateTimeDeltaType: DateTimeDelta2literal,
str: Thing2Literal, # default
set: Set2Str,
FIELD_TYPE.TINY: int,
FIELD_TYPE.SHORT: int,
FIELD_TYPE.LONG: long,
FIELD_TYPE.FLOAT: float,
FIELD_TYPE.DOUBLE: float,
FIELD_TYPE.DECIMAL: float,
FIELD_TYPE.NEWDECIMAL: float,
FIELD_TYPE.LONGLONG: long,
FIELD_TYPE.INT24: int,
FIELD_TYPE.YEAR: int,
FIELD_TYPE.SET: Str2Set,
FIELD_TYPE.TIMESTAMP: mysql_timestamp_converter,
FIELD_TYPE.DATETIME: DateTime_or_None,
FIELD_TYPE.TIME: TimeDelta_or_None,
FIELD_TYPE.DATE: Date_or_None,
*FIELD_TYPE.TINY_BLOB: _bytes_or_str,*
*FIELD_TYPE.MEDIUM_BLOB: _bytes_or_str,*
*FIELD_TYPE.LONG_BLOB: _bytes_or_str,*
*FIELD_TYPE.BLOB: _bytes_or_str,*
*FIELD_TYPE.STRING: _bytes_or_str,*
*FIELD_TYPE.VAR_STRING: _bytes_or_str,*
*FIELD_TYPE.VARCHAR: _bytes_or_str,*
}
> Add missing types in MySqlToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-3978
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3978
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.2
>Reporter: Roster
>Assignee: Roster
>Priority: Minor
> Labels: gcs
>
> There fields are missing and can not be mapped:
> TIME, BINARY , VARBINARY
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862371#comment-16862371 ] jack commented on AIRFLOW-3978: --- Partially fixed in [https://github.com/apache/airflow/pull/5196] > Add missing types in MySqlToGoogleCloudStorageOperator > -- > > Key: AIRFLOW-3978 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3978 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.2 >Reporter: Roster >Assignee: Roster >Priority: Minor > Labels: bigquery > Fix For: 1.10.4 > > > There fields are missing and can not be mapped: > TIME, BINARY , VARBINARY -- This message was sent by Atlassian JIRA (v7.6.3#76005)
