[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895136#comment-16895136 ] Jark Wu commented on FLINK-13438: - [~lzljs3620320] will help to review this. > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Critical > Fix For: 1.9.0, 1.10.0 > > Attachments: 0001-hive.patch > > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895025#comment-16895025 ] Jark Wu commented on FLINK-13438: - Hi [~lirui], will you help to fix this issue? If yes, I can assign this to you. > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Priority: Blocker > Fix For: 1.9.0, 1.10.0 > > Attachments: 0001-hive.patch > > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894837#comment-16894837 ] Caizhi Weng commented on FLINK-13438: - Hi [~lirui] [~jark] and [~lzljs3620320], sorry for the late response. I actually tried to add support for DataTypes.DATE/TIME/TIMESTAMP before submitting this issue, and my patch is provided in the attachment. In this patch, when running `HiveCatalogDataTypeTest`, two tests will fail due to this issue. Please take a look. > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Priority: Blocker > Fix For: 1.9.0, 1.10.0 > > Attachments: 0001-hive.patch > > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894690#comment-16894690 ] Jingsong Lee commented on FLINK-13438: -- Hi [~lirui] , I think should add tests to DATE/TIME/TIMESTAMP type in Hive source, sink and udx using blink-planner (and maybe using flink-planner too). Hi [~TsReaper], I think you should explain which case and code will lead to this bug in Jira to let the problem more understandable. > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Priority: Blocker > Fix For: 1.9.0, 1.10.0 > > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894616#comment-16894616 ] Rui Li commented on FLINK-13438: [~jark] yes, Hive returns `java.sql.Timestamp` for timestamp columns (at least for the Hive versions we currently support). We can do some conversion between `Timestamp` and `LocalDateTime `. But I suppose there might be issues due to different semantics of these two. E.g. DST has an impact on the string presentation of `Timestamp`, but not `LocalDateTime`. I'll do some investigation to verify. > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Priority: Blocker > Fix For: 1.9.0, 1.10.0 > > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[
https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893796#comment-16893796
]
Jark Wu commented on FLINK-13438:
-
Hi [~lirui], as default, {{DataTypes.TIMESTAMP()}} bridge to the
{{java.time.LocalDateTime}} class. However, we are returning a
{{java.sql.Timestamp}} from Hive table?
> Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
> -
>
> Key: FLINK-13438
> URL: https://issues.apache.org/jira/browse/FLINK-13438
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.9.0, 1.10.0
>
>
> Similar to JDBC connectors, Hive connectors communicate with Flink framework
> using TableSchema, which contains DataType. As the time data read from and
> write to Hive connectors must be java.sql.* types and the default conversion
> class of our time data types are java.time.*, we have to fix Hive connector
> with DataTypes.DATE/TIME/TIMESTAMP support.
> But currently when reading tables from Hive, the table schema is created
> using Hive's schema, so the time types in the created schema will be sql time
> type not local time type. If user specifies a local time type in the table
> schema when creating a table in Hive, he will get a different schema when
> reading it out. This is undesired.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[
https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893758#comment-16893758
]
Rui Li commented on FLINK-13438:
[~TsReaper] Not sure if I understand the issue correctly. When creating
TableSchema for a Hive table, we map Hive's {{TIMESTAMP}} type to
{{DataTypes.TIMESTAMP()}} in Flink. Do you mean there's something wrong with
this type mapping? Or do you mean the records we get for a timestamp column are
of wrong type? If it's the latter, we convert objects between Hive and Flink in
{{HiveInspectors}}, and we can add some conversion for date/timestamp type.
> Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
> -
>
> Key: FLINK-13438
> URL: https://issues.apache.org/jira/browse/FLINK-13438
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Hive
>Reporter: Caizhi Weng
>Priority: Major
> Fix For: 1.9.0, 1.10.0
>
>
> Similar to JDBC connectors, Hive connectors communicate with Flink framework
> using TableSchema, which contains DataType. As the time data read from and
> write to Hive connectors must be java.sql.* types and the default conversion
> class of our time data types are java.time.*, we have to fix Hive connector
> with DataTypes.DATE/TIME/TIMESTAMP support.
> But currently when reading tables from Hive, the table schema is created
> using Hive's schema, so the time types in the created schema will be sql time
> type not local time type. If user specifies a local time type in the table
> schema when creating a table in Hive, he will get a different schema when
> reading it out. This is undesired.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13438) Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support
[ https://issues.apache.org/jira/browse/FLINK-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893699#comment-16893699 ] Jark Wu commented on FLINK-13438: - cc [~lirui], [~xuefuz] > Fix Hive connector with DataTypes.DATE/TIME/TIMESTAMP support > - > > Key: FLINK-13438 > URL: https://issues.apache.org/jira/browse/FLINK-13438 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Caizhi Weng >Priority: Major > > Similar to JDBC connectors, Hive connectors communicate with Flink framework > using TableSchema, which contains DataType. As the time data read from and > write to Hive connectors must be java.sql.* types and the default conversion > class of our time data types are java.time.*, we have to fix Hive connector > with DataTypes.DATE/TIME/TIMESTAMP support. > But currently when reading tables from Hive, the table schema is created > using Hive's schema, so the time types in the created schema will be sql time > type not local time type. If user specifies a local time type in the table > schema when creating a table in Hive, he will get a different schema when > reading it out. This is undesired. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
