[
https://issues.apache.org/jira/browse/FLINK-30292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648563#comment-17648563
]
Timo Walther edited comment on FLINK-30292 at 12/16/22 10:57 AM:
-----------------------------------------------------------------
Thanks for starting this discussion [~yunfengzhou]. I agree that the current
approach using `ExternalTypeInfo` adds some serialization overhead. It was
introduced to eliminate any friction during conversion of both type systems
during conversion Table API->DataStream API. This purpose was successful.
However, we can further evolve `ExternalTypeInfo` and use more specialized type
info/serializers. In theory, if the `DataType` is of class `Row` with default
conversions in every subfield, `ExternalTypeInfo.of` can also return
`RowTypeInfo` with `BasicTypeInfo.*` etc. This specialized logic has just not
been implemented yet. Since ListView and MapView in UDFs use ExternalTypeInfo
as well we need to be careful for not introducing state incompatibilities.
In any case, there were already discussion about introducing a dedicated
`table-types` module to ease the development and expose common utilities
including the data structure converters.
was (Author: twalthr):
Thanks for starting this discussion [~yunfengzhou]. I agree that the current
approach using `ExternalTypeInfo` adds some serialization overhead. It was
introduced to eliminate any friction during conversion of both type systems
during conversion Table API->DataStream API. This purpose was successful.
However, we can further evolve `ExternalTypeInfo` and use more specialized type
info/serializers. In theory, if the `DataType` is of class `Row` with default
conversions in every subfield, `ExternalTypeInfo.of` can also return
`RowTypeInfo` with `BasicTypeInfo.*` etc. This specialized logic has just not
been implemented yet.
In any case, there were already discussion about introducing a dedicated
`table-types` module to ease the development and expose common utilities
including the data structure converters.
> Better support for conversion between DataType and TypeInformation
> ------------------------------------------------------------------
>
> Key: FLINK-30292
> URL: https://issues.apache.org/jira/browse/FLINK-30292
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
> Affects Versions: 1.15.3
> Reporter: Yunfeng Zhou
> Priority: Major
>
> In Flink 1.15, we have the following ways to convert a DataType to a
> TypeInformation. Each of them has some disadvantages.
> * `TypeConversions.fromDataTypeToLegacyInfo`
> It might lead to precision losses in face of some data types like timestamp.
> It has been deprecated.
> * `ExternalTypeInfo.of`
> It cannot be used to get detailed type information like `RowTypeInfo`
> It might bring some serialization overhead.
> Given that the ways mentioned above are both not perfect, Flink SQL should
> provide a better API to support DataType-TypeInformation conversions, and
> thus better support Table-DataStream conversions.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
