[
https://issues.apache.org/jira/browse/FLINK-24776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-24776.
--------------------------------
Fix Version/s: 1.15.0
Release Note: The DecodingFormat interfaces was used for both projectable
and non-projectable formats which led to inconsistent implementations. The
FileSystemTableSource has been updated to distinguish between those two
interfaces now. Users that implemented custom formats for FileSystemTableSource
might need to verify the implementation and make sure to implement
ProjectableDecodingFormat if necessary.
Assignee: Francesco Guardiani
Resolution: Fixed
Fixed in master: e5111c970877b5772f0326ffbc998e0f6a8d351f
> Clarify semantics of DecodingFormat and its data type
> -----------------------------------------------------
>
> Key: FLINK-24776
> URL: https://issues.apache.org/jira/browse/FLINK-24776
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Francesco Guardiani
> Assignee: Francesco Guardiani
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Today the {{org.apache.flink.table.connector.format.DecodingFormat}}
> interface has not clear requirements and it's confusing for implementers. In
> particular, it's unclear whether the format need to support projection push
> down or not, and whether the {{DataType}} provided to
> {{createRuntimeDecoder}} is projected and includes partition keys or not. An
> example of such misunderstanding is shown here:
> https://github.com/apache/flink/blob/991dd0466ff28995a22ded0727ef2a1706d9bddc/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/FileSystemTableSource.java#L107
> The PR https://github.com/apache/flink/pull/17544 partially addresses the
> issue, because it removes the need from BulkFormat implementations to take
> care of partition keys handling. Neverthless, it's still unclear whether
> formats support projections or not and if they support nested projections.
> We should refactor {{DecodingFormat}} as follows:
> - Clarify DecodingFormat and introduce ProjectableDecodingFormat.
> - Introduce ProjectedRowData and Projection to simplify implementations of
> connectors that needs to deal with projections
> - Apply the changes to most of the formats and connectors we have.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
