[
https://issues.apache.org/jira/browse/FLINK-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542614#comment-16542614
]
ASF GitHub Bot commented on FLINK-8558:
---------------------------------------
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/6323
[FLINK-8558] [FLINK-8866] [table] Finalize unified table source/sink/format
interfaces
## What is the purpose of the change
This PR finalizes the efforts done in #6264 and #6201 for having unified
interfaces for table sources, table sinks, and table formats. It reduces code
duplication and cleans up the code base around factories.
## Brief change log
- Introduction of `org.apache.table.factories.TableFactory` a common
interface for factories
- Introduction of `org.apache.table.factories.TableFormatFactory` a
specific table factory for formats
- Specific factories for `StreamTableSource`, `StreamTableSink`,
`BatchTableSource`, `BatchTableSink`, `DeserializationSchema`, and
`SerializationSchema`
- Deprecation of old format-specific table sources (sinks will be
deprecated in a follow-up PR)
- Possibility to register table source and sink under a common name (table
type `both` in SQL Client YAML)
## Verifying this change
- Existing tests verify the implementation
- Additional ITCases and unit tests have been added
- (An end-to-end test will follow in a separate PR)
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? not documented
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink UnifiedInterfacesFinal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6323.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6323
----
commit 980499f887d72ddf9a405c4ad200d0cab15d889c
Author: Timo Walther <twalthr@...>
Date: 2018-06-27T11:16:49Z
[FLINK-8558] [table] Add unified format interfaces and separate formats
from connectors
This PR introduces a format discovery mechanism based on Java Service
Providers. The general `TableFormatFactory` is similar to the existing table
source discovery mechanism. However, it allows for arbirary format interfaces
that might be introduced in the future. At the moment, a connector can request
configured instances of `DeserializationSchema` and `SerializationSchema`. In
the future we can add interfaces such as a `Writer` or
`KeyedSerializationSchema` without breaking backwards compatibility.
This PR deprecates the existing strong coupling of connector and format for
the Kafa table sources and table source factories. It introduces
descriptor-based alternatives.
commit 42a8a156d4e6f8f3d119c458350b6c897306fc48
Author: Shuyi Chen <shuyi@...>
Date: 2018-06-19T19:00:34Z
[FLINK-8866] [table] Create unified interfaces to configure and instatiate
TableSinks
This closes #6201.
commit 311dc62e59c0e4146c094b73c21b979f31b2e1d9
Author: Timo Walther <twalthr@...>
Date: 2018-07-11T11:29:03Z
Rename to TableFactory and move it to factories package
commit 1c581cba61ba321bb6de6a4d298a881840d11cfe
Author: Timo Walther <twalthr@...>
Date: 2018-07-11T11:46:31Z
Refactor format factories
commit 5c6df7598d1f1c3c698ae9b6b35eb37d7fff8295
Author: Timo Walther <twalthr@...>
Date: 2018-07-12T06:35:00Z
Unify table factories
commit 0cd7c44c006aba21c32d8785d17bfc3dbee03916
Author: Timo Walther <twalthr@...>
Date: 2018-07-12T07:05:50Z
Move table type out of descriptors
commit 6b83f2e1c0e63147f049dc5389c5633077b789a4
Author: Timo Walther <twalthr@...>
Date: 2018-07-12T08:50:09Z
Make source/sink factories environment-dependent
commit 4f1255fd003080f078afe6ef67ffa58f40ffec36
Author: Timo Walther <twalthr@...>
Date: 2018-07-12T18:48:45Z
Clean up and simplify changes
----
> Add unified format interfaces and format discovery
> --------------------------------------------------
>
> Key: FLINK-8558
> URL: https://issues.apache.org/jira/browse/FLINK-8558
> Project: Flink
> Issue Type: New Feature
> Components: Streaming Connectors
> Reporter: Timo Walther
> Assignee: Timo Walther
> Priority: Major
> Labels: pull-request-available
>
> In the last release, we introduced a new module {{flink-formats}}. Currently
> only {{flink-avro}} is located there but we will add more formats such as
> {{flink-json}}, {{flink-protobuf}}, and so on. For better separation of
> concerns we want decouple connectors from formats: e.g., remove
> {{KafkaAvroTableSource}} and {{KafkaJsonTableSource}}.
> A newly introduced {{FormatFactory}} will use Java service loaders to
> discovery available formats in the classpath (similar to how file systems are
> discovered now). A {{Format}} will provide a method for converting {{byte[]}}
> to target record type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
