[
https://issues.apache.org/jira/browse/FLINK-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533509#comment-16533509
]
ASF GitHub Bot commented on FLINK-8558:
---------------------------------------
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/6264
[FLINK-8558] [table] Add unified format interfaces and separate formats
from connectors
## What is the purpose of the change
This PR introduces a format discovery mechanism based on Java Service
Providers. The general `TableFormatFactory` is similar to the existing table
source discovery mechanism. However, it allows for arbitrary format interfaces
that might be introduced in the future. At the moment, a connector can request
configured instances of `DeserializationSchema` and `SerializationSchema`. In
the future we can add interfaces such as a `Writer` or
`KeyedSerializationSchema` without breaking backwards compatibility.
This PR deprecates the existing strong coupling of connector and format for
the Kafa table sources and table source factories. It introduces
descriptor-based alternatives.
## Brief change log
- Introduction of `TableFormatService` with `TableFormatFactory` and
specific `DeserializationSchemaFactory` and `SerializationSchemaFactory`
- Decoupling of existing connectors (i.e. Kafka) from formats (i.e. JSON
and Avro)
- Exposing the descriptor-based approach, deprecate the old builders and
make table source internal
## Verifying this change
- Existing tests for coupled sources and factories are still working
- New tests for format discovery, formats, and decoupled table sources
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? more docs for the descriptors
will follow in a separate PR
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink FLINK-8558
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6264.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6264
----
commit 9415066578d6fac420ea68edfa4509d5625f2a82
Author: Timo Walther <twalthr@...>
Date: 2018-06-27T11:16:49Z
[FLINK-8558] [table] Add unified format interfaces and separate formats
from connectors
This PR introduces a format discovery mechanism based on Java Service
Providers. The general `TableFormatFactory` is similar to the existing table
source discovery mechanism. However, it allows for arbirary format interfaces
that might be introduced in the future. At the moment, a connector can request
configured instances of `DeserializationSchema` and `SerializationSchema`. In
the future we can add interfaces such as a `Writer` or
`KeyedSerializationSchema` without breaking backwards compatibility.
This PR deprecates the existing strong coupling of connector and format for
the Kafa table sources and table source factories. It introduces
descriptor-based alternatives.
----
> Add unified format interfaces and format discovery
> --------------------------------------------------
>
> Key: FLINK-8558
> URL: https://issues.apache.org/jira/browse/FLINK-8558
> Project: Flink
> Issue Type: New Feature
> Components: Streaming Connectors
> Reporter: Timo Walther
> Assignee: Timo Walther
> Priority: Major
> Labels: pull-request-available
>
> In the last release, we introduced a new module {{flink-formats}}. Currently
> only {{flink-avro}} is located there but we will add more formats such as
> {{flink-json}}, {{flink-protobuf}}, and so on. For better separation of
> concerns we want decouple connectors from formats: e.g., remove
> {{KafkaAvroTableSource}} and {{KafkaJsonTableSource}}.
> A newly introduced {{FormatFactory}} will use Java service loaders to
> discovery available formats in the classpath (similar to how file systems are
> discovered now). A {{Format}} will provide a method for converting {{byte[]}}
> to target record type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
