[
https://issues.apache.org/jira/browse/FLINK-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352657#comment-16352657
]
Fabian Hueske commented on FLINK-8538:
--------------------------------------
Hi [~xccui], please see my comments:
# Yes, the idea is to make this quite modular. Not only for users that
implement other connectors, but also for us. For instance, we'd like to reuse
code for JSON formats when reading from Kafka, Kinesis, Pravega, Files, etc.
# This is not clear yet and will be driven by user demand. We'll focus on the
most popular systems for which there are source functions that we can wrap
(Kafka, Kinesis, file system, etc.)
# I think so too. Most of the existing sources (which are not too many) would
need to be refactored to become more modular. For instance, it does not really
make sense to have Kafka table sources for each version of Kafka and (Avro,
JSON). This will become hard to maintain once we add more formats like
ProtoBuf, CSV, or ...
# The current version does not feature format factories, but this is something
we are thinking about to add soon as well. A format factory would need to
provide a format parser that parses data in the format but also returns the
{{TypeInformation}} of the returned type.
It's great that you want to help with this effort! It would be great if we
could have a first {{KafkaJsonTableSourceFactory}} soon, to have a Kafka source
that can be used with the SQL CLI client. Starting from the
+CsvTableSourceFactory+ sounds like a good idea to me. We should make sure
though, that the things I discussed above can be later refactored without
breaking too much API.
Thanks, Fabian
> Add a Kafka table source factory with JSON format support
> ---------------------------------------------------------
>
> Key: FLINK-8538
> URL: https://issues.apache.org/jira/browse/FLINK-8538
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
> Reporter: Timo Walther
> Assignee: Xingcan Cui
> Priority: Major
>
> Similar to CSVTableSourceFactory a Kafka table source factory for JSON should
> be added. This issue includes improving the existing JSON descriptor with
> validation that can be used for other connectors as well. It is up for
> discussion if we want to split the KafkaJsonTableSource into connector and
> format such that we can reuse the format for other table sources as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
