Thanks for working on this great design document Jark. I think having
well-defined terminilogy and semantics around tables, changelogs, table
sources/sinks, and DDL should have been done much earlier. I will take a
closer look at the concepts and give feedback soon. I think having those
concepts defined and implemented should be the goal for Flink 1.10. It
also allows us to align it to the efforts of FLIP-27.
Introducing a DDL is a step that cannot be evolved easily as a DDL is
basically just a string that is being parsed. We should aim to involve
as many people as possible to have a future-proof design.
Thanks,
Timo
Am 27.05.19 um 10:40 schrieb Kurt Young:
Thanks Jark for bringing this topic. I think proper concepts is very
important for users who are using Table API & SQL. Especially for
them to have a clear understanding about the behavior of the SQL job. Also
this is essential for connector developers to have a better
understanding why we abstracted the interfaces in this way, and have a
smooth experience when developing connectors for Table & SQL.
Best,
Kurt
On Mon, May 27, 2019 at 3:35 PM Jark Wu <imj...@gmail.com> wrote:
Hi all,
We have prepared a design doc [1] about source and sink concepts in Flink
SQL. This is actually an extended discussion about SQL DDL [2].
In the design doc, we want to figure out some concept problems. For
examples:
1. How to define boundedness in DDL
2. How to define a changelog in DDL, what's the behavior of a changelog
source and changelog sink?
3. How to define primary key in DDL and what's the semantic when we have a
primary key on a table and stream?
They are mostly related to DDL because DDL is plain text and we need to
keep close to standard as much as possible.
This is an important step before we starting to refactor our
TableSource/TableSink/TableFactory interfaces. Because we need to know what
changes we need to introduce to support these concepts.
Please feel free to leave feedbacks in the thread or the design doc.
Regards,
Jark
[1].
https://docs.google.com/document/d/1yrKXEIRATfxHJJ0K3t6wUgXAtZq8D-XgvEnvl2uUcr0/edit#
[2].
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-SQL-DDL-Design-tt25006.html
