Bowen Li created FLINK-15985:
--------------------------------
Summary: offload runtime params from DDL to table hints in
DML/queries
Key: FLINK-15985
URL: https://issues.apache.org/jira/browse/FLINK-15985
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Reporter: Bowen Li
Assignee: Danny Chen
Fix For: 1.11.0
background:
Currently Flink DDL mixes three types of params all together:
* External data’s metadata: defines what the data looks like (schema), where
it is (location/url), how it should be accessed (username/pwd)
* Source/sink runtime params: defines how and usually how fast Flink
source/sink reads/writes data, not affecting the results
* Kafka “sink-partitioner”
* Elastic “bulk-flush.interval/max-size/...”
* Semantics params: defines aspects like how much data Flink reads/writes, how
the result will look like
* Kafka “startup-mode”, “offset”
* Watermark, timestamp column
Problems of the current mix-up: Flink cannot leverage catalogs and external
system metadata alone to run queries with all the non-metadata params involved
in DDL. E.g. when we add a catalog for Confluent Schema Registry, the expected
user experience should be that Flink users just configure the catalog with url
and usr/pwd, and should be able to run queries immediately; however, that’s not
the case right now because users still have to use DDL to define a bunch params
like “startup-mode”, “offset”, timestamp column, etc, along with the schema
redundantly. We’ve heard many user complaints on this.
cc [~ykt836] [~lirui] [~lzljs3620320] [~jark] [~twalthr] [~dwysakowicz]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
