[
https://issues.apache.org/jira/browse/FLINK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209381#comment-17209381
]
Jark Wu edited comment on FLINK-19517 at 10/7/20, 8:17 AM:
-----------------------------------------------------------
Hi [~ksp0422], what's the difference between "Kafka" and "Confluent Kafka"? If
the only difference is the confluent avro format, that has been supported in
the 1.12 which allows you to define "avro-confluent" format and
"avro-confluent.schema-registry.url", you can see the documentation for more
details:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/formats/avro-confluent.html
Regarding to
1) "parallelism" configuration
This is under discussion, please watch the discuss thread in dev mailing list:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-146-Improve-new-TableSource-and-TableSink-interfaces-td45161.html
2) specify custom properties within WITH
This has been supported in "kafka" connector by the "properties.*" options.
3) have remote access to SQL client in cluster from local environment
I guess this is SQL gateway? Could you check whether this project
[flink-sql-gateway|https://github.com/ververica/flink-sql-gateway] is what you
are looking for?
was (Author: jark):
Hi [~ksp0422], what's the difference between "Kafka" and "Confluent Kafka"? If
the only difference is the confluent avro format, that has been supported in
the 1.12 which allows you to define "avro-confluent" format and
"avro-confluent.schema-registry.url", you can see the documentation for more
details:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/formats/avro-confluent.html
Regarding to
1) "parallelism" configuration
This is under discussion, please watch the discuss thread in dev mailing list:
2) specify custom properties within WITH
This has been supported in "kafka" connector by the "properties.*" options.
3) have remote access to SQL client in cluster from local environment
I guess this is SQL gateway? Could you check whether this project
[flink-sql-gateway|https://github.com/ververica/flink-sql-gateway] is what you
are looking for?
> Support for Confluent Kafka of Table Creation in Flink SQL Client
> -----------------------------------------------------------------
>
> Key: FLINK-19517
> URL: https://issues.apache.org/jira/browse/FLINK-19517
> Project: Flink
> Issue Type: Wish
> Affects Versions: 1.12.0
> Reporter: Kevin Kwon
> Priority: Critical
>
> Currently, table creation from SQL client such as below works well
> {code:sql}
> CREATE TABLE kafkaTable (
> user_id BIGINT,
> item_id BIGINT,
> category_id BIGINT,
> behavior STRING,
> ts TIMESTAMP(3)
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behavior',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'testGroup',
> 'format' = 'avro',
> 'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> Although I would wish for the table creation to support Confluent Kafka
> configuration as well. For example something like
> {code:sql}
> CREATE TABLE kafkaTable (
> user_id BIGINT,
> item_id BIGINT,
> category_id BIGINT,
> behavior STRING,
> ts TIMESTAMP(3)
> ) WITH (
> 'connector' = 'confluent-kafka',
> 'topic' = 'user_behavior',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'testGroup',
> 'schema-registry' = 'http://schema-registry.com',
> 'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> If this is enabled, it will be much more convenient to test queries
> on-the-fly that business analysts want to test against with 'Confluent Kafka'
> Additionally, it will be better if we can
> - specify 'parallelism' within WITH clause to support parallel partition
> processing
> - specify custom properties within WITH clause specified in
> [https://docs.confluent.io/5.4.2/installation/configuration/consumer-configs.html]
> - have remote access to SQL client in cluster from local environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
