Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi guys! I've created the first draft of the design document [1] and attached it to the Jira ticket [2]. Please let's continue our discussion in the ticket or in comments in Google Docs. [1] https://docs.google.com/document/d/14thwgV2RY1AA9KgYztv_kLYSz4K1TckJ-YiGfkB5650/edit?usp=sharing [2] https

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Great! I linked that JIRA to FLINK-11275 , and put it along with JIRAs for HiveCatalog and GenericHiveMetastoreCatalog. I have some initial thoughts on the solution you described, but I'll wait till a more complete google design doc comes up, sin

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Sorry guys I've attached the wrong link for Jira ticket in the previous email. This is the correct link : https://issues.apache.org/jira/browse/FLINK-12256 On Thu, 18 Apr 2019 at 18:29, Artsem Semianenka wrote: > Thank you guys so much! > > You provided me a lot of helpful information. > I've cr

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Thank you guys so much! You provided me a lot of helpful information. I've created the Jira ticket[1] and added to it an initial description only with the main purpose of the new feature. More detailed implementation description will be added further. Hi Rong, to tell the truth, my first idea was

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi Artsem, having a catalog support for Confluent Schema Registry would be a great addition. Although the implementation of FLIP-30 is still ongoing, we merged the stable interfaces today [0]. This should unblock people from contributing new catalog implementations. So you could already start

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi, Thanks Artsem and Rong for bringing up the demand from user perspective. A Kafka/Confluent Schema Registry catalog would have a good use case in Flink. We actually mentioned the potential of Unified Catalog APIs for Kafka in our talk a couple weeks ago at Flink Forward SF [1], and glad to lear

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi Rong, Thanks for pointing out the missing FLIPs in the FLIP main page. I added all the missing FLIP (incl. FLIP-14, FLIP-22, FLIP-29, FLIP-30, FLIP-31) to the page. I also include @xuef...@alibaba-inc.com and @Bowen Li into the thread who are familiar with the latest catalog design. Thank

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Thanks Artsem for looking into this problem and Thanks Dawid for bringing up the discussion on FLIP-30. We've observe similar scenarios when we also would like to reuse the schema registry of both Kafka stream as well as the raw ingested kafka messages in datalake. FYI another more catalog-oriente

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Thank you, Dawid! This is very helpful information. I will keep a close eye on the updates of FLIP-30 and contribute whenever it possible. I guess I may create a Jira ticket for my proposal in which I describe the idea and attach intermediate pull request based on current API(just for initial discu

Re: [DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi Artsem, I think it totally makes sense to have a catalog for the Schema Registry. It is also good to hear you want to contribute that. There is few important things to consider though: 1. The Catalog interface is currently under rework. You make take a look at the corresponding FLIP-30[1], and

[DISCUSS] [FLINK SQL] External catalog for Confluent Kafka

Hi guys! I'm working on External Catalog for Confluent Kafka. The main idea to register the external catalog which provides the list of Kafka topics and execute SQL queries like : Select * form kafka.topic_name I'm going to receive the table schema from Confluent schema registry. The main disadva