If you want to define the sql query via configuration, does that mean you
know the query when you launch the pipeline (as configuration is set at
launch time)? If so, you can also dynamically set the schema at pipeline
launch time.
On Sun, Jan 28, 2024 at 11:21 AM Sigalit Eliazov
wrote:
> Im
Im trying to define a generic pipeline that reads message from kafka where
the topic is configurable.
We read a generic record and schema id is part of thr consumed message . So
this part can be generalized
Then we would like to activate different queries on the stream.
I would like to be able to
Can you explain the use case a bit more? In order to write a SQL statement
(at least one that doesn't use wildcard selection) you also need to know
the schema ahead of time. What are you trying to accomplish with these
dynamic schemas?
Reuven
On Sun, Jan 28, 2024 at 2:30 AM Sigalit Eliazov
Hello, In the upcoming process, we extract Avro messages from Kafka
utilizing the Confluent Schema Registry.
Our intention is to implement SQL queries on the streaming data.
As far as I understand, since I am using the Flink runner, when
creating the features PCollection, I must specify the
