Hi @taokelu,
This proposal is nice!! We also disscuss another topic "Decoupling connectors from compute engines"[1], I have a question how to manager flink sql connector? [1] https://lists.apache.org/thread/j99crn7nkfpwovng6ycbxhw65sxg9xn2 Thanks, leo65535 At 2022-04-29 11:06:42, "陶克路" <[email protected]> wrote: The background https://github.com/apache/incubator-seatunnel/issues/1753 Let me have a brief introduction about the background. I found the Flink SQL support in Seatunnel is very simple, so I want to do some improvements on this story. And now seatunnel uses many deprecated datastream apis, which are encouraged to be replaced with SQL, such as `StreamTableEnvironment.connect`. Maybe SQL would be an alternative. Here are the improvement details: 1. refactor start-seatunnel-sql.sh. Now start-seatunnel-flink.sh and start-seatunnel-spark.sh have been refactored, and the main logic has been rewritten by java code. I think we can first keep them consistent. 2. enrich sql config file. Now flink sql job config is very simple, and it's all about the sql script. I think we can add more sub-config into it. 3. sql connectors management. Flink community supports a rich set of SQL connectors. Only with connectors, we can run our job successfully end-to-end 4. sql related logic. Such as validation before job running, throwing the error as soon as possible 5. Catalog support. With catalog, we can reuse tables/udfs defined in catalog. 6. kubernetes native mode support. Actually, this is a universal feature, not just about sql. In Flink, to run job in kubernetes native mode, we must bundle the main jar and dependency files into the Flink image. This is not user-friendly. Community support a workaround for this, namely podTemplate 7. ... This is a long-term plan. We can implement it step by step. What do you think about this PROPOSAL? Feel free to give any comment or suggestion. Thanks. Kelu. -- Hello, Find me here: www.legendtkl.com.
