Hi Devs, I would like to get some thoughts around the addition of a couple of extensions related to schema evolution inside Flink SQL. There are a couple of pain points that I have observed with Flink SQL jobs.
1. The first challenge is addition of `ALTER TABLE <table> ADD COLUMN` to a Flink job makes the job kind of non-idempotent. This means the job will run fine for the first time but later it'll fail until the ALTER TABLE command is removed. A broader idea is to support something like ALTER TABLE <table> ADD [COLUMN] IF NOT EXISTS pattern. This pattern is pretty common with many datastores like Postgres, Cassandra, Clickhouse etc. 2. The second challenge is supporting schema evolution with statements like `INSERT INTO dst_table SELECT * from src_table`. From interface perspectives, a user can enable automatic schema evolution by having a table configuration that can be set like `SET 'table-evolve-scheme' = 'true'`. Similar patterns are used by Snowflake and Databricks for doing schema evolution. Looking forward to hearing some feedback on the above from community. I do have some details carved out and willing to share as a FLIP soon. Thanks, Muhammad
