Hi all, I need help in understanding if we can add columns with defaults, let's say NULL to the existing table and recover the job from the savepoint.
We are using flink-1.16.0 table API and RocksDB as backend to provide a service to our users to run sql queries. The tables are created using the avro schema and when the schema is changed in a compatible manner i.e adding a field with default, we are unable to recover the job from the savepoint. This is the error we get after the schema is upgraded. Caused by: org.apache.flink.util.StateMigrationException: The new state serializer (org.apache.flink.table.runtime.typeutils.RowDataSerializer@aad5b03a) must not be incompatible with the old state serializer (org.apache.flink.table.runtime.typeutils.RowDataSerializer@9d089984). We tried to debug the issue and this error originates from org.apache.flink.table.runtime.typeutils.RowDataSerializer.RowDataSerializerSnapshot -> resolveSchemaCompatibility line 343:345 which checks the length of the type array and also the logicalType for each element or you can say columns. Is there a way to restore and evolve a table using table-api when the avro schema evolves in a compatible manner? If not, is there any plan to provide upgrades and evolutions with table apis? Cheers, Ashish Khatkar
