Hi All, I'm starting this thread to consolidate the discussion about maintaining the JDBC schema for new features.
As we have an influx of proposals that include JDBC persistence components (Idempotency, Open Lineage, Metrics, Events), I think it is time to review our approach to schema maintenance. I'd like to propose keeping separate DDL scripts for each Persistence SPI with a JDBC backend. That is, one script for the MetaStore (BasePersistence), one for Lineage, etc. The benefits are: * Users that opt in to certain features need to keep track only of the relevant tables. * While a feature is in beta/experimental/WIP state, edits to its section of the DDL do not affect other features. * Schema evolution can progress at different paces per feature. Different evolution paces are a reality we must face anyway. Separating DDL scripts should make maintenance easier, though. Granted, this will require some related and potentially intrusive changes in the Admin Tool and the bootstrap process. However, I believe they are well worth the benefits. I am willing to make a POC PR for an in-depth review if the community accepts this idea at the high level. Thoughts? Thanks, Dmitri.
