JingsongLi commented on PR #10850: URL: https://github.com/apache/paimon/pull/10850#issuecomment-6572177199
Could we also evaluate a dedicated metadata-only loading path for this use case, while preserving the existing validation behavior of the Core runtime APIs? The immediate requirement seems to be: - `SHOW CREATE TABLE` reads the persisted schema and options without constructing a runtime store. - `ALTER TABLE` loads the existing definition, applies all requested changes, validates the resulting schema, and commits it atomically. An unrelated ALTER that leaves the configuration invalid should still fail. - `DROP TABLE` uses persisted metadata for deletion and cleanup, preserving the existing branch and external-path cleanup semantics. #10830 already provides part of this separation at the Catalog layer. Could we build on that by exposing/reusing the existing schema/`TableMetadata` loading machinery through the Catalog abstraction, and adapting the Flink/Spark table-loading paths to use metadata-backed table handles? Actual data operations would then construct and validate a runtime table from the captured definition and the final effective options. Changing only the three command execution methods would not be sufficient: the engines can load the table through their common lookup paths before executing the command. Those lookup paths would also need to support metadata-only access, with runtime construction occurring when the table is used for data operations. This alternative would have a narrower behavioral scope: metadata management could recover invalid historical tables without changing the validation timing of Core `getTable()`/`copy()` for all callers, or moving some streaming configuration errors into asynchronous initialization. It may require more adapter code, whereas this PR provides one shared boundary in Core. I think it is worth comparing these two designs before making the broader API behavior change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
