rymurr opened a new pull request #3579: URL: https://github.com/apache/iceberg/pull/3579
## Motivation for this change The use cases are as follows: 1. add more iceberg specific procedures to the existing catalog. This is when we are actually operating on iceberg tables (not "to expose custom stored procedures for non-Iceberg systems") 2. modify existing OSS iceberg procedures. eg custom versions of compaction or a migrate table procedure that also re-writes the data 3. disable existing OSS iceberg procedures. eg maybe regular users shouldn't be running the snapshot or migrate action. Or ensuring that regular users with write permissions can't also execute potentially destructive actions (eg expire) ## Suggestion This is one of several options to have the ability to register new procedures. This simply adds an overlay `Map` which takes priority over the existing `static Map`. This has the advantage of being small and simple but the disadvantage of being a bit hacky and unintuitive. ## Other options Other potential options: 1) make the map in `SparkProcedures` mutable and register procedures at the `SparkProcedures` level - I am not a fan of this as it creates mutable static state 2) add a register method to `ProcedureCatalog` - this makes the `ProcedureCatalog` a bit more complete in that it would have 'read' and 'write' methods however its not immediately clear how to get stateless procedures (currently we use `ProcedureBuilder`). Perhaps `register(name, Supplier<Procedure>)`? 3) dynamically register procedures. There is precedence for doing this in Spark with other spark components but I am not sure how disabling or overloading procedures would work. This also has the advantage that it would get rid of the manual accounting in `SparkProcedures` I am not sure which option is best but I feel that having an unmodifiable set of procedures shipped with Iceberg is a bit challenging for administrators or operators. Looking forward to peoples thoughts @nastra @ajantha-bhat @RussellSpitzer @aokolnychyi @karuppayya @rdblue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
