paul-rogers commented on pull request #2251: URL: https://github.com/apache/drill/pull/2251#issuecomment-881155695
Next, let's discuss plugins. Let's start with three goals: 1. System plugins work for everyone (we can add security later). 2. By default, plugins are available to everyone. That is, the Drill out-of-the-box plugins are available so that, say, `cp` works and people can try examples from the documentation. 3. There is an optional way to create per-user plugin (configs). What is the simplest way to do this? Maybe by changing the plugin resolution rules. When the planner looks for a plugin, it tries two names: `$$<user>$$<plugin>` and `<plugin>`. We use the resulting name in the query plan. So, if I have (unwisely) created my own `cp` plugin, when I ask for `cp`, that's what I get: `$$paul$$cp`. If I do not have my own `dfs`, then when I ask for it, I get the (global) `dfs`. (The `$$<user>$$` is something I just made up; it can be anything as long as the marker is not likely to ever appear in a real plugin name. Maybe `..<user>..` or `\<user>\` would be better since they are not legal SQL.) When a user stores a plugin, and the per-user config feature is enabled, then the UI appends the user prefix. The UI filters out all plugins owned by other users. When the admin works with plugins, there is no prefix. This allows the admin to see, and change, both `cp` and `$$paul$$cp`. If I leave the organization, the admin can remove all the `$$paul$$...` plugins. I suspect you'll find that, with this solution, *no* change is needed in the plugin registry. Plans will resolve to the correct reader configs. The reader configs in the query plan will just work (as they would in your design.) How does this help sharing? Simply ask the admin to change the name of the plugin from `$$paul$$whizbang` to just `whizbang`. Now, everyone can see it and my queries work without change. Super simple. We have only two security levels (everyone or single user), but that is **far** better than reuse by copy/paste. The design could be extended to the role-based one without breaking anything. The next step is store the per-user plugins in separate persistent stores. This is the "cleaner" design, but now you have to create the means for an admin to see, change, and remove the user-level plugins. You also have to extend the plugin registry, which is, as we noted, a complex beast. Any change requires *extensive* unit tests. By the way, when I revised the plugin registry a while back, I fixed dozens of bugs. Most were simply because I added tests and found all kinds of broken stuff. Concurrency was broken, upgrades were broken, expiring of old plugins was broken. and so on. I still have scars. So, I'm trying to save you from similar pain. Will something like this work and avoid the hard work we discussed earlier? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
