After chewing a little more on this, I don't think it goes anywhere
interesting without classloader isolation. Probably just have to bite
the bullet and do that upfront.
On 2022/08/16 19:12, James Turton wrote:
A little bit more information about the proposed first stage that
moves a minimally modified contrib/ out to a new "drill-contrib"
repository based on my experiments.
* It is possible to build and test the plugins after doing this.
* Interactive debugging of plugin unit tests still works but requires
an "Attach source" step if you don't want to debug into decompiled
Drill code.
* Simultaneous changes to core Drill and a plugin would probably
require a bit more juggling than in the past and would lead to two
PRs, one to each repo. On the other hand this arrangement might make
it easier to keep plugin-specific code out of Drill core, promoting
better system design.
* We can use git subtree to take the commit history relevant to
plugins with.
* There is a user installation question that comes up - how, after
this first stage, should users install plugins now that the Drill
they downloaded was no longer the "kitchen sink"?
On 2022/08/16 16:01, James Turton wrote:
Hi all
This topic has come up on-and-off for a while now and circumstances
mean we should probably consider it again. To save myself from having
to structure an essay I'm going let my writing deteriorate to short
bullets now.
* In a recent Phoenix plugin PR we learned that Drill users want to
use plugins based on both the "thin" and "thick" Phoenix JDBC
drivers. A change to move to the thick driver was recently expedited
because of an Avatica classpath conflict in the thin driver which
tanks Drill if the classloader dice fall the wrong way. We had
started getting reports of that happening in Drill master. Now we're
left with the question of how we could retain the thin driver.
* We want to support building plugins (contrib/) away from the main
source tree, shipping a leaner base Drill and the ability to install
plugins from Maven coordinates created by third parties.
* We recently started publishing the Drill SNAPSHOT artifacts that
plugin authors targeting the coming Drill release depend on from
our CI.
* I propose a new drill-contrib repo to hold what is currently
contrib/ in the main repo. The drill-contrib repo would probably
have to be versioned and released in lockstep with the main repo,
making one question the point of it, but it would force us to eat
our own dogfood in that we'd be forced to build and ship plugins
away from the main source tree.
* Plugins becoming installed individually we could reintroduce the
thin Phoenix driver as, say, storage-pqs and not worry about
crashing the default Drill build.
* A second iteration could push modularity and isolation further. OSGi
looks complex and bigger than we need but JPMS module layers (see
the end of this article
<https://inside.java/2021/09/10/what-are-modules-about/>) look like
they might do the trick and offer classloader isolation. An upshot
would be that Drill 2.0 would then not run on the ageing Java 8 - I
guess we'd have to be prepared to accept that.
* Introducing JPMS would also allow us to limit how much of the core
Drill APIs get programmed against by plugin authors by using its
visibility features.
* If we got module layers working to provide classloader isolation
then the Phoenix thin driver and the Avatica shaded into it (the
thin driver is a little fat in this sense 🙂) could in principle
return to Drill unmodified. Otherwise we're looking at building and
distributing a new thin Phoenix driver just for Drill.
If you really let your imagination go, you can imagine all plugins
running as totally decoupled microservices with integration over a
some network protocol and a new API. I must admit that sounds slow
and painful to me. And it could also be achieved by a new
storage-external-microservice-adapter plugin for anyone who really
wants it.
What do you think?
James
