Hi all
This topic has come up on-and-off for a while now and circumstances mean
we should probably consider it again. To save myself from having to
structure an essay I'm going let my writing deteriorate to short bullets
now.
* In a recent Phoenix plugin PR we learned that Drill users want to
use plugins based on both the "thin" and "thick" Phoenix JDBC
drivers. A change to move to the thick driver was recently expedited
because of an Avatica classpath conflict in the thin driver which
tanks Drill if the classloader dice fall the wrong way. We had
started getting reports of that happening in Drill master. Now we're
left with the question of how we could retain the thin driver.
* We want to support building plugins (contrib/) away from the main
source tree, shipping a leaner base Drill and the ability to install
plugins from Maven coordinates created by third parties.
* We recently started publishing the Drill SNAPSHOT artifacts that
plugin authors targeting the coming Drill release depend on from our CI.
* I propose a new drill-contrib repo to hold what is currently
contrib/ in the main repo. The drill-contrib repo would probably
have to be versioned and released in lockstep with the main repo,
making one question the point of it, but it would force us to eat
our own dogfood in that we'd be forced to build and ship plugins
away from the main source tree.
* Plugins becoming installed individually we could reintroduce the
thin Phoenix driver as, say, storage-pqs and not worry about
crashing the default Drill build.
* A second iteration could push modularity and isolation further. OSGi
looks complex and bigger than we need but JPMS module layers (see
the end of this article
<https://inside.java/2021/09/10/what-are-modules-about/>) look like
they might do the trick and offer classloader isolation. An upshot
would be that Drill 2.0 would then not run on the ageing Java 8 - I
guess we'd have to be prepared to accept that.
* Introducing JPMS would also allow us to limit how much of the core
Drill APIs get programmed against by plugin authors by using its
visibility features.
* If we got module layers working to provide classloader isolation
then the Phoenix thin driver and the Avatica shaded into it (the
thin driver is a little fat in this sense 🙂) could in principle
return to Drill unmodified. Otherwise we're looking at building and
distributing a new thin Phoenix driver just for Drill.
If you really let your imagination go, you can imagine all plugins
running as totally decoupled microservices with integration over a some
network protocol and a new API. I must admit that sounds slow and
painful to me. And it could also be achieved by a new
storage-external-microservice-adapter plugin for anyone who really wants it.
What do you think?
James
