Of course! This was just my tongue-in-cheek way of saying *bump*
-- Jacques Nadeau CTO and Co-Founder, Dremio On Mon, Oct 12, 2015 at 5:37 PM, Julian Hyde <[email protected]> wrote: > Dead air means I’ve had ‘flu and been busy… can you give me another day to > think about this? > > > On Oct 12, 2015, at 4:12 PM, Jacques Nadeau <[email protected]> wrote: > > > > The dead air must mean that everyone is onboard with my recommendation > > > > PlannerIntegration StoragePlugin.getPlannerIntegrations() > > > > interface PlannerIntegration{ > > void initialize(Planner, Phase) > > } > > > > Right :D > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Fri, Oct 9, 2015 at 7:03 AM, Jacques Nadeau <[email protected]> > wrote: > > > >> A number of us were meeting last week to work through integrating the > >> Phoenix storage plugin. This plugin is interesting because it also uses > >> Calcite for planning. In some ways, this should make integration easy. > >> However, it also allowed us to see certain constraints who how we expose > >> planner integration between storage plugins and Drill internals. > >> Currently, Drill asks the plugin to provide a set of optimizer rules > which > >> it incorporates into one of the many stages of planning. This is too > >> constraining in two ways: > >> > >> 1. it doesn't allow a plugin to decide which phase of planning to > >> integrate with. (This was definitely a problem in the Phoenix case. Our > >> hack solution for now is to incorporate storage plugin rules in phases > >> instead of just one [1].) > >> 2. it doesn't allow arbitrary transformations. Calcite provides a > program > >> concept. It may be that a plugin needs to do some of its own work using > the > >> Hep planner. Currently there isn't an elegant way to do this in the > context > >> of the rule. > >> 3. There is no easy way to incorporate additional planner initialization > >> options. This was almost a problem in the case of the JDBC plugin. It > >> turned out that a hidden integration using register() here [2] allowed > us > >> to continue throughout the planning phases. However, we have to register > >> all the rules for all the phases of planning which is a bit unclean. > We're > >> hitting the same problem in the case of Phoenix where we need to > register > >> materialized views as part of planner initialization but the hack from > the > >> JDBC case won't really work. > >> > >> I suggest we update the interface to allow better support for these > types > >> of integrations. > >> > >> These seem to be the main requirements: > >> 1. Expose concrete planning phases to storage plugins > >> 2. Allow a storage plugin to provide additional planner initialization > >> behavior > >> 3. Allow a storage plugin to provide rules to include a particular > >> planning phase (merged with other rules during that phase). > >> 4. (possibly) allow a storage plugin to provide transformation programs > >> that are to be executed in between the concrete planning phases. > >> > >> Item (4) above is the most questionable to me as I wonder whether or not > >> this could simply be solved by creating a transformation rule (or > program > >> rule in Calcite's terminology) that creates an alternative tree and > thus be > >> solved by (3). > >> > >> A simple solution might be (if we ignore #4): > >> > >> PlannerIntegration StoragePlugin.getPlannerIntegrations() > >> > >> interface PlannerIntegration{ > >> void initialize(Planner, Phase) > >> } > >> > >> This way, a storage plugin could register rules (or materialized views) > at > >> setup time. > >> > >> What do others think? > >> > >> [1] > >> > https://github.com/apache/drill/blob/master/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java#L145 > >> [2] > >> > https://github.com/jacques-n/drill/commit/d463f9098ef63b9a2844206950334cb16fc00327#diff-e67ba82ec2fbb8bc15eed30ec6a5379cR119 > >> > >> -- > >> Jacques Nadeau > >> CTO and Co-Founder, Dremio > >> > >
