On Mon, Jun 6, 2016 at 12:02 PM, Wesley Chow <[email protected]> wrote:
> I have some general questions that I've been unable to google. I'm > particularly interested in co-locating drillbits with nodes in a custom > store of ours, so I've been poking around in source and searching about for > examples of this. > > 1. My understanding is that Drill understands HDFS and if you co-locate a > drillbit with a data node, then Drill will automatically distribute queries > to the drillbits on the nodes that contain the relevant files. > > 1a. Where does drill run a join then? On the node that initiated the query, > or on one of the nodes that contain the data? > The join is also distributed. At some point one side of the join may be broadcast to all the nodes or an exchange operation will distribute data appropriately across the nodes to get maximum utilization of the cluster. > > 1b. Does Drill automatically look up which nodes hold the data in question, > or is this specified in the query somehow? > > Drill will look it up as part of the query planning (in the case of Parquet file data, this information could be cached in a metadata cache file) > 2. Does drill also understand data distribution in HBase? Do queries get > sent to nodes that contain the HBase rows in question? > > 3. We have a custom data store that we'd like to be Drill aware, but want a > drillbit on the machine itself. Are there any examples of co-locating > drillbits with non-HDFS data sources? > > Don't have examples to share with you but there is no reason why you cannot colocate drill with your custom store. > 4. If we place files on a bunch of different servers and install drillbits > on each one, and we determine which servers contain which files > out-of-band, is there a way to submit a query to drill that tells it which > nodes contain local files to read? > > You won't be able to specify data locality information as part of the query; this is typically discovered by Drill by calling the storage plugin. You might need a custom storage plugin for your store. > Btw, I would be really interested in chatting /drinking with someone who > nows the Drill code well and is based in NYC. > > Thanks, > Wes >
