Hi Tim,
Thanks for the reply.
My usecase is following:
- My main DB table is huge so it is sharded amongs multiple
storage-nodes.
- Each stroage-node is storing the assigned shard in a local relational
db engine.
I was planning to use Drill as a distributed query engine that can
scatter-gather data from these storage-nodes.
So, my overall plan for such architecture, as per my limited understanding
of Drill so far, is:
- Have a DrilBit instance run on each storage-node, and this fleet will
act as a leaf-worker fleet.
- (I will write a Storage Plugin to transform data from my local
relational DB engine to Drill record fromat)
- Maintain another fleet that will serve as Foreman and Intermeidate
query workers, still part of the same Drill cluster.
- The reason I intended to have the leaf-query fleet (storage-nodes)
segregated from Foreman/Intermediate workers (working on major fragments
is):
- storage-nodes (acting as leaf-workers) are premium commodity in
my cluster, involved in data ingestion as well as query traffic
servers as
leaf-worker.
- So, I do not intend to overload them further with intermediate
query fragment processing and aggregation that Foreman and Intermeidate
pool of workers are involved in.
Does the above make sense?
Thanks,
Lokendra
On Tue, Nov 13, 2018 at 4:17 PM Timothy Farkas <[email protected]> wrote:
> Hi Lokendra,
>
> All Drillbits can function as a foreman if a query is sent to them, and all
> drillbits are considered worker nodes. This ingrained deeply into the
> design of Drill and it was done with the intention of making Drill
> symmetric. Symmetric here means that each Drillbit is identical to all the
> others. Making this change would be a significant design change.
>
> Why are you interested in running Drill in this way? Do you have a specific
> use case in mind?
>
> Thanks,
> Tim
>
> On Tue, Nov 13, 2018 at 3:37 PM Lokendra Singh Panwar <
> [email protected]>
> wrote:
>
> > Hi,
> >
> > Is it possible to configure Drill such that the Foreman and leaf-worker
> > fleets are separate fleets of nodes?
> > Or if this needs changing the source of Drill, any pointers are
> appreciated
> > too.
> >
> > Thanks,
> > Lokendra
> >
>
