Hi Charles,
I'm not quite sure what "dynamic queue allocation" means: all YARN containers
are allocated dynamically through YARN via queues.
It may be helpful to review how Drill-on-YARN (DoY) works. DoY does NOT attempt
to use YARN for each query. Impala tried that with Llama and discovered that
the latency of YARN allocation is not compatible with the needs of a query
engine: YARN takes 10s of seconds to launch containers, queries must complete
in fractions of a second.
Instead, DoY treats YARN as a resource manger for long-running applications.
Think of it as an old-school Kubernetes. That is, you use YARN to launch Drill,
and to account for the cluster resources used by Drill. The DoY UI allows you
to grow/shrink the cluster, which turns around and asks YARN for more or fewer
containers.
(Jyothsna; we should integrate your graceful shutdown work into DoY for cluster
shrinking.)
To be very clear, Drill is long-running and clusters grow or shrink over long
periods of time (perhaps over a day: more Drill during the day, less at night.)
Queries are rapid-fire and run on the available Drillbits.
DoY is designed for a multi-tenant setup. The only trick is that each tenant
cluster must assign distinct ports and ZK roots. The details are spelled out in
the DoY docs. (K8s avoids the need for mucking with ports via an overlay
network, something that YARN does not provide.)
IIf the above leaves questions open, please do provide a bit more detail about
what you want to achieve.
Thanks,
- Paul
On Monday, December 17, 2018, 8:53:54 AM PST, Charles Givre
<[email protected]> wrote:
Hello all,
We are trying to set up a Drill cluster on our corporate data lake. Our
cluster requires dynamic YARN queue allocation for multi-tenant environment.
Is this something that Drill supports or is there a workaround?
Thanks!
—C
