Paul Rogers created DRILL-4543:
----------------------------------
Summary: Advertise Drill-bit ports, status, capabilities in
ZooKeeper
Key: DRILL-4543
URL: https://issues.apache.org/jira/browse/DRILL-4543
Project: Apache Drill
Issue Type: Improvement
Components: Server
Reporter: Paul Rogers
Fix For: 2.0.0
Today Drill uses ZooKeeper (ZK) to advertise the existence of a Drill-bit,
providing just the host name/IP Address of the Drill-bit. All other information
(ports, status, capabilities) are assumed to be the same across all Drill-bits
in the cluster as specified in the Drill config file.
Moving forward, as Drill becomes more sophisticated, Drill should advertise the
specifics of each Drill-bit so that one Drill bit can differ from another.
For example, when running on YARN, we need a way for Drill to gracefully shut
down. Advertising a status of Ready or Unavailable will help. Ready is the
normal state. Unavailable means the Drill-bit will finish in-flight queries,
but won't accept new ones. (The actual status is a separate enhancement.)
In a YARN cluster, Drill should take advantage of machines with more memory,
but live with machines with less. (Perhaps some are newer, some are older or
more heavily loaded.) Drill should use ZK to identify its available memory and
CPUs so that the planner can use them. (Use of the info is a separate
enhancement.)
There may be times when two drill bits run on a single machine. If so, they
must use separate ports. So, each Drill-bit should advertise its ports in ZK.
For backward compatibility, the information is optional; if not present, the
receiver should assume the information defaults to that in the config file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
