[
https://issues.apache.org/jira/browse/DRILL-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216514#comment-15216514
]
Paul Rogers commented on DRILL-4543:
------------------------------------
Thanks, Jacques. I see my confusion. When using the ZK client, I see the host
name, but ports appear as "noise". They are encoded as a Protobuf block and are
thus visible only to Drill code (or other code that knows how to decode the
Protobuf format.)
So, let me rephrase the port suggestion: store ports in an easily readable
format such as plain text:
drill://host-name:123:456:789
Or, if ZK allows it (can auto remove a subtree as well as a single node), as
children of the znode:
<drill bit id xyz>
- host-name: my-host
- user-port: 123
- data-port: 456
- ...
- memory-mb: 128
- cores: 5
> Advertise Drill-bit ports, status, capabilities in ZooKeeper
> ------------------------------------------------------------
>
> Key: DRILL-4543
> URL: https://issues.apache.org/jira/browse/DRILL-4543
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Reporter: Paul Rogers
> Fix For: 2.0.0
>
>
> Today Drill uses ZooKeeper (ZK) to advertise the existence of a Drill-bit,
> providing just the host name/IP Address of the Drill-bit. All other
> information (ports, status, capabilities) are assumed to be the same across
> all Drill-bits in the cluster as specified in the Drill config file.
> Moving forward, as Drill becomes more sophisticated, Drill should advertise
> the specifics of each Drill-bit so that one Drill bit can differ from another.
> For example, when running on YARN, we need a way for Drill to gracefully shut
> down. Advertising a status of Ready or Unavailable will help. Ready is the
> normal state. Unavailable means the Drill-bit will finish in-flight queries,
> but won't accept new ones. (The actual status is a separate enhancement.)
> In a YARN cluster, Drill should take advantage of machines with more memory,
> but live with machines with less. (Perhaps some are newer, some are older or
> more heavily loaded.) Drill should use ZK to identify its available memory
> and CPUs so that the planner can use them. (Use of the info is a separate
> enhancement.)
> There may be times when two drill bits run on a single machine. If so, they
> must use separate ports. So, each Drill-bit should advertise its ports in ZK.
> For backward compatibility, the information is optional; if not present, the
> receiver should assume the information defaults to that in the config file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
