[
https://issues.apache.org/jira/browse/DRILL-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216504#comment-15216504
]
Paul Rogers commented on DRILL-1170:
------------------------------------
We have considered Slider. Several factors nudged us in the direction of
writing an AM directly on YARN:
1. Slider has much documentation, but it is incomplete and out-of-date in
important places.
2. We could make up for the documenation by reading the source code. However,
Slider is composed of a large amount of Python code. Our team are mostly Java
developers. If we have to learn a bunch of code, we might as well learn YARN
directly.
3. Drill needs certain features that Slider does not (yet) provide, such as
monitoring ZooKeeper to track Drill-bit health, perhaps offering a connection
proxy, etc.
4. Slider is a general-purpose tool with many cool features. As it turns out,
many are not needed for Drill. This means that Slider introduces a bit of
unnecessary complexity for Drill admins.
5. Slider adds its own level of configuration files on top of those that we'd
need for Drill. Not a big issue, but it is just additional complexity for Drill
admins to learn and manage.
In balance, we like where Slider is going. Those Drill users who want to
roll-their-own YARN integration should certainly give Slider a try as a
short-term solution. This is particularly true for shops that already use
Slider for other apps.
On balance, however, Drill has a number of specialized needs that would seem to
justify the cost of a custom AM. We will, of course, continue to revisit the
issue as analysis proceeds.
> YARN support for Drill
> ----------------------
>
> Key: DRILL-1170
> URL: https://issues.apache.org/jira/browse/DRILL-1170
> Project: Apache Drill
> Issue Type: New Feature
> Reporter: Neeraja
> Assignee: Paul Rogers
> Fix For: Future
>
>
> This is a tracking item to make Drill work with YARN.
> Below are few requirements/needs to consider.
> - Drill should run as an YARN based application, side by side with other YARN
> enabled applications (on same nodes or different nodes). Both memory and CPU
> resources of Drill should be controlled in this mechanism.
> - As an YARN enabled application, Drill resource consumption should be
> adaptive to the load on the cluster. For ex: When there is no load on the
> Drill , Drill should consume no resources on the cluster. As the load on
> Drill increases, resources permitting, usage should grow proportionally.
> - Low latency is a key requirement for Apache Drill along with support for
> multiple users (concurrency in 100s-1000s). This should be supported when run
> as YARN application as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
