[
https://issues.apache.org/jira/browse/DRILL-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207566#comment-15207566
]
Paul Rogers commented on DRILL-1170:
------------------------------------
A brief "starter set" of requirements:
* Configuration file to gather the cluster configuration (memory, cores, number
of nodes and so on.)
* Launcher to start/stop Drill within YARN
* Drill-specific Application Master (AM)
* AM requests YARN Node Manager (AM) to launch drill-bits.
* Use YARN localization feature to depoy Drill files to each node.
* Add nodes (drill-bits) to a running Drill cluster
* Remove nodes from a running Drill cluster (see DRILL-2656)
* Detect and restart failed drill-bits
* Status/statistics about the cluster as a whole (number of active nodes,
number of restarts, etc.)
* Allow existing users to run "unmanaged" Drill clusters (YARN is optional)
* Possibly allow multiple "Drill clusters" (independent clusters of drill bits)
on the same YARN-managed physical cluster.
> YARN support for Drill
> ----------------------
>
> Key: DRILL-1170
> URL: https://issues.apache.org/jira/browse/DRILL-1170
> Project: Apache Drill
> Issue Type: New Feature
> Reporter: Neeraja
> Assignee: Paul Rogers
> Fix For: Future
>
>
> This is a tracking item to make Drill work with YARN.
> Below are few requirements/needs to consider.
> - Drill should run as an YARN based application, side by side with other YARN
> enabled applications (on same nodes or different nodes). Both memory and CPU
> resources of Drill should be controlled in this mechanism.
> - As an YARN enabled application, Drill resource consumption should be
> adaptive to the load on the cluster. For ex: When there is no load on the
> Drill , Drill should consume no resources on the cluster. As the load on
> Drill increases, resources permitting, usage should grow proportionally.
> - Low latency is a key requirement for Apache Drill along with support for
> multiple users (concurrency in 100s-1000s). This should be supported when run
> as YARN application as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
