[
https://issues.apache.org/jira/browse/MYRIAD-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santosh Marella reopened MYRIAD-7:
----------------------------------
> Run MyriadMesosScheduler inside YARN's resource manager JVM.
> ------------------------------------------------------------
>
> Key: MYRIAD-7
> URL: https://issues.apache.org/jira/browse/MYRIAD-7
> Project: Myriad
> Issue Type: Improvement
> Reporter: Santosh Marella
> Assignee: Santosh Marella
> Fix For: Myriad 0.1.0
>
>
> The objective of this change is to run MyriadMesosScheduler inside YARN's
> resource manager JVM.
> 1. Added Myriad{Fair,Capacity,Fifo}Scheduler classes that extend from Yarn's
> {Fair,Capacity,Fifo}Scheduler classes respectively.
> 2. Added build dependencies on Hadoop.
> 3. Encountered slf4j's "multiple binding" conflicts because drop wizard uses
> logback implementation while YARN uses slf4j-log4j12 implementation for
> slf4j API.
> After discussions with Mohit, we've decided to remove dependencies on drop
> wizard.
> 4. Refactored the rest of the code to that effect. Retained non-conflicting
> deps like
> codahale metrics, jackson's dataformat/databind/annotations/yaml and
> hibernate validator.
> 5. Added build rules to package the myriad jar and the dependencies to
> build/libs dir.
> These DO NOT INCLUDE hadoop jars and their dependencies as the intent is to
> deploy myriad jar and it's deps into an existing YARN installation.
> Build and Deployment guidelines:
> 1. From myriad dir in the local git repo, run "./gradlew jar".
> 2. This should produce myriad-0.0.1.jar and other deps under build/libs dir.
> 3. Copy build/libs/*.jar to HADOOP_HOME/share/hadoop/yarn.
> 4. Modify HADOOP_HOME/etc/hadoop/yarn-site.xml to have the following entry:
> {code:xml}
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
> </property>
> {code}
> 5. Restart Resource Manager process.
> CAVEATS:
> - The REST API to myriad is currently broken as we eliminated the
> dependencies on dropwizard.
> We need to bring up a web app for myriad and expose the
> REST API through that.
> - Mesos also requires a native library (libmesos.so). Our build process
> currently
> does not add that to myriad jar. We need to manually add that to YARN's
> native lib dir
> $HADOOP_HOME/lib/native.
> I'll open tasks for both the above and track them separately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
