Documentation Comments
Today, I will be playing the role of the fool/jester trying to get Myriad
running. Basically, since getting Myriad running with Santosh quite a while
ago, and now trying again with new versions of Hadoop, MapR, and Myriad, I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and outline
points that as a non-dev living the code, are unclear to someone trying to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the code
or look at this page, but I will try to outline my thought processes as I
reviewed the current docs. Sometimes the way I approached the problem led
me down a path of to a certain page, missing the answer in a different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire to
improve how accessible Myriad is to the community, this is not a critique
of the hard work everyone has done on the project. I also understand that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so that
those folks who are SMEs on various points may be able to quickly add stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs where
and where that component gets its files. For example, I think it would
help people to understand that the whole tarball created in step 6 has all
the files for node managers and resource managers. Basically, everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down, this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager. (I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note, this is
different from when I originally set things up... before, I could run the
resource manager/myriad without a nodemanager, now it seems it's required
based on the config in the src... could we expound on this in the docs
somewhere?)
Option 2: (Are there other ways to launch the resource manager?)
Step 6: So something that is unclear to me is the handling of the
hadoop/yarn config files. In Step 6 on this page, there is sudo rm hadoop-
2.5.0/etc/hadoop/*.xml This doesn't makes sense to me. I actually ignored
this step. For me, if I remove these xml files, then there is no place to
get my files... I think? Since I am running the RM and NM from the same
tarball, and Myriad config is here, and my goal is to not have anything
installed on a node, where would I set yarn settings? This could be much
clearer to me, and probably others.
Step 2: Should we just be copying the Myriad files to
/share/hadoop/yarn/lib folder? Do we worry about potentially overwrites of
jars or version conflicts?
*Configuring Cgroups*
https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
At some point a little bit more about why one would want CGroups and issues
that could occur with them. While many folks using Mesos/Myriad may
understand this, others may not, and it's a good way to help people think
positively about our project if we help educate them along the way.
Minor point on enabling CGroups. This is confusing given my questions in
remote distribution. in this it says I need to edit my yarn-site.xml, but
in remote distribution it says delete my hadoop xml files. We need to
address this conflict cause it can be confusing for a user coming onboard
Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups for
mesos-slave
*Myriad Configuration Properties*:
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
Based on the conversation on list with Yuliya,
Currently, this file is built into Myriad Scheduler jar. So, if you need
to modify some of the properties in this file, modify them before building
Myriad Scheduler.
isn't accurate any more, and we should address that.
The configuration file in the wiki is an old one, the nmInstances isn't in
it, (and see my question about that above).
Frameworks and usernames. I think the users that the framework runs as,
the actual node and resource managers, etc is confusing to a user (I am
very confused!) When I first got Myriad up I set my user under the
Re: myriad scheduler startup with HDP2.7
This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/InputStream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2011) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2.3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0-2557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/hadoop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar -rw-r--r-- 1 root root 18336 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar -rw-r--r-- 1 root root 780664 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar -rw-r--r-- 1 root root 27084 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-xc-1.9.13.jar -bash-4.1$ jar tf /usr/hdp/current/hadoop-yarn-client/lib/jackson-dataformat-yaml-2.5.1.jar | grep YAMLFactory
Node manager issues now
So thanks everyone on the NMInstances bug. I am not getting a different issue with Myriad in that I have a permissions error with the remote tar ball distribution. In my old setup (Hadoop 2.5.0, MapR 4.1, some preincubator version of Myriad) I would run with the config having nodemanager: jvmMaxMemoryMB: 1024 # Xmx for NM JVM process. user: mapr # The user to run NM process as. cpus: 0.2 # CPU needed by NM process. cgroups: false# Whether NM should support CGroups. If set to 'true', myriad automatically # configures yarn-site.xml to attach YARN's cgroups under Me So user: mapr. Now, I realized that this no longer works in the verion I just cloned, the error message was clear to me that this was no longer an acceptable item. frameworkUser: mapr # Should be the same user running the resource manager. frameworkSuperUser: darkness # Must be root or have passwordless sudo on all nodes! So these are the settings I use, also, I run marathon with user: mapr (the resource manager). So I see three different places to set users. darkness is a Superuser with passwordless Sudo as requested. mapr is my cluster user, and mapr worked before, and I run the resource manager as that user in marathon. Myriad spins up fine, but then when it tries to kick off a nodemanager, I get the error below. Note, user 700 is the mapr user. Any thoughts on who I should run this as would be appreciated! STARTUP_MSG: build = g...@github.com:mapr/private-hadoop-common.git -r 5264b1d5c5c2a849ee0eb09cfcbbed19fb0bfb53; compiled by 'root' on 2015-07-02T23:46Z STARTUP_MSG: java = 1.8.0_45-internal / 15/08/19 07:01:09 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 15/08/19 07:01:10 WARN nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 24 ExitCodeException exitCode=24: File /tmp/mesos/slaves/20150818-152209-1677764800-5050-22280-S2/frameworks/20150818-152209-1677764800-5050-22280-/executors/myriad_executor20150818-152209-1677764800-5050-22280-S2/runs/1d06f1f1-02a9-4413-80de-7393e9e0935e must be owned by root, but is owned by 700 at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511) 15/08/19 07:01:10 INFO nodemanager.ContainerExecutor: 15/08/19 07:01:10 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:212) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511) Caused by: java.io.IOException: Linux container executor not configured properly (error=24) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:188) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210) ... 3 more Caused by: ExitCodeException exitCode=24: File /tmp/mesos/slaves/20150818-152209-1677764800-5050-22280-S2/frameworks/20150818-152209-1677764800-5050-22280-/executors/myriad_executor20150818-152209-1677764800-5050-22280-S2/runs/1d06f1f1-02a9-4413-80de-7393e9e0935e must be owned by root, but is owned by 700 at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182) ... 4 more 15/08/19 07:01:10 WARN service.AbstractService: When stopping the service NodeManager : java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:162) at
Re: myriad scheduler startup with HDP2.7
Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha d oop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar
Re: myriad scheduler startup with HDP2.7
Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha d oop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22
Re: Documentation Comments
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page. Just
sign up for a wiki account, send me your accountId, and I'll grant you edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com wrote:
Today, I will be playing the role of the fool/jester trying to get Myriad
running. Basically, since getting Myriad running with Santosh quite a while
ago, and now trying again with new versions of Hadoop, MapR, and Myriad, I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
outline
points that as a non-dev living the code, are unclear to someone trying to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the code
or look at this page, but I will try to outline my thought processes as I
reviewed the current docs. Sometimes the way I approached the problem led
me down a path of to a certain page, missing the answer in a different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire to
improve how accessible Myriad is to the community, this is not a critique
of the hard work everyone has done on the project. I also understand that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so that
those folks who are SMEs on various points may be able to quickly add stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs where
and where that component gets its files. For example, I think it would
help people to understand that the whole tarball created in step 6 has all
the files for node managers and resource managers. Basically, everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down, this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager. (I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note, this is
different from when I originally set things up... before, I could run the
resource manager/myriad without a nodemanager, now it seems it's required
based on the config in the src... could we expound on this in the docs
somewhere?)
Option 2: (Are there other ways to launch the resource manager?)
Step 6: So something that is unclear to me is the handling of the
hadoop/yarn config files. In Step 6 on this page, there is sudo rm
hadoop-
2.5.0/etc/hadoop/*.xml This doesn't makes sense to me. I actually ignored
this step. For me, if I remove these xml files, then there is no place to
get my files... I think? Since I am running the RM and NM from the same
tarball, and Myriad config is here, and my goal is to not have anything
installed on a node, where would I set yarn settings? This could be much
clearer to me, and probably others.
Step 2: Should we just be copying the Myriad files to
/share/hadoop/yarn/lib folder? Do we worry about potentially overwrites of
jars or version conflicts?
*Configuring Cgroups*
https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
At some point a little bit more about why one would want CGroups and issues
that could occur with them. While many folks using Mesos/Myriad may
understand this, others may not, and it's a good way to help people think
positively about our project if we help educate them along the way.
Minor point on enabling CGroups. This is confusing given my questions in
remote distribution. in this it says I need to edit my yarn-site.xml, but
in remote distribution it says delete my hadoop xml files. We need to
address this conflict cause it can be confusing for a user coming onboard
Nitpick: Enabling cgroups for mess-slave - should be - Enabling cgroups for
mesos-slave
*Myriad Configuration Properties*:
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Configuration+Properties
Based on the conversation on list
Re: myriad scheduler startup with HDP2.7
Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso ur c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j ar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us r/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp /2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2. 3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h dp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/ us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-m ap r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/ us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0. 0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ ha d
Re: Documentation Comments
hi John,
Thank you for your feedback. I'm the assigned technical writer for the
Myriad project. I'll also be working on updating the information.
I'm also ok with what Adam indicated: directly editing or adding a John's
page. But please be aware that I'll also be working on the content. Last
week I did some cleanup work on the original files in GitHub and then
brought them into the Wiki, although, I still have some more work in terms
of organizing and identifying holes.
If you like, I can create a John's comments page and then work on
incorporating some of the obvious things that you mentioned.
Thanks, Ruth
Ruth Harris
Sr. Tech. Writer
rhar...@mapr.com
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik j...@omernik.com wrote:
Thanks Adam, I signed up with mandoskippy.
I am honored to help in this capacity, for updating etc, do we go through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page and
then have someone review the work? If I have a comment on the page, is that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon a...@mesosphere.io wrote:
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page. Just
sign up for a wiki account, send me your accountId, and I'll grant you
edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com wrote:
Today, I will be playing the role of the fool/jester trying to get
Myriad
running. Basically, since getting Myriad running with Santosh quite a
while
ago, and now trying again with new versions of Hadoop, MapR, and
Myriad,
I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
outline
points that as a non-dev living the code, are unclear to someone trying
to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the
code
or look at this page, but I will try to outline my thought processes
as I
reviewed the current docs. Sometimes the way I approached the problem
led
me down a path of to a certain page, missing the answer in a different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire to
improve how accessible Myriad is to the community, this is not a
critique
of the hard work everyone has done on the project. I also understand
that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so
that
those folks who are SMEs on various points may be able to quickly add
stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs
where
and where that component gets its files. For example, I think it would
help people to understand that the whole tarball created in step 6 has
all
the files for node managers and resource managers. Basically,
everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down, this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager.
(I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note, this
is
different from when I originally set things up... before, I could run
the
resource manager/myriad without a nodemanager, now it seems it's
required
based on the config in the src... could we expound on this in the docs
somewhere?)
Option 2: (Are there other ways to launch the resource manager?)
Step 6: So something that is unclear to me is the handling of the
hadoop/yarn config files. In Step 6 on this page, there is sudo rm
hadoop-
2.5.0/etc/hadoop/*.xml This
Re: Documentation Comments
Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
skills and suggestions on how to help. Happy to help in the way that you'd
find easiest.
John
On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris rhar...@maprtech.com wrote:
hi John,
Thank you for your feedback. I'm the assigned technical writer for the
Myriad project. I'll also be working on updating the information.
I'm also ok with what Adam indicated: directly editing or adding a John's
page. But please be aware that I'll also be working on the content. Last
week I did some cleanup work on the original files in GitHub and then
brought them into the Wiki, although, I still have some more work in terms
of organizing and identifying holes.
If you like, I can create a John's comments page and then work on
incorporating some of the obvious things that you mentioned.
Thanks, Ruth
Ruth Harris
Sr. Tech. Writer
rhar...@mapr.com
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik j...@omernik.com wrote:
Thanks Adam, I signed up with mandoskippy.
I am honored to help in this capacity, for updating etc, do we go through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page and
then have someone review the work? If I have a comment on the page, is
that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon a...@mesosphere.io
wrote:
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page.
Just
sign up for a wiki account, send me your accountId, and I'll grant you
edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com
wrote:
Today, I will be playing the role of the fool/jester trying to get
Myriad
running. Basically, since getting Myriad running with Santosh quite a
while
ago, and now trying again with new versions of Hadoop, MapR, and
Myriad,
I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
outline
points that as a non-dev living the code, are unclear to someone
trying
to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the
code
or look at this page, but I will try to outline my thought processes
as I
reviewed the current docs. Sometimes the way I approached the
problem
led
me down a path of to a certain page, missing the answer in a
different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire
to
improve how accessible Myriad is to the community, this is not a
critique
of the hard work everyone has done on the project. I also understand
that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so
that
those folks who are SMEs on various points may be able to quickly add
stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs
where
and where that component gets its files. For example, I think it
would
help people to understand that the whole tarball created in step 6
has
all
the files for node managers and resource managers. Basically,
everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down,
this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager.
(I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note,
this
is
different from when I originally set things up... before, I could run
the
resource manager/myriad without a nodemanager,
Re: Documentation Comments
Thanks Adam, I signed up with mandoskippy.
I am honored to help in this capacity, for updating etc, do we go through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page and
then have someone review the work? If I have a comment on the page, is that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon a...@mesosphere.io wrote:
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page. Just
sign up for a wiki account, send me your accountId, and I'll grant you edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com wrote:
Today, I will be playing the role of the fool/jester trying to get Myriad
running. Basically, since getting Myriad running with Santosh quite a
while
ago, and now trying again with new versions of Hadoop, MapR, and Myriad,
I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
outline
points that as a non-dev living the code, are unclear to someone trying
to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the code
or look at this page, but I will try to outline my thought processes as I
reviewed the current docs. Sometimes the way I approached the problem
led
me down a path of to a certain page, missing the answer in a different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire to
improve how accessible Myriad is to the community, this is not a critique
of the hard work everyone has done on the project. I also understand
that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so that
those folks who are SMEs on various points may be able to quickly add
stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs
where
and where that component gets its files. For example, I think it would
help people to understand that the whole tarball created in step 6 has
all
the files for node managers and resource managers. Basically, everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down, this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager. (I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note, this
is
different from when I originally set things up... before, I could run the
resource manager/myriad without a nodemanager, now it seems it's required
based on the config in the src... could we expound on this in the docs
somewhere?)
Option 2: (Are there other ways to launch the resource manager?)
Step 6: So something that is unclear to me is the handling of the
hadoop/yarn config files. In Step 6 on this page, there is sudo rm
hadoop-
2.5.0/etc/hadoop/*.xml This doesn't makes sense to me. I actually
ignored
this step. For me, if I remove these xml files, then there is no place
to
get my files... I think? Since I am running the RM and NM from the same
tarball, and Myriad config is here, and my goal is to not have anything
installed on a node, where would I set yarn settings? This could be much
clearer to me, and probably others.
Step 2: Should we just be copying the Myriad files to
/share/hadoop/yarn/lib folder? Do we worry about potentially overwrites
of
jars or version conflicts?
*Configuring Cgroups*
https://cwiki.apache.org/confluence/display/MYRIAD/Configuring+Cgroups
At some point a little bit more about why one would want CGroups and
issues
that could occur with them. While many folks using Mesos/Myriad may
understand this, others may not, and it's a good way
Re: myriad scheduler startup with HDP2.7
as you can imagine you need matching versions of jackson* jars otherwise you might get into issues of incompatibility Easiest for you at the moment to put myriad dependency jars on classpath before others From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Cc: yuliya Feldman yufeld...@yahoo.com Sent: Wednesday, August 19, 2015 10:48 AM Subject: Re: myriad scheduler startup with HDP2.7 Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso ur c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j ar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us r/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp /2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2. 3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h dp /
Re: myriad scheduler startup with HDP2.7
Fundamentally, I would imagine that the goals of Myriad, even using a distribution's build of hadoop, is to have the classpath be entirely contained. I.e. There should be no need for any classpath on a node to run resource manager or node manager. This may post challenges, in that I know in MapR some libs are linked to the /opt/mapr/lib folder. Perhaps when we talk about building the tarball in the remote distribution, we should explore this idea, and perhaps use flags that include the files if they are links. John On Wed, Aug 19, 2015 at 1:53 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: as you can imagine you need matching versions of jackson* jars otherwise you might get into issues of incompatibility Easiest for you at the moment to put myriad dependency jars on classpath before others From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Cc: yuliya Feldman yufeld...@yahoo.com Sent: Wednesday, August 19, 2015 10:48 AM Subject: Re: myriad scheduler startup with HDP2.7 Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at
Re: Documentation Comments
john,
the details of where to send it on in the pdf that must be filled out and
signed. the fastest way is to send the ICLA to secret...@apache.org
mailto:secret...@apache.org
ken
On Aug 19, 2015, at 1:56 PM, John Omernik j...@omernik.com wrote:
Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
skills and suggestions on how to help. Happy to help in the way that you'd
find easiest.
John
On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris rhar...@maprtech.com wrote:
hi John,
Thank you for your feedback. I'm the assigned technical writer for the
Myriad project. I'll also be working on updating the information.
I'm also ok with what Adam indicated: directly editing or adding a John's
page. But please be aware that I'll also be working on the content. Last
week I did some cleanup work on the original files in GitHub and then
brought them into the Wiki, although, I still have some more work in terms
of organizing and identifying holes.
If you like, I can create a John's comments page and then work on
incorporating some of the obvious things that you mentioned.
Thanks, Ruth
Ruth Harris
Sr. Tech. Writer
rhar...@mapr.com
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik j...@omernik.com wrote:
Thanks Adam, I signed up with mandoskippy.
I am honored to help in this capacity, for updating etc, do we go through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page and
then have someone review the work? If I have a comment on the page, is
that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon a...@mesosphere.io
wrote:
John, thanks a ton for your valuable feedback! We're glad to have your
perspective as a user of the project, and I'm ready+willing to give you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page.
Just
sign up for a wiki account, send me your accountId, and I'll grant you
edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com
wrote:
Today, I will be playing the role of the fool/jester trying to get
Myriad
running. Basically, since getting Myriad running with Santosh quite a
while
ago, and now trying again with new versions of Hadoop, MapR, and
Myriad,
I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home) and
outline
points that as a non-dev living the code, are unclear to someone
trying
to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the
code
or look at this page, but I will try to outline my thought processes
as I
reviewed the current docs. Sometimes the way I approached the
problem
led
me down a path of to a certain page, missing the answer in a
different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a desire
to
improve how accessible Myriad is to the community, this is not a
critique
of the hard work everyone has done on the project. I also understand
that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here, so
that
those folks who are SMEs on various points may be able to quickly add
stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what runs
where
and where that component gets its files. For example, I think it
would
help people to understand that the whole tarball created in step 6
has
all
the files for node managers and resource managers. Basically,
everything
runs from there. Here is a small example I am currently working with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down,
this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to run node managers by
pulling the tarball to the slave nodes and executing the nodemanager.
(I
would imagine the work with history server etc would also use this
tarball?).
From here it will us NMInstances to launch a node manager. (Note,
this
is
different from when I originally set things up... before, I could run
the
Re: Documentation Comments
hi John,
I just got some content onto the wiki that wasn't there before. Right now,
I'm working on organizing the content better... your comments will help a
LOT.
Thanks, Ruth
On Wed, Aug 19, 2015 at 11:56 AM, John Omernik j...@omernik.com wrote:
Happy to sign the ICLA. Who do I send it to? Ruth, I defer to your writing
skills and suggestions on how to help. Happy to help in the way that you'd
find easiest.
John
On Wed, Aug 19, 2015 at 1:48 PM, Ruth Harris rhar...@maprtech.com wrote:
hi John,
Thank you for your feedback. I'm the assigned technical writer for the
Myriad project. I'll also be working on updating the information.
I'm also ok with what Adam indicated: directly editing or adding a John's
page. But please be aware that I'll also be working on the content. Last
week I did some cleanup work on the original files in GitHub and then
brought them into the Wiki, although, I still have some more work in
terms
of organizing and identifying holes.
If you like, I can create a John's comments page and then work on
incorporating some of the obvious things that you mentioned.
Thanks, Ruth
Ruth Harris
Sr. Tech. Writer
rhar...@mapr.com
On Wed, Aug 19, 2015 at 11:37 AM, John Omernik j...@omernik.com wrote:
Thanks Adam, I signed up with mandoskippy.
I am honored to help in this capacity, for updating etc, do we go
through
some kind of review? Is it better to ask questions on the dev list then
update when consensus occurs? How about when I'd like to post a page
and
then have someone review the work? If I have a comment on the page, is
that
public or can I just send to author? Just curious on any guidelines I
should be following in that regard.
John
On Wed, Aug 19, 2015 at 1:29 PM, Adam Bordelon a...@mesosphere.io
wrote:
John, thanks a ton for your valuable feedback! We're glad to have
your
perspective as a user of the project, and I'm ready+willing to give
you
edit access to the wiki if you want to update it with your learnings,
elaborate anything that's unclear, or add a new John's tips page.
Just
sign up for a wiki account, send me your accountId, and I'll grant
you
edit
access.
(I'll let others answer your specific questions)
On Wed, Aug 19, 2015 at 6:28 AM, John Omernik j...@omernik.com
wrote:
Today, I will be playing the role of the fool/jester trying to get
Myriad
running. Basically, since getting Myriad running with Santosh
quite a
while
ago, and now trying again with new versions of Hadoop, MapR, and
Myriad,
I
wanted to hit up the wiki (
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home)
and
outline
points that as a non-dev living the code, are unclear to someone
trying
to
utilize myriad or understand it's operation.
Obviously, some of my points can be answered with look here in the
code
or look at this page, but I will try to outline my thought
processes
as I
reviewed the current docs. Sometimes the way I approached the
problem
led
me down a path of to a certain page, missing the answer in a
different
page, and thus some cross linking could be helpful.
Please do not let my points be taken as anything other than a
desire
to
improve how accessible Myriad is to the community, this is not a
critique
of the hard work everyone has done on the project. I also
understand
that
given the work load and other issues, that fixing these issues in
documentation may not be a priority. I am listing them out here,
so
that
those folks who are SMEs on various points may be able to quickly
add
stuff
and we'll organize it later.
*Remote Distribution: *
https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Remote+Distribution
This whole section could use some work from a standpoint of what
runs
where
and where that component gets its files. For example, I think it
would
help people to understand that the whole tarball created in step 6
has
all
the files for node managers and resource managers. Basically,
everything
runs from there. Here is a small example I am currently working
with:
Starting Myriad:
Option 1: Use Marathon (provide example json, here is mine)
{
cmd: env export
YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=myriad.marathon.mesos
hadoop-2.7.0/bin/yarn resourcemanager,
uris: [maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz],
cpus: 1.0,
mem: 1024,
id: myriad,
instances: 1,
user: mapr
}
In this case, Marathon grabs the hadoop tarball and pulls it down,
this
tarball also has the Myriad yml file. When it executes the resource
manager, it is brought up in Myriad and ready to
[jira] [Commented] (MYRIAD-121) Myriad-Issue-60 Added ability to launch JHS ...
[ https://issues.apache.org/jira/browse/MYRIAD-121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703996#comment-14703996 ] Yuliya Feldman commented on MYRIAD-121: --- All the comments are going on https://github.com/mesos/myriad/pull/121 Myriad-Issue-60 Added ability to launch JHS ... --- Key: MYRIAD-121 URL: https://issues.apache.org/jira/browse/MYRIAD-121 Project: Myriad Issue Type: Bug Reporter: Yuliya Feldman ... and other services that fit under Myriad umbrella from MyriadScheduler using default Mesos slave executor. Will update following doc https://docs.google.com/document/d/1TGGU2bn1cVPSmHeriNrtkAEhdvsRKcFtuWRaaGp9bSY/edit?usp=sharing with more details -- This message was sent by Atlassian JIRA (v6.3.4#6332)
making contributor
Hello guys, Could you make me contributor to Myriad, so I could assign JIRAs to myself? Thanks,Yuliya
Complete Myriad HA implementation
Hi All, I have updated my pull request with the complete Myriad HA implementation rebased on top of the FGS changes here https://github.com/mesos/myriad/pull/123 I am planning to send out another email with details on how to configure it. Regards Swapnil
