Re: myriad scheduler startup with HDP2.7
This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/InputStream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2011) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2.3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0-2557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/hadoop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar -rw-r--r-- 1 root root 18336 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar -rw-r--r-- 1 root root 780664 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar -rw-r--r-- 1 root root 27084 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-xc-1.9.13.jar -bash-4.1$ jar tf /usr/hdp/current/hadoop-yarn-client/lib/jackson-dataformat-yaml-2.5.1.jar | grep YAMLFactory com/fasterxml/jackson/dataformat/yaml
Re: myriad scheduler startup with HDP2.7
Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha d oop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar
Re: myriad scheduler startup with HDP2.7
Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha d oop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp
Re: myriad scheduler startup with HDP2.7
Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso ur c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j ar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us r/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp /2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2. 3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h dp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/ us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-m ap r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/ us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0. 0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ ha d oop-yarn
Re: myriad scheduler startup with HDP2.7
as you can imagine you need matching versions of jackson* jars otherwise you might get into issues of incompatibility Easiest for you at the moment to put myriad dependency jars on classpath before others From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Cc: yuliya Feldman yufeld...@yahoo.com Sent: Wednesday, August 19, 2015 10:48 AM Subject: Re: myriad scheduler startup with HDP2.7 Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso ur c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j ar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us r/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp /2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2. 3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h dp / 2.3.0.0-2557/hadoop-yarn/lib
Re: myriad scheduler startup with HDP2.7
Fundamentally, I would imagine that the goals of Myriad, even using a distribution's build of hadoop, is to have the classpath be entirely contained. I.e. There should be no need for any classpath on a node to run resource manager or node manager. This may post challenges, in that I know in MapR some libs are linked to the /opt/mapr/lib folder. Perhaps when we talk about building the tarball in the remote distribution, we should explore this idea, and perhaps use flags that include the files if they are links. John On Wed, Aug 19, 2015 at 1:53 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: as you can imagine you need matching versions of jackson* jars otherwise you might get into issues of incompatibility Easiest for you at the moment to put myriad dependency jars on classpath before others From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Cc: yuliya Feldman yufeld...@yahoo.com Sent: Wednesday, August 19, 2015 10:48 AM Subject: Re: myriad scheduler startup with HDP2.7 Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163
