Could you provide your yarn-site.xml and myriad-config-default.yml ?
Let's make sure you have following in your yarn-site.xml:
<property> <name>yarn.resourcemanager.scheduler.class</name>
<value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
<description>One can configure other scehdulers as well from following list:
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler,
com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property>
<property> <name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${nodemanager.resource.cpu-vcores}</value> </property> <property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${nodemanager.resource.memory-mb}</value> </property> <property>
<name>yarn.nodemanager.address</name>
<value>${myriad.yarn.nodemanager.address}</value> </property> <property>
<name>yarn.nodemanager.webapp.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value> </property>
<property> <name>yarn.nodemanager.webapp.https.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value> </property>
<property> <name>yarn.nodemanager.localizer.address</name>
<value>${myriad.yarn.nodemanager.localizer.address}</value> </property>
<property> <name>mapreduce.shuffle.port</name>
<value>${myriad.mapreduce.shuffle.port}</value> </property> <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,myriad_executor</value> </property> <property>
<name>yarn.nodemanager.aux-services.myriad_executor.class</name>
<value>com.ebay.myriad.executor.MyriadExecutorAuxService</value> </property>
And in your yml file (should be at least linked to */etc/hadoop place)You have
following line defined correctly:
mesosMaster: zk://<zk_address>/mesos
Thanks,Yuliya
From: Björn Hagemeier <[email protected]>
To: [email protected]
Sent: Tuesday, February 23, 2016 6:21 AM
Subject: Re: Not able to launch Node Managers via Myriad
Hi all,
Am 23.02.2016 um 12:55 schrieb Björn Hagemeier:
> Hi Santosh,
>
> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
>> Hey guys.. the flow of things is as below:
>>
>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
>> UI should show this)
does Myriad register itself as a Mesos framework automatically, just by
starting the RM as one would do it without Mesos? Or do I need to start
the RM as a Mesos framework to make the magic happen, i.e. by "mesos
execute"?
In the latter case, I think I found out why my Myriad does not seem to
do anything. I did not start the resource manager as a Mesos framework.
I somehow reckoned, the myriad integration would take care of that by
itself. When I do this, I do see the framework, but with classpath
issues (missing myriad for some reason).
I'll keep digging in that direction.
Cheers,
Björn
> I think this is where it is already failing for me. I do not see the
> Myriad framework in the UI.
>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>> process is up.
>> This is queued up until Myriad receives an offer from Mesos that is big
>> enough to launch a medium profle NM.
> Yes, I do see the medium profile NM in the Myriad UI.
>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad
>> attempted to launch a NM).
> Nope, unfortunately it does not.
>
> What makes me wonder is that I do have a running Myriad, i.e. I can
> access the Myriad UI on port 8192, but it does not show up in Mesos.
>
> I guess my other problem, not seeing any offers, is a follow-on of this.
>
>>
>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>> starting point is to check what are the sizes of the offers Myriad is
>> receiving (Mesos' master log should show this). You can also enable DEBUG
>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>> log4j.properties.
> I will try and increase the log level to DEBUG.
>
>
> Cheers,
> Björn
>>
>>
>>
>> Santosh
>>
>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier <[email protected]
>>> wrote:
>>
>>> Hi Mayank,
>>>
>>> it seems you and I are having similar issues. Let's see whether they can
>>> be traced to the same causes.
>>>
>>>
>>> Best regards,
>>> Björn
>>>
>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
>>>> Hi,
>>>>
>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x
>>>>
>>>>
>>>> I have followed all the instructions in the wiki
>>>>
>>>> So the resource manager is up and i am able to launch the web UI of
>>> myriad
>>>> as well however whenever I try to use flex up it launches some tasks and
>>>> put them in to pending queue and after that nothing launches.
>>>>
>>>> I am attaching below the resourcemanager.out and log snippets.
>>>>
>>>> Please let me know if I am missing something.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> *I see some exceptions in resourcemanager.out files those are as follows*
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> java.lang.IllegalAccessException: Class
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> can not access a member of class javax.ws.rs.core.Response with modifiers
>>>> "protected"
>>>>
>>>> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
>>>>
>>>> at java.lang.Class.newInstance(Class.java:436)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
>>>>
>>>> at
>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> at
>>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>>
>>>> at
>>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>>>>
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>>>>
>>>> at
>>>>
>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>>>>
>>>> at
>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>>
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>>
>>>> at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>>
>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>>
>>>> at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>>
>>>> at
>>>>
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>>>
>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>>>
>>>> at
>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>>>
>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>>
>>>> at
>>>>
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>>>
>>>> at
>>>>
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>>
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.configuration.MyriadConfiguration
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.GetSchedulerStateResponse
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
>>>> attachTypes
>>>>
>>>> INFO: Couldn't find JAX-B element for class
>>>> org.apache.myriad.api.model.FlexUpServiceRequest
>>>>
>>>> Feb 22, 2016 6:21:42 PM
>>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
>>>> resolve
>>>>
>>>> SEVERE: null
>>>>
>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231:
>>>> Transport endpoint is not connected [107]
>>>>
>>>> *This is the resource manager logs*
>>>>
>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> listener on 8033: starting
>>>>
>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 17:39:31,266 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of
>>>> pending queue for nm is: 1
>>>>
>>>> 2016-02-22 17:47:25,450 INFO
>>>>
>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>>>> Release request cache is cleaned up
>>>>
>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource:
>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: null
>>>>
>>>> 2016-02-22 18:22:22,094 INFO
>>> org.apache.myriad.scheduler.MyriadOperations:
>>>> Adding 1 NM instances to cluster
>>>>
>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState:
>>> Marked
>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of
>>>> pending queue for nm is: 2
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>
>>>
>>> --
>>> Dipl.-Inform. Björn Hagemeier
>>> Federated Systems and Data
>>> Juelich Supercomputing Centre
>>> Institute for Advanced Simulation
>>>
>>> Phone: +49 2461 61 1584
>>> Fax : +49 2461 61 6656
>>> Email: [email protected]
>>> Skype: bhagemeier
>>> WWW : http://www.fz-juelich.de/jsc
>>>
>>> JSC is the coordinator of the
>>> John von Neumann Institute for Computing
>>> and member of the
>>> Gauss Centre for Supercomputing
>>>
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Prof. Dr. Sebastian M. Schmidt
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------------------
>>>
>>>
>>
>
>
--
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation
Phone: +49 2461 61 1584
Fax : +49 2461 61 6656
Email: [email protected]
Skype: bhagemeier
WWW : http://www.fz-juelich.de/jsc
JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
