This probably requires a code change in Myriad. Currently Myriad uses "$YARN_HOME/bin/yarn nodemanager" as the command to launch the NM. This makes the NM's log4j output to be streamed to stdout. ( https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/java/org/apache/myriad/scheduler/NMExecutorCLGenImpl.java#L65 )
I think we should use "$YARN_HOME/sbin/yarn-daemon.sh start nodemanager" so that the log4j output is written to $YARN_HOME/logs/xxx-nodemanger-yyy.log. Do you happen to have a Myriad git clone to try it out and see if it helps? Thanks, Santosh On Tue, Feb 23, 2016 at 1:49 PM, Mayank Bansal <[email protected]> wrote: > Thanks Santosh for the prompt reply and all the great help. > > Is there a way I can redirect mesos task logs to NodeManager logs. > Users/Admins/tools are used to look at the logs in specific location. Its > very hard to change all of it. Is there any way we can achieve that? > > Thanks, > Mayank > > On Tue, Feb 23, 2016 at 11:21 AM, Santosh Marella <[email protected]> > wrote: > >> Great to hear your cluster is up. >> >> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however >> logs are not going there >> NM logs are accessible via the Mesos UI (click on the sandbox for >> nm.medium.xxx task and then on stderr link). >> Since NMs are launched via Mesos, the NM logs currently do not go to >> YARN_HOME/logs dir. >> >> 2. Log aggregation is also not working in YARN , don't know why , as I >> don't have logs >> This should work, as Myriad is not really touching any of the traditional >> YARN behavior/features. >> >> 3. Other issue is when i do first time flex up then it launches 2 node >> managers instead of one. >> This is expected. Once Myriad/RM starts, one medium profile NM is >> launched automatically without requiring an explicit flexup. >> >> Thanks, >> Santosh >> >> On Tue, Feb 23, 2016 at 11:06 AM, Mayank Bansal <[email protected]> >> wrote: >> >>> Hi Santosh, >>> >>> Thanks for looking into it. >>> >>> I got it working and now I am able to run Node Managers through Myriad. >>> >>> There are couple of issues which i am still facing , I am sure you guys >>> would have faced those and already have the solutions for those. >>> >>> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however >>> logs are not going there >>> 2. Log aggregation is also not working in YARN , don't know why , as I >>> don't have logs >>> 3. Other issue is when i do first time flex up then it launches 2 node >>> managers instead of one. >>> >>> Please let me know if u know about those issues. >>> >>> Thanks, >>> Mayank >>> >>> >>> On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <[email protected]> >>> wrote: >>> >>>> Hey guys.. the flow of things is as below: >>>> >>>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on >>>> Mesos >>>> UI should show this) >>>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as >>>> RM >>>> process is up. >>>> This is queued up until Myriad receives an offer from Mesos that is >>>> big >>>> enough to launch a medium profle NM. >>>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad >>>> attempted to launch a NM). >>>> >>>> Mayank - in your cluster, does Mesos UI show Myriad as a registered >>>> framework? Does it show if Myriad launched NM tasks? Perhaps a good >>>> starting point is to check what are the sizes of the offers Myriad is >>>> receiving (Mesos' master log should show this). You can also enable >>>> DEBUG >>>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's >>>> log4j.properties. >>>> >>>> >>>> >>>> Santosh >>>> >>>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier < >>>> [email protected] >>>> > wrote: >>>> >>>> > Hi Mayank, >>>> > >>>> > it seems you and I are having similar issues. Let's see whether they >>>> can >>>> > be traced to the same causes. >>>> > >>>> > >>>> > Best regards, >>>> > Björn >>>> > >>>> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal: >>>> > > Hi, >>>> > > >>>> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x >>>> > > >>>> > > >>>> > > I have followed all the instructions in the wiki >>>> > > >>>> > > So the resource manager is up and i am able to launch the web UI of >>>> > myriad >>>> > > as well however whenever I try to use flex up it launches some >>>> tasks and >>>> > > put them in to pending queue and after that nothing launches. >>>> > > >>>> > > I am attaching below the resourcemanager.out and log snippets. >>>> > > >>>> > > Please let me know if I am missing something. >>>> > > >>>> > > Thanks, >>>> > > >>>> > > >>>> > > *I see some exceptions in resourcemanager.out files those are as >>>> follows* >>>> > > >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>>> > > resolve >>>> > > >>>> > > SEVERE: null >>>> > > >>>> > > java.lang.IllegalAccessException: Class >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>>> > > can not access a member of class javax.ws.rs.core.Response with >>>> modifiers >>>> > > "protected" >>>> > > >>>> > > at >>>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) >>>> > > >>>> > > at java.lang.Class.newInstance(Class.java:436) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) >>>> > > >>>> > > at >>>> > > >>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89) >>>> > > >>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> Method) >>>> > > >>>> > > at >>>> > > >>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> > > >>>> > > at >>>> > > >>>> > >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> > > >>>> > > at java.lang.reflect.Method.invoke(Method.java:498) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) >>>> > > >>>> > > at >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >>>> > > >>>> > > at >>>> > > >>>> > >>>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>>> > > >>>> > > at >>>> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>>> > > >>>> > > at >>>> > > >>>> > >>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>>> > > >>>> > > at >>>> > > >>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>>> > > >>>> > > at >>>> > > >>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>>> > > >>>> > > at >>>> > > >>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>>> > > >>>> > > at org.mortbay.jetty.Server.handle(Server.java:326) >>>> > > >>>> > > at >>>> > > >>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>>> > > >>>> > > at >>>> > > >>>> > >>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>>> > > >>>> > > at >>>> org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>>> > > >>>> > > at >>>> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>>> > > >>>> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>>> > > >>>> > > at >>>> > > >>>> > >>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>>> > > >>>> > > at >>>> > > >>>> > >>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>>> > > >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>>> > > attachTypes >>>> > > >>>> > > INFO: Couldn't find JAX-B element for class >>>> javax.ws.rs.core.Response >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>>> > > attachTypes >>>> > > >>>> > > INFO: Couldn't find JAX-B element for class >>>> > > org.apache.myriad.configuration.MyriadConfiguration >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>>> > > attachTypes >>>> > > >>>> > > INFO: Couldn't find JAX-B element for class >>>> > > org.apache.myriad.api.model.GetSchedulerStateResponse >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>>> > > attachTypes >>>> > > >>>> > > INFO: Couldn't find JAX-B element for class >>>> > > org.apache.myriad.api.model.FlexUpServiceRequest >>>> > > >>>> > > Feb 22, 2016 6:21:42 PM >>>> > > >>>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>>> > > resolve >>>> > > >>>> > > SEVERE: null >>>> > > >>>> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on >>>> fd=231: >>>> > > Transport endpoint is not connected [107] >>>> > > >>>> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on >>>> fd=231: >>>> > > Transport endpoint is not connected [107] >>>> > > >>>> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on >>>> fd=231: >>>> > > Transport endpoint is not connected [107] >>>> > > >>>> > > *This is the resource manager logs* >>>> > > >>>> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC >>>> Server >>>> > > listener on 8033: starting >>>> > > >>>> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource: >>>> > > Received flexup request. Profile: medium, Instances: 1, >>>> Constraints: null >>>> > > >>>> > > 2016-02-22 17:39:31,266 INFO >>>> > org.apache.myriad.scheduler.MyriadOperations: >>>> > > Adding 1 NM instances to cluster >>>> > > >>>> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: >>>> > Marked >>>> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size >>>> of >>>> > > pending queue for nm is: 1 >>>> > > >>>> > > 2016-02-22 17:47:25,450 INFO >>>> > > >>>> > >>>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: >>>> > > Release request cache is cleaned up >>>> > > >>>> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource: >>>> > > Received flexup request. Profile: medium, Instances: 1, >>>> Constraints: null >>>> > > >>>> > > 2016-02-22 18:22:22,094 INFO >>>> > org.apache.myriad.scheduler.MyriadOperations: >>>> > > Adding 1 NM instances to cluster >>>> > > >>>> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: >>>> > Marked >>>> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size >>>> of >>>> > > pending queue for nm is: 2 >>>> > > >>>> > > >>>> > > >>>> > > Thanks, >>>> > > >>>> > >>>> > >>>> > -- >>>> > Dipl.-Inform. Björn Hagemeier >>>> > Federated Systems and Data >>>> > Juelich Supercomputing Centre >>>> > Institute for Advanced Simulation >>>> > >>>> > Phone: +49 2461 61 1584 >>>> > Fax : +49 2461 61 6656 >>>> > Email: [email protected] >>>> > Skype: bhagemeier >>>> > WWW : http://www.fz-juelich.de/jsc >>>> > >>>> > JSC is the coordinator of the >>>> > John von Neumann Institute for Computing >>>> > and member of the >>>> > Gauss Centre for Supercomputing >>>> > >>>> > >>>> > >>>> ------------------------------------------------------------------------------------- >>>> > >>>> > >>>> ------------------------------------------------------------------------------------- >>>> > Forschungszentrum Juelich GmbH >>>> > 52425 Juelich >>>> > Sitz der Gesellschaft: Juelich >>>> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >>>> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), >>>> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >>>> > Prof. Dr. Sebastian M. Schmidt >>>> > >>>> > >>>> ------------------------------------------------------------------------------------- >>>> > >>>> > >>>> ------------------------------------------------------------------------------------- >>>> > >>>> > >>>> >>> >>> >>> >>> -- >>> - Mayank >>> >> >> > > > -- > - Mayank >
