Thanks Santosh for the prompt reply and all the great help. Is there a way I can redirect mesos task logs to NodeManager logs. Users/Admins/tools are used to look at the logs in specific location. Its very hard to change all of it. Is there any way we can achieve that?
Thanks, Mayank On Tue, Feb 23, 2016 at 11:21 AM, Santosh Marella <[email protected]> wrote: > Great to hear your cluster is up. > > 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however > logs are not going there > NM logs are accessible via the Mesos UI (click on the sandbox for > nm.medium.xxx task and then on stderr link). > Since NMs are launched via Mesos, the NM logs currently do not go to > YARN_HOME/logs dir. > > 2. Log aggregation is also not working in YARN , don't know why , as I > don't have logs > This should work, as Myriad is not really touching any of the traditional > YARN behavior/features. > > 3. Other issue is when i do first time flex up then it launches 2 node > managers instead of one. > This is expected. Once Myriad/RM starts, one medium profile NM is launched > automatically without requiring an explicit flexup. > > Thanks, > Santosh > > On Tue, Feb 23, 2016 at 11:06 AM, Mayank Bansal <[email protected]> wrote: > >> Hi Santosh, >> >> Thanks for looking into it. >> >> I got it working and now I am able to run Node Managers through Myriad. >> >> There are couple of issues which i am still facing , I am sure you guys >> would have faced those and already have the solutions for those. >> >> 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however >> logs are not going there >> 2. Log aggregation is also not working in YARN , don't know why , as I >> don't have logs >> 3. Other issue is when i do first time flex up then it launches 2 node >> managers instead of one. >> >> Please let me know if u know about those issues. >> >> Thanks, >> Mayank >> >> >> On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <[email protected]> >> wrote: >> >>> Hey guys.. the flow of things is as below: >>> >>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on >>> Mesos >>> UI should show this) >>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM >>> process is up. >>> This is queued up until Myriad receives an offer from Mesos that is >>> big >>> enough to launch a medium profle NM. >>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad >>> attempted to launch a NM). >>> >>> Mayank - in your cluster, does Mesos UI show Myriad as a registered >>> framework? Does it show if Myriad launched NM tasks? Perhaps a good >>> starting point is to check what are the sizes of the offers Myriad is >>> receiving (Mesos' master log should show this). You can also enable DEBUG >>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's >>> log4j.properties. >>> >>> >>> >>> Santosh >>> >>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier < >>> [email protected] >>> > wrote: >>> >>> > Hi Mayank, >>> > >>> > it seems you and I are having similar issues. Let's see whether they >>> can >>> > be traced to the same causes. >>> > >>> > >>> > Best regards, >>> > Björn >>> > >>> > Am 22.02.2016 um 19:33 schrieb Mayank Bansal: >>> > > Hi, >>> > > >>> > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x >>> > > >>> > > >>> > > I have followed all the instructions in the wiki >>> > > >>> > > So the resource manager is up and i am able to launch the web UI of >>> > myriad >>> > > as well however whenever I try to use flex up it launches some tasks >>> and >>> > > put them in to pending queue and after that nothing launches. >>> > > >>> > > I am attaching below the resourcemanager.out and log snippets. >>> > > >>> > > Please let me know if I am missing something. >>> > > >>> > > Thanks, >>> > > >>> > > >>> > > *I see some exceptions in resourcemanager.out files those are as >>> follows* >>> > > >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>> > > resolve >>> > > >>> > > SEVERE: null >>> > > >>> > > java.lang.IllegalAccessException: Class >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>> > > can not access a member of class javax.ws.rs.core.Response with >>> modifiers >>> > > "protected" >>> > > >>> > > at >>> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) >>> > > >>> > > at java.lang.Class.newInstance(Class.java:436) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) >>> > > >>> > > at >>> > > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89) >>> > > >>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> > > >>> > > at >>> > > >>> > >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> > > >>> > > at >>> > > >>> > >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> > > >>> > > at java.lang.reflect.Method.invoke(Method.java:498) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) >>> > > >>> > > at >>> > > >>> > >>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) >>> > > >>> > > at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >>> > > >>> > > at >>> > > >>> > >>> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) >>> > > >>> > > at >>> > > >>> > >>> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) >>> > > >>> > > at >>> > > >>> > >>> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) >>> > > >>> > > at >>> > > >>> > >>> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) >>> > > >>> > > at >>> > > >>> > >>> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) >>> > > >>> > > at >>> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) >>> > > >>> > > at >>> > > >>> > >>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >>> > > >>> > > at >>> > > >>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) >>> > > >>> > > at >>> > > >>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >>> > > >>> > > at >>> > > >>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >>> > > >>> > > at org.mortbay.jetty.Server.handle(Server.java:326) >>> > > >>> > > at >>> > > >>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >>> > > >>> > > at >>> > > >>> > >>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) >>> > > >>> > > at >>> org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) >>> > > >>> > > at >>> > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >>> > > >>> > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >>> > > >>> > > at >>> > > >>> > >>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) >>> > > >>> > > at >>> > > >>> > >>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) >>> > > >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>> > > attachTypes >>> > > >>> > > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>> > > attachTypes >>> > > >>> > > INFO: Couldn't find JAX-B element for class >>> > > org.apache.myriad.configuration.MyriadConfiguration >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>> > > attachTypes >>> > > >>> > > INFO: Couldn't find JAX-B element for class >>> > > org.apache.myriad.api.model.GetSchedulerStateResponse >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator >>> > > attachTypes >>> > > >>> > > INFO: Couldn't find JAX-B element for class >>> > > org.apache.myriad.api.model.FlexUpServiceRequest >>> > > >>> > > Feb 22, 2016 6:21:42 PM >>> > > >>> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 >>> > > resolve >>> > > >>> > > SEVERE: null >>> > > >>> > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on >>> fd=231: >>> > > Transport endpoint is not connected [107] >>> > > >>> > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on >>> fd=231: >>> > > Transport endpoint is not connected [107] >>> > > >>> > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on >>> fd=231: >>> > > Transport endpoint is not connected [107] >>> > > >>> > > *This is the resource manager logs* >>> > > >>> > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server >>> > > listener on 8033: starting >>> > > >>> > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource: >>> > > Received flexup request. Profile: medium, Instances: 1, Constraints: >>> null >>> > > >>> > > 2016-02-22 17:39:31,266 INFO >>> > org.apache.myriad.scheduler.MyriadOperations: >>> > > Adding 1 NM instances to cluster >>> > > >>> > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: >>> > Marked >>> > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size >>> of >>> > > pending queue for nm is: 1 >>> > > >>> > > 2016-02-22 17:47:25,450 INFO >>> > > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: >>> > > Release request cache is cleaned up >>> > > >>> > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource: >>> > > Received flexup request. Profile: medium, Instances: 1, Constraints: >>> null >>> > > >>> > > 2016-02-22 18:22:22,094 INFO >>> > org.apache.myriad.scheduler.MyriadOperations: >>> > > Adding 1 NM instances to cluster >>> > > >>> > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: >>> > Marked >>> > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size >>> of >>> > > pending queue for nm is: 2 >>> > > >>> > > >>> > > >>> > > Thanks, >>> > > >>> > >>> > >>> > -- >>> > Dipl.-Inform. Björn Hagemeier >>> > Federated Systems and Data >>> > Juelich Supercomputing Centre >>> > Institute for Advanced Simulation >>> > >>> > Phone: +49 2461 61 1584 >>> > Fax : +49 2461 61 6656 >>> > Email: [email protected] >>> > Skype: bhagemeier >>> > WWW : http://www.fz-juelich.de/jsc >>> > >>> > JSC is the coordinator of the >>> > John von Neumann Institute for Computing >>> > and member of the >>> > Gauss Centre for Supercomputing >>> > >>> > >>> > >>> ------------------------------------------------------------------------------------- >>> > >>> > >>> ------------------------------------------------------------------------------------- >>> > Forschungszentrum Juelich GmbH >>> > 52425 Juelich >>> > Sitz der Gesellschaft: Juelich >>> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher >>> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), >>> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, >>> > Prof. Dr. Sebastian M. Schmidt >>> > >>> > >>> ------------------------------------------------------------------------------------- >>> > >>> > >>> ------------------------------------------------------------------------------------- >>> > >>> > >>> >> >> >> >> -- >> - Mayank >> > > -- - Mayank
