Please check the white listed users in the following properties file : HADOOP_HOME/etc/hadoop/container-executor.cfg
Need to set either of these 2 properties. min.user.id=500 (make it < 99) allowed.system.users=mapr (or add user here) Need to give permissions for the user to run the job. Thank you Aashreya On Fri, Feb 26, 2016 at 12:22 AM, Björn Hagemeier <[email protected] > wrote: > Dear Santosh, > > thanks a lot for having me check the myriad-config-default.yml. I did > not have my own configuration and due to a lack of warnings in any of > the logs, I had assumed everything to be ok to some extent. > > So, now I have the myriad-config-default.yml with much of the needed > configuration. On the way I ran into an issue also mentioned in [1], so > I have set JAVA_HOME and YARN_NODEMANAGER_OPTS now as well as > HADOOP_CONF_DIR and HADOOP_LIBEXEC_DIR, which are used by the NM startup > scripts. > > I can now see NMs being started and torn down as a result of flexing > up/down from the Myriad UI. However, when sending Yarn jobs, e.g. > terasort, I am having issues with the user mapping. > > I get the following message: > > ================================================== > Diagnostics: Application application_1456472909573_0002 initialization > failed (exitCode=255) with output: main : command provided 0 > main : run as user is nobody > main : requested yarn user is bjoernh > Requested user nobody is not whitelisted and has id 99,which is below > the minimum allowed 1000 > ================================================== > > When I still used my Yarn installation w/o Myriad, this worked fine. > Could this be related to the framework user? I set it to "yarn" as this > is that the package installation expects and has been working to the > best of my knowledge. > > > Cheers, > Björn > > > [1] > > https://mail-archives.apache.org/mod_mbox/myriad-dev/201512.mbox/%3cca+kw_myt4wpr-b54kjyo0hxkwsrthez2x41fftcuomvmqrq...@mail.gmail.com%3E > > Am 23.02.2016 um 20:00 schrieb Santosh Marella: > >> What makes me wonder is that I do have a running Myriad, i.e. I can > >> access the Myriad UI on port 8192, but it does not show up in Mesos. > > > > It's possible you have a wrong Myriad configuration. Can you please check > > the mesosMaster address in your myriad-config-default.yml file? > > > > e.g.: > > > https://github.com/apache/incubator-myriad/blob/master/myriad-scheduler/src/main/resources/myriad-config-default.yml#L19 > > > > Santosh > > > > On Tue, Feb 23, 2016 at 3:55 AM, Björn Hagemeier < > [email protected]> > > wrote: > > > >> Hi Santosh, > >> > >> Am 23.02.2016 um 01:21 schrieb Santosh Marella: > >>> Hey guys.. the flow of things is as below: > >>> > >>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on > >> Mesos > >>> UI should show this) > >> I think this is where it is already failing for me. I do not see the > >> Myriad framework in the UI. > >>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as > RM > >>> process is up. > >>> This is queued up until Myriad receives an offer from Mesos that is > >> big > >>> enough to launch a medium profle NM. > >> Yes, I do see the medium profile NM in the Myriad UI. > >>> (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad > >>> attempted to launch a NM). > >> Nope, unfortunately it does not. > >> > >> What makes me wonder is that I do have a running Myriad, i.e. I can > >> access the Myriad UI on port 8192, but it does not show up in Mesos. > >> > >> I guess my other problem, not seeing any offers, is a follow-on of this. > >> > >>> > >>> Mayank - in your cluster, does Mesos UI show Myriad as a registered > >>> framework? Does it show if Myriad launched NM tasks? Perhaps a good > >>> starting point is to check what are the sizes of the offers Myriad is > >>> receiving (Mesos' master log should show this). You can also enable > DEBUG > >>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in > RM's > >>> log4j.properties. > >> I will try and increase the log level to DEBUG. > >> > >> > >> Cheers, > >> Björn > >>> > >>> > >>> > >>> Santosh > >>> > >>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier < > >> [email protected] > >>>> wrote: > >>> > >>>> Hi Mayank, > >>>> > >>>> it seems you and I are having similar issues. Let's see whether they > can > >>>> be traced to the same causes. > >>>> > >>>> > >>>> Best regards, > >>>> Björn > >>>> > >>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal: > >>>>> Hi, > >>>>> > >>>>> I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x > >>>>> > >>>>> > >>>>> I have followed all the instructions in the wiki > >>>>> > >>>>> So the resource manager is up and i am able to launch the web UI of > >>>> myriad > >>>>> as well however whenever I try to use flex up it launches some tasks > >> and > >>>>> put them in to pending queue and after that nothing launches. > >>>>> > >>>>> I am attaching below the resourcemanager.out and log snippets. > >>>>> > >>>>> Please let me know if I am missing something. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> > >>>>> *I see some exceptions in resourcemanager.out files those are as > >> follows* > >>>>> > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > >> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > >>>>> resolve > >>>>> > >>>>> SEVERE: null > >>>>> > >>>>> java.lang.IllegalAccessException: Class > >>>>> > >> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > >>>>> can not access a member of class javax.ws.rs.core.Response with > >> modifiers > >>>>> "protected" > >>>>> > >>>>> at > >> sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) > >>>>> > >>>>> at java.lang.Class.newInstance(Class.java:436) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) > >>>>> > >>>>> at > >>>>> com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89) > >>>>> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >>>>> > >>>>> at > >>>>> > >>>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>>> > >>>>> at > >>>>> > >>>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>> > >>>>> at java.lang.reflect.Method.invoke(Method.java:498) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > >>>>> > >>>>> at > javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) > >>>>> > >>>>> at > >>>>> > >>>> > >> > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > >>>>> > >>>>> at > >>>>> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > >>>>> > >>>>> at > >>>>> > >>>> > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > >>>>> > >>>>> at > >>>>> > >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > >>>>> > >>>>> at > >>>>> > >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > >>>>> > >>>>> at > >>>>> > >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > >>>>> > >>>>> at org.mortbay.jetty.Server.handle(Server.java:326) > >>>>> > >>>>> at > >>>>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > >>>>> > >>>>> at > >>>>> > >>>> > >> > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > >>>>> > >>>>> at > org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > >>>>> > >>>>> at > >>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > >>>>> > >>>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > >>>>> > >>>>> at > >>>>> > >>>> > >> > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > >>>>> > >>>>> at > >>>>> > >>>> > >> > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > >>>>> > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > >>>>> attachTypes > >>>>> > >>>>> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > >>>>> attachTypes > >>>>> > >>>>> INFO: Couldn't find JAX-B element for class > >>>>> org.apache.myriad.configuration.MyriadConfiguration > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > >>>>> attachTypes > >>>>> > >>>>> INFO: Couldn't find JAX-B element for class > >>>>> org.apache.myriad.api.model.GetSchedulerStateResponse > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > >>>>> attachTypes > >>>>> > >>>>> INFO: Couldn't find JAX-B element for class > >>>>> org.apache.myriad.api.model.FlexUpServiceRequest > >>>>> > >>>>> Feb 22, 2016 6:21:42 PM > >>>>> > >> > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > >>>>> resolve > >>>>> > >>>>> SEVERE: null > >>>>> > >>>>> E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on > fd=231: > >>>>> Transport endpoint is not connected [107] > >>>>> > >>>>> E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on > fd=231: > >>>>> Transport endpoint is not connected [107] > >>>>> > >>>>> E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on > fd=231: > >>>>> Transport endpoint is not connected [107] > >>>>> > >>>>> *This is the resource manager logs* > >>>>> > >>>>> 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server > >>>>> listener on 8033: starting > >>>>> > >>>>> 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource: > >>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: > >> null > >>>>> > >>>>> 2016-02-22 17:39:31,266 INFO > >>>> org.apache.myriad.scheduler.MyriadOperations: > >>>>> Adding 1 NM instances to cluster > >>>>> > >>>>> 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: > >>>> Marked > >>>>> taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size > of > >>>>> pending queue for nm is: 1 > >>>>> > >>>>> 2016-02-22 17:47:25,450 INFO > >>>>> > >>>> > >> > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: > >>>>> Release request cache is cleaned up > >>>>> > >>>>> 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource: > >>>>> Received flexup request. Profile: medium, Instances: 1, Constraints: > >> null > >>>>> > >>>>> 2016-02-22 18:22:22,094 INFO > >>>> org.apache.myriad.scheduler.MyriadOperations: > >>>>> Adding 1 NM instances to cluster > >>>>> > >>>>> 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: > >>>> Marked > >>>>> taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size > of > >>>>> pending queue for nm is: 2 > >>>>> > >>>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>> > >>>> > >>>> -- > >>>> Dipl.-Inform. Björn Hagemeier > >>>> Federated Systems and Data > >>>> Juelich Supercomputing Centre > >>>> Institute for Advanced Simulation > >>>> > >>>> Phone: +49 2461 61 1584 > >>>> Fax : +49 2461 61 6656 > >>>> Email: [email protected] > >>>> Skype: bhagemeier > >>>> WWW : http://www.fz-juelich.de/jsc > >>>> > >>>> JSC is the coordinator of the > >>>> John von Neumann Institute for Computing > >>>> and member of the > >>>> Gauss Centre for Supercomputing > >>>> > >>>> > >>>> > >> > ------------------------------------------------------------------------------------- > >>>> > >>>> > >> > ------------------------------------------------------------------------------------- > >>>> Forschungszentrum Juelich GmbH > >>>> 52425 Juelich > >>>> Sitz der Gesellschaft: Juelich > >>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > >>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > >>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > >>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > >>>> Prof. Dr. Sebastian M. Schmidt > >>>> > >>>> > >> > ------------------------------------------------------------------------------------- > >>>> > >>>> > >> > ------------------------------------------------------------------------------------- > >>>> > >>>> > >>> > >> > >> > >> -- > >> Dipl.-Inform. Björn Hagemeier > >> Federated Systems and Data > >> Juelich Supercomputing Centre > >> Institute for Advanced Simulation > >> > >> Phone: +49 2461 61 1584 > >> Fax : +49 2461 61 6656 > >> Email: [email protected] > >> Skype: bhagemeier > >> WWW : http://www.fz-juelich.de/jsc > >> > >> JSC is the coordinator of the > >> John von Neumann Institute for Computing > >> and member of the > >> Gauss Centre for Supercomputing > >> > >> > >> > ------------------------------------------------------------------------------------- > >> > >> > ------------------------------------------------------------------------------------- > >> Forschungszentrum Juelich GmbH > >> 52425 Juelich > >> Sitz der Gesellschaft: Juelich > >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > >> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > >> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > >> Prof. Dr. Sebastian M. Schmidt > >> > >> > ------------------------------------------------------------------------------------- > >> > >> > ------------------------------------------------------------------------------------- > >> > >> > > > > > -- > Dipl.-Inform. Björn Hagemeier > Federated Systems and Data > Juelich Supercomputing Centre > Institute for Advanced Simulation > > Phone: +49 2461 61 1584 > Fax : +49 2461 61 6656 > Email: [email protected] > Skype: bhagemeier > WWW : http://www.fz-juelich.de/jsc > > JSC is the coordinator of the > John von Neumann Institute for Computing > and member of the > Gauss Centre for Supercomputing > > > ------------------------------------------------------------------------------------- > > ------------------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > ------------------------------------------------------------------------------------- > > ------------------------------------------------------------------------------------- > > >
