Thanks Darin, and thank you to Yuliya as well. Darin, your fix worked,
Yuliya helped me troubleshoot what appears to be a failed MapR 5.1
upgrade. Basically many of the libs that MapR links into the hadoop
classpath didn't get linked. I reinstalled hadoop components and all seems
well now.
Thanks!
John
On Mon, Apr 4, 2016 at 7:57 PM, Darin Johnson <dbjohnson1...@gmail.com>
wrote:
> Hey John,
>
> I noticed these lines in your yarn-site.xml:
> <!-- Some Basic Resource Constraints -->
> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name>
> <value>512</value>
>
> </property>
>
> <property>
> <name>yarn.scheduler.minimum-allocation-vcores</name>
>
> <value>1</value>
> </property>
>
> If your attempting to launch a zero resource nodemanager for fgs that will
> result in the first stack trace. Both should be explicitly 0 for that
> feature to work (defaults are 1024 and 1 resp, which will fail). You do
> have them set below to 0, however I'm in certain which would take
> precedence.
> On Apr 4, 2016 5:19 PM, "John Omernik" <j...@omernik.com> wrote:
>
> > This was a Upgrade from 5.0. I will post here, note: I have removed the
> > mapr_shuffle to get node managers to work, however, I am seeing other odd
> > things, so any help would be appreciated.
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <configuration>
> >
> > <!-- Site-specific YARN configuration properties -->
> > <property>
> > <name>yarn.nodemanager.aux-services</name>
> > <value>mapreduce_shuffle,myriad_executor</value>
> > <!-- If using MapR distro, please use the following value:
> >
> > <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> -->
> > </property>
> > <property>
> > <name>yarn.resourcemanager.hostname</name>
> > <value>myriadprod.marathonprod.mesos</value>
> > </property>
> > <property>
> >
> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
> > <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.aux-services.myriad_executor.class</name>
> >
> <value>org.apache.myriad.executor.MyriadExecutorAuxService</value>
> > </property>
> > <property>
> > <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
> > <value>2000</value>
> > </property>
> > <property>
> > <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
> > <value>10000</value>
> > </property>
> > <property>
> > <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
> > <value>1000</value>
> > </property>
> > <!-- (more) Site-specific YARN configuration properties -->
> > <property>
> > <name>yarn.nodemanager.resource.cpu-vcores</name>
> > <value>${nodemanager.resource.cpu-vcores}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.resource.memory-mb</name>
> > <value>${nodemanager.resource.memory-mb}</value>
> > </property>
> >
> > <!-- Some Basic Resource Constraints -->
> > <property>
> > <name>yarn.scheduler.minimum-allocation-mb</name>
> > <value>512</value>
> > </property>
> >
> > <property>
> > <name>yarn.scheduler.minimum-allocation-vcores</name>
> > <value>1</value>
> > </property>
> >
> >
> > <!-- Dynamic Port Assignment enablement by Mesos -->
> > <property>
> >
> > <name>yarn.nodemanager.address</name>
> > <value>${myriad.yarn.nodemanager.address}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.webapp.address</name>
> > <value>${myriad.yarn.nodemanager.webapp.address}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.webapp.https.address</name>
> > <value>${myriad.yarn.nodemanager.webapp.address}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.localizer.address</name>
> > <value>${myriad.yarn.nodemanager.localizer.address}</value>
> > </property>
> >
> > <!-- Myriad Scheduler configuration -->
> > <property>
> > <name>yarn.resourcemanager.scheduler.class</name>
> >
> <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value>
> > </property>
> >
> > <!-- Needed for Fine Grain Scaling -->
> > <property>
> > <name>yarn.scheduler.minimum-allocation-vcores</name>
> > <value>0</value>
> > </property>
> > <property>
> > <name>yarn.scheduler.minimum-allocation-vcores</name>
> > <value>0</value>
> > </property>
> > <!-- Cgroups specific configuration -->
> > <property>
> > <description>Who will execute(launch) the
> containers.</description>
> > <name>yarn.nodemanager.container-executor.class</name>
> > <value>${yarn.nodemanager.container-executor.class}</value>
> > </property>
> > <property>
> > <description>The class which should help the LCE handle
> > resources.</description>
> >
> >
> >
> <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
> >
> >
> >
> <value>${yarn.nodemanager.linux-container-executor.resources-handler.class}</value>
> > </property>
> > <property>
> >
> > <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
> >
> >
> >
> <value>${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}</value>
> > </property>
> > <property>
> >
> > <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
> >
> > <value>${yarn.nodemanager.linux-container-executor.cgroups.mount}</value>
> > </property>
> > <property>
> >
> > <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
> >
> >
> >
> <value>${yarn.nodemanager.linux-container-executor.cgroups.mount-path}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.linux-container-executor.group</name>
> > <value>${yarn.nodemanager.linux-container-executor.group}</value>
> > </property>
> > <property>
> > <name>yarn.nodemanager.linux-container-executor.path</name>
> > <value>${yarn.home}/bin/container-executor</value>
> > </property>
> > <property>
> > <name>yarn.http.policy</name>
> > <value>HTTP_ONLY</value>
> > </property>
> > </configuration>
> >
> > On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > >
> > wrote:
> >
> > > YarnDefaultProperties.java that defines class for mapr_direct_shuffle
> > > should be there even in 5.0, so nothing new there even if maprfs jar is
> > > outdated - could you also check that?
> > > Also could you paste content of your yarn-site.xml here?
> > > Thanks,Yuliya
> > >
> > > From: yuliya Feldman <yufeld...@yahoo.com.INVALID>
> > > To: "dev@myriad.incubator.apache.org" <
> dev@myriad.incubator.apache.org>
> > > Sent: Monday, April 4, 2016 1:43 PM
> > > Subject: Re: Challenges after MapR 5.1 Upgrade.
> > >
> > > Hello John,
> > > Did you upgrade to 5.1 or installed new one?
> > > Feels like MapR default properties were not loaded - I need to poke
> > around
> > > and then I will ask you for additional info
> > > Thanks,Yuliya
> > >
> > > From: John Omernik <j...@omernik.com>
> > > To: dev@myriad.incubator.apache.org
> > > Sent: Monday, April 4, 2016 12:29 PM
> > > Subject: Challenges after MapR 5.1 Upgrade.
> > >
> > > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1,
> > and
> > > repackaged my hadoop tgz for remote distribution and now I have two
> > > problems occurring.
> > >
> > > 1. At first when I had the mapr direct shuffle enabled per the
> > > yarn-site.xml on the myriad documentaion, node managers would not
> start,
> > > and would fail with the error below.
> > >
> > > 2. Once I removed the mapr shuffle from the yarn-site, I got node
> > managers
> > > started however, when I tried to launch a size 0, I got the other error
> > > below. Not sure what's happening here.
> > >
> > > Any thoughts would be appreciated. Like I said, this was working with
> > 5.0,
> > > and now doesn't work in 5.1.
> > >
> > > Thanks!
> > >
> > > John
> > >
> > > SHuffle Error
> > >
> > > 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager
> > failed
> > > in state INITED; cause: java.lang.RuntimeException: No class defined
> for
> > > mapr_direct_shuffle
> > > java.lang.RuntimeException: No class defined for mapr_direct_shuffle
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139)
> > > at
> > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> > > at
> > >
> > >
> >
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250)
> > > at
> > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> > > at
> > >
> > >
> >
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256)
> > > at
> > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
> > > 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager
> > metrics
> > > system...
> > >
> > >
> > > Zero Sized Node Manager Error:
> > >
> > > 16/04/04 14:22:49 INFO service.AbstractService: Service
> > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed
> in
> > > state STARTED; cause:
> > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
> SHUTDOWN
> > > signal from Resourcemanager ,Registration of NodeManager failed,
> Message
> > > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com
> > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> > > NodeManager.
> > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
> SHUTDOWN
> > > signal from Resourcemanager ,Registration of NodeManager failed,
> Message
> > > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com
> > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> > > NodeManager.
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230)
> > > at
> > >
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> > > at
> > >
> > >
> >
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267)
> > > at
> > >
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
> > > Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > Recieved
> > > SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
> failed,
> > > Message from ResourceManager: NodeManager from
> > > hadoopmapr4.brewingintel.com
> > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> > > NodeManager.
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298)
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224)
> > > ... 6 more
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
