Hmmm...
You did not have to specify
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
either
Can you check if maprfs*.jar is on the classpath (should be - otherwise you
would probably get more weird errors) and it has following class:
YarnDefaultProperties
Thanks,Yuliya
From: John Omernik <j...@omernik.com>
To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com>
Sent: Monday, April 4, 2016 2:18 PM
Subject: Re: Challenges after MapR 5.1 Upgrade.
This was a Upgrade from 5.0. I will post here, note: I have removed the
mapr_shuffle to get node managers to work, however, I am seeing other odd
things, so any help would be appreciated.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Site-specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,myriad_executor</value>
<!-- If using MapR distro, please use the following value:
<value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> -->
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>myriadprod.marathonprod.mesos</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.myriad_executor.class</name>
<value>org.apache.myriad.executor.MyriadExecutorAuxService</value>
</property>
<property>
<name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.am.liveness-monitor.expiry-interval-ms</name>
<value>10000</value>
</property>
<property>
<name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
<value>1000</value>
</property>
<!-- (more) Site-specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${nodemanager.resource.cpu-vcores}</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${nodemanager.resource.memory-mb}</value>
</property>
<!-- Some Basic Resource Constraints -->
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<!-- Dynamic Port Assignment enablement by Mesos -->
<property>
<name>yarn.nodemanager.address</name>
<value>${myriad.yarn.nodemanager.address}</value>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
<name>yarn.nodemanager.webapp.https.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
<name>yarn.nodemanager.localizer.address</name>
<value>${myriad.yarn.nodemanager.localizer.address}</value>
</property>
<!-- Myriad Scheduler configuration -->
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value>
</property>
<!-- Needed for Fine Grain Scaling -->
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>0</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>0</value>
</property>
<!-- Cgroups specific configuration -->
<property>
<description>Who will execute(launch) the containers.</description>
<name>yarn.nodemanager.container-executor.class</name>
<value>${yarn.nodemanager.container-executor.class}</value>
</property>
<property>
<description>The class which should help the LCE handle
resources.</description>
<name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
<value>${yarn.nodemanager.linux-container-executor.resources-handler.class}</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
<value>${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
<value>${yarn.nodemanager.linux-container-executor.cgroups.mount}</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
<value>${yarn.nodemanager.linux-container-executor.cgroups.mount-path}</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>${yarn.nodemanager.linux-container-executor.group}</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.path</name>
<value>${yarn.home}/bin/container-executor</value>
</property>
<property>
<name>yarn.http.policy</name>
<value>HTTP_ONLY</value>
</property>
</configuration>
On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman <yufeld...@yahoo.com.invalid>
wrote:
> YarnDefaultProperties.java that defines class for mapr_direct_shuffle
> should be there even in 5.0, so nothing new there even if maprfs jar is
> outdated - could you also check that?
> Also could you paste content of your yarn-site.xml here?
> Thanks,Yuliya
>
> From: yuliya Feldman <yufeld...@yahoo.com.INVALID>
> To: "dev@myriad.incubator.apache.org" <dev@myriad.incubator.apache.org>
> Sent: Monday, April 4, 2016 1:43 PM
> Subject: Re: Challenges after MapR 5.1 Upgrade.
>
> Hello John,
> Did you upgrade to 5.1 or installed new one?
> Feels like MapR default properties were not loaded - I need to poke around
> and then I will ask you for additional info
> Thanks,Yuliya
>
> From: John Omernik <j...@omernik.com>
> To: dev@myriad.incubator.apache.org
> Sent: Monday, April 4, 2016 12:29 PM
> Subject: Challenges after MapR 5.1 Upgrade.
>
> I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and
> repackaged my hadoop tgz for remote distribution and now I have two
> problems occurring.
>
> 1. At first when I had the mapr direct shuffle enabled per the
> yarn-site.xml on the myriad documentaion, node managers would not start,
> and would fail with the error below.
>
> 2. Once I removed the mapr shuffle from the yarn-site, I got node managers
> started however, when I tried to launch a size 0, I got the other error
> below. Not sure what's happening here.
>
> Any thoughts would be appreciated. Like I said, this was working with 5.0,
> and now doesn't work in 5.1.
>
> Thanks!
>
> John
>
> SHuffle Error
>
> 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed
> in state INITED; cause: java.lang.RuntimeException: No class defined for
> mapr_direct_shuffle
> java.lang.RuntimeException: No class defined for mapr_direct_shuffle
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
>
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
>
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
> 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics
> system...
>
>
> Zero Sized Node Manager Error:
>
> 16/04/04 14:22:49 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in
> state STARTED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
> signal from Resourcemanager ,Registration of NodeManager failed, Message
> from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com
> doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> NodeManager.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
> signal from Resourcemanager ,Registration of NodeManager failed, Message
> from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com
> doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> NodeManager.
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
>
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
> SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed,
> Message from ResourceManager: NodeManager from
> hadoopmapr4.brewingintel.com
> doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
> NodeManager.
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298)
> at
>
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224)
> ... 6 more
>
>
>
>
>
>
