So some progress: I am getting the error below complaining about ownership of files. In marathon I have user:root on my task, in the myriad config, I have mapr is user 700, so I am unsure on that, I will try with framworkUser being root, see if that works?
frameworkUser: mapr # Should be the same user running the resource manager. frameworkSuperUser: darkness # Must be root or have passwordless sudo on all nodes! at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210) ... 3 more Caused by: ExitCodeException exitCode=24: File /tmp/mesos/slaves/20150907-111332-1660987584-5050-8033-S1/frameworks/20150907-111332-1660987584-5050-8033-0005/executors/myriad_executor20150907-111332-1660987584-5050-8033-000520150907-111332-1660987584-5050-8033-O12269720150907-111332-1660987584-5050-8033-S1/runs/8c48f443-f768-45b1-8cb2-55ff5b5a99d8 must be owned by root, but is owned by 700 at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182) ... 4 more 15/09/08 15:23:24 WARN service.AbstractService: When stopping the service NodeManager : java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:162) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:274) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511) 15/09/08 15:23:24 FATAL nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:212) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511) Caused by: java.io.IOException: Linux container executor not configured properly (error=24) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:188) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210) ... 3 more Caused by: ExitCodeException exitCode=24: File /tmp/mesos/slaves/20150907-111332-1660987584-5050-8033-S1/frameworks/20150907-111332-1660987584-5050-8033-0005/executors/myriad_executor20150907-111332-1660987584-5050-8033-000520150907-111332-1660987584-5050-8033-O12269720150907-111332-1660987584-5050-8033-S1/runs/8c48f443-f768-45b1-8cb2-55ff5b5a99d8 must be owned by root, but is owned by 700 at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182) ... 4 more 15/09/08 15:23:24 INFO nodemanager.NodeManager: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at hadoopmapr2.brewingintel.com/192.168.0.99 ************************************************************/ On Tue, Sep 8, 2015 at 3:23 PM, John Omernik <[email protected]> wrote: > Also a side note: The Flexing up and now having to have at least one node > manager specified at startup: > > nmInstances: # NMs to start with. Requires at least 1 NM with a non-zero > profile. > > medium: 1 # <profile_name : instances> > > > Is going to lead to task failures with mesos dns because the name won't be > ready right away (1 minute delay after kicking off Myriad) do we NEED to > have a non-0 profile nodemanager startup with the resource manager? > > On Tue, Sep 8, 2015 at 3:16 PM, John Omernik <[email protected]> wrote: > >> Cool. Question about the yarn-site.xml in general. >> >> I was struggling with some things in the wiki on this page: >> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators >> >> Basically in step 5: >> Step 5: Configure YARN to use Myriad >> >> Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as >> instructed in Sample: myriad-config-default.yml >> <https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml> >> . >> >> >> (It should not link to the yml, but to the yarn site, side issue) it has >> us put that information in the yarn-site.xml This makes sense. The >> resource manager needs to be aware of the myriad stuff. >> >> Then I go to create a tarbal, (which I SHOULD be able to use for both >> resource manager and nodemanager... right?) However, the instructions state >> to remove the *.xml files. >> >> Step 6: Create the Tarball >> >> The tarball has all of the files needed for the Node Managers and >> Resource Managers. The following shows how to create the tarball and place >> it in HDFS: >> cd ~ >> sudo cp -rp /opt/hadoop-2.7.0 . >> sudo rm hadoop-2.7.0/etc/hadoop/*.xml >> sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0 >> hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist >> >> >> What I ended up doing... since I am running the resourcemanager (myriad) >> in marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz >> which has the all the xml files still in the tar ball for shipping to >> marathon. Then other is hadoop-2.7.0-NM.tar.gz which per the instructions >> removes the *.xml files from the /etc/hadoop/ directory. >> >> >> I guess... my logic is that myriad creates the conf directory for the >> nodemanagers... but then I thought, and I overthinking something? Am I >> missing something? Could that be factoring into what I am doing here? >> >> >> Obviously my first steps are to add the extra yarn-site.xml entries, but >> in this current setup, they are only going into the resource manager >> yarn-site as the the node-managers don't have a yarn-site in their >> directories. >> >> >> >> >> >> >> >> On Tue, Sep 8, 2015 at 3:09 PM, yuliya Feldman < >> [email protected]> wrote: >> >>> Take a look at : https://github.com/mesos/myriad/pull/128 >>> for yarn-site.xml updates >>> >>> From: John Omernik <[email protected]> >>> To: [email protected] >>> Sent: Tuesday, September 8, 2015 12:38 PM >>> Subject: Getting Nodes to be "Running" in Mesos >>> >>> So I am playing around with a recent build of Myriad, and I am using MapR >>> 5.0 (hadoop-2.7.0) I hate to use the dev list as a "help Myriad won't >>> run" >>> forum, so please forgive me if I am using the list wrong. >>> >>> Basically, I seem to be able to get myriad running, and the things up, >>> and >>> it tries to start a nodemanager. >>> >>> In mesos, the status of the nodemanager task never gets past staging, and >>> eventually, fails. The logs for both the node manager and myriad, seem >>> to >>> look healthy, and I am not sure where I should look next to troubleshoot >>> what is happening. Basically you can see the registration of the >>> nodemanager, and then it fails with no error in the logs... Any thoughts >>> would be appreciated on where I can look next for troubleshooting. >>> >>> >>> Node Manager Logs (complete) >>> >>> STARTUP_MSG: build = [email protected]:mapr/private-hadoop-common.git >>> -r fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on >>> 2015-08-19T20:02Z >>> STARTUP_MSG: java = 1.8.0_45-internal >>> ************************************************************/ >>> 15/09/08 14:35:23 INFO nodemanager.NodeManager: registered UNIX signal >>> handlers for [TERM, HUP, INT] >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for >>> class org.apache.hadoop.yarn.server.nodemanager.NodeManager >>> 15/09/08 14:35:24 INFO impl.MetricsConfig: loaded properties from >>> hadoop-metrics2.properties >>> 15/09/08 14:35:24 INFO impl.MetricsSystemImpl: Scheduled snapshot >>> period at 10 second(s). >>> 15/09/08 14:35:24 INFO impl.MetricsSystemImpl: NodeManager metrics >>> system started >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadService >>> 15/09/08 14:35:24 INFO localizer.ResourceLocalizationService: per >>> directory file limit = 8192 >>> 15/09/08 14:35:24 INFO localizer.ResourceLocalizationService: >>> usercache path : >>> file:///tmp/hadoop-mapr/nm-local-dir/usercache_DEL_1441740924753 >>> 15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class >>> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType >>> for class >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker >>> 15/09/08 14:35:24 WARN containermanager.AuxServices: The Auxilurary >>> Service named 'mapreduce_shuffle' in the configuration is for class >>> org.apache.hadoop.mapred.ShuffleHandler which has a name of >>> 'httpshuffle'. Because these are not the same tools trying to send >>> ServiceData and read Service Meta Data may have issues unless the >>> refer to the name in the config. >>> 15/09/08 14:35:24 INFO containermanager.AuxServices: Adding auxiliary >>> service httpshuffle, "mapreduce_shuffle" >>> 15/09/08 14:35:24 INFO monitor.ContainersMonitorImpl: Using >>> ResourceCalculatorPlugin : >>> org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@1a5b6f42 >>> 15/09/08 14:35:24 INFO monitor.ContainersMonitorImpl: Using >>> ResourceCalculatorProcessTree : null >>> 15/09/08 14:35:24 INFO monitor.ContainersMonitorImpl: Physical memory >>> check enabled: true >>> 15/09/08 14:35:24 INFO monitor.ContainersMonitorImpl: Virtual memory >>> check enabled: false >>> 15/09/08 14:35:24 INFO nodemanager.NodeStatusUpdaterImpl: Initialized >>> nodemanager for null: physical-memory=16384 virtual-memory=34407 >>> virtual-cores=4 disks=4.0 >>> 15/09/08 14:35:24 INFO ipc.CallQueueManager: Using callQueue class >>> java.util.concurrent.LinkedBlockingQueue >>> 15/09/08 14:35:24 INFO ipc.Server: Starting Socket Reader #1 for port >>> 55449 >>> 15/09/08 14:35:24 INFO pb.RpcServerFactoryPBImpl: Adding protocol >>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the server >>> 15/09/08 14:35:24 INFO containermanager.ContainerManagerImpl: Blocking >>> new container-requests as container manager rpc server is still >>> starting. >>> 15/09/08 14:35:24 INFO ipc.Server: IPC Server Responder: starting >>> 15/09/08 14:35:24 INFO ipc.Server: IPC Server listener on 55449: starting >>> 15/09/08 14:35:24 INFO security.NMContainerTokenSecretManager: >>> Updating node address : hadoopmapr5.brewingintel.com:55449 >>> 15/09/08 14:35:24 INFO ipc.CallQueueManager: Using callQueue class >>> java.util.concurrent.LinkedBlockingQueue >>> 15/09/08 14:35:24 INFO ipc.Server: Starting Socket Reader #1 for port >>> 8040 >>> 15/09/08 14:35:24 INFO pb.RpcServerFactoryPBImpl: Adding protocol >>> org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB >>> to the server >>> 15/09/08 14:35:24 INFO ipc.Server: IPC Server Responder: starting >>> 15/09/08 14:35:24 INFO ipc.Server: IPC Server listener on 8040: starting >>> 15/09/08 14:35:24 INFO localizer.ResourceLocalizationService: >>> Localizer started on port 8040 >>> 15/09/08 14:35:24 INFO mapred.IndexCache: IndexCache created with max >>> memory = 10485760 >>> 15/09/08 14:35:24 INFO mapred.ShuffleHandler: httpshuffle listening on >>> port 13562 >>> 15/09/08 14:35:24 INFO containermanager.ContainerManagerImpl: >>> ContainerManager started at hadoopmapr5/192.168.0.96:55449 >>> 15/09/08 14:35:24 INFO containermanager.ContainerManagerImpl: >>> ContainerManager bound to 0.0.0.0/0.0.0.0:0 >>> 15/09/08 14:35:24 INFO webapp.WebServer: Instantiating NMWebApp at >>> 0.0.0.0:8042 >>> 15/09/08 14:35:24 INFO mortbay.log: Logging to >>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >>> org.mortbay.log.Slf4jLog >>> 15/09/08 14:35:24 INFO http.HttpRequestLog: Http request log for >>> http.requests.nodemanager is not defined >>> 15/09/08 14:35:24 INFO http.HttpServer2: Added global filter 'safety' >>> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) >>> 15/09/08 14:35:24 INFO http.HttpServer2: Added filter >>> static_user_filter >>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) >>> to context node >>> 15/09/08 14:35:24 INFO http.HttpServer2: Added filter >>> static_user_filter >>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) >>> to context static >>> 15/09/08 14:35:24 INFO http.HttpServer2: Added filter >>> static_user_filter >>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) >>> to context logs >>> 15/09/08 14:35:24 INFO http.HttpServer2: adding path spec: /node/* >>> 15/09/08 14:35:24 INFO http.HttpServer2: adding path spec: /ws/* >>> 15/09/08 14:35:24 INFO http.HttpServer2: Jetty bound to port 8042 >>> 15/09/08 14:35:24 INFO mortbay.log: jetty-6.1.26 >>> 15/09/08 14:35:24 INFO mortbay.log: Extract >>> >>> jar:file:/tmp/mesos/slaves/20150907-111332-1660987584-5050-8033-S3/frameworks/20150907-111332-1660987584-5050-8033-0003/executors/myriad_executor20150907-111332-1660987584-5050-8033-000320150907-111332-1660987584-5050-8033-O11824820150907-111332-1660987584-5050-8033-S3/runs/67cc8f37-b6d4-4018-a9b4-0071d020c9a5/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/webapps/node >>> to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp >>> 15/09/08 14:35:25 INFO mortbay.log: Started >>> [email protected]:8042 >>> 15/09/08 14:35:25 INFO webapp.WebApps: Web app /node started at 8042 >>> 15/09/08 14:35:25 INFO webapp.WebApps: Registered webapp guice modules >>> 15/09/08 14:35:25 INFO client.RMProxy: Connecting to ResourceManager >>> at myriad.marathon.mesos/192.168.0.99:8031 >>> 15/09/08 14:35:25 INFO nodemanager.NodeStatusUpdaterImpl: Sending out >>> 0 NM container statuses: [] >>> 15/09/08 14:35:25 INFO nodemanager.NodeStatusUpdaterImpl: Registering >>> with RM using containers :[] >>> 15/09/08 14:35:25 INFO security.NMContainerTokenSecretManager: Rolling >>> master-key for container-tokens, got key with id 338249572 >>> 15/09/08 14:35:25 INFO security.NMTokenSecretManagerInNM: Rolling >>> master-key for container-tokens, got key with id -362725484 >>> 15/09/08 14:35:25 INFO nodemanager.NodeStatusUpdaterImpl: Registered >>> with ResourceManager as hadoopmapr5.brewingintel.com:55449 with total >>> resource of <memory:16384, vCores:4, disks:4.0> >>> 15/09/08 14:35:25 INFO nodemanager.NodeStatusUpdaterImpl: Notifying >>> ContainerManager to unblock new container-requests >>> >>> >>> Except of Myriad logs: >>> >>> /09/08 14:35:12 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:13 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:15 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:16 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:17 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:18 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:19 INFO handlers.StatusUpdateEventHandler: Status >>> Update for task: value: >>> "nm.medium.323f6664-11ca-477b-9e6e-41fb7547eacf" >>> | state: TASK_FAILED >>> 15/09/08 14:35:19 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:19 INFO scheduler.DownloadNMExecutorCLGenImpl: Using >>> remote distribution >>> 15/09/08 14:35:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: >>> Getting Hadoop distribution >>> from:maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz >>> 15/09/08 14:35:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: >>> Getting config from:http://myriad.marathon.mesos:8088/conf >>> 15/09/08 14:35:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: Slave >>> will execute command:sudo tar -zxpf hadoop-2.7.0.tar.gz && sudo chown >>> mapr . && cp conf hadoop-2.7.0/etc/hadoop/yarn-site.xml; export >>> YARN_HOME=hadoop-2.7.0; sudo -E -u mapr -H env >>> YARN_HOME="hadoop-2.7.0" >>> YARN_NODEMANAGER_OPTS="-Dnodemanager.resource.io-spindles=4.0 >>> -Dyarn.resourcemanager.hostname=myriad.marathon.mesos >>> >>> -Dyarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor >>> -Dnodemanager.resource.cpu-vcores=4 >>> -Dnodemanager.resource.memory-mb=16384 >>> -Dmyriad.yarn.nodemanager.address=0.0.0.0:31000 >>> -Dmyriad.yarn.nodemanager.localizer.address=0.0.0.0:31001 >>> -Dmyriad.yarn.nodemanager.webapp.address=0.0.0.0:31002 >>> -Dmyriad.mapreduce.shuffle.port=0.0.0.0:31003" $YARN_HOME/bin/yarn >>> nodemanager >>> 15/09/08 14:35:19 INFO handlers.ResourceOffersEventHandler: Launching >>> task: nm.medium.323f6664-11ca-477b-9e6e-41fb7547eacf using offer: >>> value: "20150907-111332-1660987584-5050-8033-O118248" >>> >>> 15/09/08 14:35:20 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> 15/09/08 14:35:21 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:21 INFO util.AbstractLivelinessMonitor: >>> Expired:hadoopmapr5.brewingintel.com:52878 Timed out after 2 secs >>> 15/09/08 14:35:21 INFO rmnode.RMNodeImpl: Deactivating Node >>> hadoopmapr5.brewingintel.com:52878 as it is now LOST >>> 15/09/08 14:35:21 INFO rmnode.RMNodeImpl: >>> hadoopmapr5.brewingintel.com:52878 Node Transitioned from RUNNING to >>> LOST >>> 15/09/08 14:35:21 INFO fair.FairScheduler: Removed node >>> hadoopmapr5.brewingintel.com:52878 cluster capacity: <memory:0, >>> vCores:0, disks:0.0> >>> 15/09/08 14:35:22 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:23 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:25 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:25 INFO util.RackResolver: Resolved >>> hadoopmapr5.brewingintel.com to /default-rack >>> 15/09/08 14:35:25 INFO resourcemanager.ResourceTrackerService: >>> NodeManager from node hadoopmapr5.brewingintel.com(cmPort: 55449 >>> httpPort: 8042) registered with capability: <memory:16384, vCores:4, >>> disks:4.0>, assigned nodeId hadoopmapr5.brewingintel.com:55449 >>> 15/09/08 14:35:25 INFO rmnode.RMNodeImpl: >>> hadoopmapr5.brewingintel.com:55449 Node Transitioned from NEW to >>> RUNNING >>> 15/09/08 14:35:25 INFO fair.FairScheduler: Added node >>> hadoopmapr5.brewingintel.com:55449 cluster capacity: <memory:16384, >>> vCores:4, disks:4.0> >>> 15/09/08 14:35:26 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:27 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:28 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:30 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:31 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:32 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:33 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:35 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:36 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:37 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:38 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:40 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:41 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:42 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:43 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:45 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:46 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:47 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:48 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:50 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:51 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:52 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:53 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:55 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:35:56 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:57 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:35:58 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:00 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:36:01 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:02 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:03 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:05 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:36:06 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:07 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:08 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:10 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:36:11 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:12 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:13 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:15 INFO handlers.ResourceOffersEventHandler: Received >>> offers 3 >>> 15/09/08 14:36:16 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:17 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:18 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:19 INFO handlers.StatusUpdateEventHandler: Status >>> Update for task: value: >>> "nm.medium.323f6664-11ca-477b-9e6e-41fb7547eacf" >>> | state: TASK_FAILED >>> 15/09/08 14:36:19 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:19 INFO scheduler.DownloadNMExecutorCLGenImpl: Using >>> remote distribution >>> 15/09/08 14:36:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: >>> Getting Hadoop distribution >>> from:maprfs:///mesos/myriad/hadoop-2.7.0.tar.gz >>> 15/09/08 14:36:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: >>> Getting config from:http://myriad.marathon.mesos:8088/conf >>> 15/09/08 14:36:19 INFO scheduler.TaskFactory$NMTaskFactoryImpl: Slave >>> will execute command:sudo tar -zxpf hadoop-2.7.0.tar.gz && sudo chown >>> mapr . && cp conf hadoop-2.7.0/etc/hadoop/yarn-site.xml; export >>> YARN_HOME=hadoop-2.7.0; sudo -E -u mapr -H env >>> YARN_HOME="hadoop-2.7.0" >>> YARN_NODEMANAGER_OPTS="-Dnodemanager.resource.io-spindles=4.0 >>> -Dyarn.resourcemanager.hostname=myriad.marathon.mesos >>> >>> -Dyarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor >>> -Dnodemanager.resource.cpu-vcores=4 >>> -Dnodemanager.resource.memory-mb=16384 >>> -Dmyriad.yarn.nodemanager.address=0.0.0.0:31000 >>> -Dmyriad.yarn.nodemanager.localizer.address=0.0.0.0:31001 >>> -Dmyriad.yarn.nodemanager.webapp.address=0.0.0.0:31002 >>> -Dmyriad.mapreduce.shuffle.port=0.0.0.0:31003" $YARN_HOME/bin/yarn >>> nodemanager >>> 15/09/08 14:36:19 INFO handlers.ResourceOffersEventHandler: Launching >>> task: nm.medium.323f6664-11ca-477b-9e6e-41fb7547eacf using offer: >>> value: "20150907-111332-1660987584-5050-8033-O118392" >>> >>> 15/09/08 14:36:20 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> 15/09/08 14:36:20 INFO util.AbstractLivelinessMonitor: >>> Expired:hadoopmapr5.brewingintel.com:55449 Timed out after 2 secs >>> 15/09/08 14:36:20 INFO rmnode.RMNodeImpl: Deactivating Node >>> hadoopmapr5.brewingintel.com:55449 as it is now LOST >>> 15/09/08 14:36:20 INFO rmnode.RMNodeImpl: >>> hadoopmapr5.brewingintel.com:55449 Node Transitioned from RUNNING to >>> LOST >>> 15/09/08 14:36:20 INFO fair.FairScheduler: Removed node >>> hadoopmapr5.brewingintel.com:55449 cluster capacity: <memory:0, >>> vCores:0, disks:0.0> >>> 15/09/08 14:36:22 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> 15/09/08 14:36:23 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:24 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:25 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> 15/09/08 14:36:25 INFO util.RackResolver: Resolved >>> hadoopmapr5.brewingintel.com to /default-rack >>> 15/09/08 14:36:25 INFO resourcemanager.ResourceTrackerService: >>> NodeManager from node hadoopmapr5.brewingintel.com(cmPort: 40378 >>> httpPort: 8042) registered with capability: <memory:16384, vCores:4, >>> disks:4.0>, assigned nodeId hadoopmapr5.brewingintel.com:40378 >>> 15/09/08 14:36:25 INFO rmnode.RMNodeImpl: >>> hadoopmapr5.brewingintel.com:40378 Node Transitioned from NEW to >>> RUNNING >>> 15/09/08 14:36:25 INFO fair.FairScheduler: Added node >>> hadoopmapr5.brewingintel.com:40378 cluster capacity: <memory:16384, >>> vCores:4, disks:4.0> >>> 15/09/08 14:36:27 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> 15/09/08 14:36:28 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:29 INFO handlers.ResourceOffersEventHandler: Received >>> offers 1 >>> 15/09/08 14:36:30 INFO handlers.ResourceOffersEventHandler: Received >>> offers 2 >>> >>> >>> >> >> >
