Re: Trying to build/run Myriad
To subscribe to the myriad dev list, send an email to: dev-subscr...@myriad.incubator.apache.org For other commands, and the full archive, see: http://mail-archives.apache.org/mod_mbox/incubator-myriad-dev/ On Mon, Aug 17, 2015 at 10:18 AM, Evgeniy Chupriyanov t...@tchu.ru wrote: 17 авг. 2015 г., в 9:46, yuliya Feldman yufeld...@yahoo.com написал(а): I second Adam here. Also - is your job failed to run or it is hanging - AM is not starting? It was stuck in SCHEDULED state and no AM was starting. And as Adam said - the problem was I had no NMs running in Mesos. Now I was able to start job and it finished successfully, though I had to start really large NM for small wordcount example. Eugene
Re: Trying to build/run Myriad
17 авг. 2015 г., в 9:46, yuliya Feldman yufeld...@yahoo.com написал(а): I second Adam here. Also - is your job failed to run or it is hanging - AM is not starting? It was stuck in SCHEDULED state and no AM was starting. And as Adam said - the problem was I had no NMs running in Mesos. Now I was able to start job and it finished successfully, though I had to start really large NM for small wordcount example. Eugene smime.p7s Description: S/MIME cryptographic signature
Re: Trying to build/run Myriad
17 авг. 2015 г., в 9:28, Adam Bordelon a...@mesosphere.io написал(а): ++ dev@myriad mailing list Evgeniy, Thanks for trying out Myriad. Sorry for my delayed response. Just got back from vacation. Have you flexed up to add any NodeManagers to your YARN cluster? See https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup Yes, that was the reason task was stuck in SCHEDULED state. Now I was able to run wordcount example successfully. But to make it complete, I had to flexup with 1 ‘large’ instance. I think its a bit overkill for such simple task with 2 map jobs. As I understand, YARN operates in terms of vCores, and each Job consumes at least 1 vCore + 1 for AM. When I run pure YARN, I can declare NodeManager has vCores = 2xpCores, so I can allow more parallel jobs which are not CPU hungry. With Myriad it looks like 1 pCore from Mesos == 1 YARN vCore and I have to allocate so many CPUs from Mesos, is it so? Is there a way to set how many YARN vCores maps to 1 Mesos CPU? Thanks for your help! Eugene P.S. Can I subscribe to dev@myriad list too? Couldn’t figure it out from Myriad Incubator page.. smime.p7s Description: S/MIME cryptographic signature
Re: Trying to build/run Myriad
++ dev@myriad mailing list Evgeniy, Thanks for trying out Myriad. Sorry for my delayed response. Just got back from vacation. Have you flexed up to add any NodeManagers to your YARN cluster? See https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup On Fri, Aug 7, 2015 at 10:56 AM, Evgeniy Chupriyanov t...@tchu.ru wrote: Hello, Adam! Sorry to contact you directly, but I couldn’t find where is the better place to ask questions about Myriad. I am working for small russian startup Cybertonica ( https://www.cyberonica.com) and we are useng Mesos as a base for our distributed system. Currently, I am exploring for better way to run Hadoop MR tasks on Mesos and Myriad looks very promising. I’ve tried to build Myriad/Mesos in local vagrant VM as described here: https://github.com/mesos/myriad/blob/phase1/docs/vagrant.md Everything went well and build/setup was successful. I can see Myriad framewrok registered in Mesos. Having that done, I have tried to small wordcount example like this: hduser@vagrant-ubuntu-trusty-64:/vagrant$ cat th.sh #!/bin/bash echo “I love UCSB” /tmp/file0 echo “Do you love UCSB?” /tmp/file1 hadoop fs -mkdir -p /tmp/foo/data hadoop fs -put /tmp/file? /tmp/foo/data yarn jar $YARN_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /tmp/foo/data /tmp/foo/out hadoop fs -ls /tmp/foo/out hadoop fs -cat /tmp/foo/out/part* hduser@vagrant-ubuntu-trusty-64:/vagrant$ But job fails to run and I see following message in resourcemanager log: 2015-08-07 16:51:35,635 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1 2015-08-07 16:51:36,533 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user hduser 2015-08-07 16:51:36,534 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1438966241928_0001 2015-08-07 16:51:36,536 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hduser IP=10.0.2.15OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1438966241928_0001 2015-08-07 16:51:36,537 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from NEW to NEW_SAVING 2015-08-07 16:51:36,541 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1438966241928_0001 2015-08-07 16:51:36,542 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from NEW_SAVING to SUBMITTED 2015-08-07 16:51:36,556 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Accepted application application_1438966241928_0001 from user: hduser, in queue: default, currently num of applications: 1 2015-08-07 16:51:36,582 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from SUBMITTED to ACCEPTED 2015-08-07 16:51:36,582 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1438966241928_0001_01 2015-08-07 16:51:36,587 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438966241928_0001_01 State change from NEW to SUBMITTED 2015-08-07 16:51:36,612 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1438966241928_0001_01 to scheduler from user: hduser 2015-08-07 16:51:36,614 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438966241928_0001_01 State change from SUBMITTED to SCHEDULED 2015-08-07 16:51:37,002 INFO com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler: Received offers 1 2015-08-07 16:51:37,002 INFO com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler: No pending tasks, declining all offers As I can see application gets status SCHEDULED and stays there. No any attempts are done to launch Mesos jobs are done. myriad scheduler reports it has no tasks pending. What I am doing wrong here? The only difference with the guide I did - I’m using Mesos 0.23.0, instead of 0.21 as described in guide (I changed version in build.gradle) Great thanks for a wonderful project. Sorry for any inconvenience Eugene
Re: Trying to build/run Myriad
I second Adam here. Also - is your job failed to run or it is hanging - AM is not starting? From: Adam Bordelon a...@mesosphere.io To: Evgeniy Chupriyanov t...@tchu.ru; dev@myriad.incubator.apache.org Sent: Sunday, August 16, 2015 11:28 PM Subject: Re: Trying to build/run Myriad ++ dev@myriad mailing list Evgeniy, Thanks for trying out Myriad. Sorry for my delayed response. Just got back from vacation. Have you flexed up to add any NodeManagers to your YARN cluster? See https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup On Fri, Aug 7, 2015 at 10:56 AM, Evgeniy Chupriyanov t...@tchu.ru wrote: Hello, Adam! Sorry to contact you directly, but I couldn’t find where is the better place to ask questions about Myriad. I am working for small russian startup Cybertonica ( https://www.cyberonica.com) and we are useng Mesos as a base for our distributed system. Currently, I am exploring for better way to run Hadoop MR tasks on Mesos and Myriad looks very promising. I’ve tried to build Myriad/Mesos in local vagrant VM as described here: https://github.com/mesos/myriad/blob/phase1/docs/vagrant.md Everything went well and build/setup was successful. I can see Myriad framewrok registered in Mesos. Having that done, I have tried to small wordcount example like this: hduser@vagrant-ubuntu-trusty-64:/vagrant$ cat th.sh #!/bin/bash echo “I love UCSB” /tmp/file0 echo “Do you love UCSB?” /tmp/file1 hadoop fs -mkdir -p /tmp/foo/data hadoop fs -put /tmp/file? /tmp/foo/data yarn jar $YARN_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /tmp/foo/data /tmp/foo/out hadoop fs -ls /tmp/foo/out hadoop fs -cat /tmp/foo/out/part* hduser@vagrant-ubuntu-trusty-64:/vagrant$ But job fails to run and I see following message in resourcemanager log: 2015-08-07 16:51:35,635 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1 2015-08-07 16:51:36,533 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user hduser 2015-08-07 16:51:36,534 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1438966241928_0001 2015-08-07 16:51:36,536 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hduser IP=10.0.2.15 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1438966241928_0001 2015-08-07 16:51:36,537 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from NEW to NEW_SAVING 2015-08-07 16:51:36,541 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1438966241928_0001 2015-08-07 16:51:36,542 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from NEW_SAVING to SUBMITTED 2015-08-07 16:51:36,556 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Accepted application application_1438966241928_0001 from user: hduser, in queue: default, currently num of applications: 1 2015-08-07 16:51:36,582 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1438966241928_0001 State change from SUBMITTED to ACCEPTED 2015-08-07 16:51:36,582 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1438966241928_0001_01 2015-08-07 16:51:36,587 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438966241928_0001_01 State change from NEW to SUBMITTED 2015-08-07 16:51:36,612 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1438966241928_0001_01 to scheduler from user: hduser 2015-08-07 16:51:36,614 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1438966241928_0001_01 State change from SUBMITTED to SCHEDULED 2015-08-07 16:51:37,002 INFO com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler: Received offers 1 2015-08-07 16:51:37,002 INFO com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler: No pending tasks, declining all offers As I can see application gets status SCHEDULED and stays there. No any attempts are done to launch Mesos jobs are done. myriad scheduler reports it has no tasks pending. What I am doing wrong here? The only difference with the guide I did - I’m using Mesos 0.23.0, instead of 0.21 as described in guide (I changed version in build.gradle) Great thanks for a wonderful project. Sorry for any inconvenience Eugene
