Re: proposal to change time for Myriad biweekly meeting

[yuliya Feldman]

Good question. 
Adam - could you chime in - since you are the meeting organizer? 
  From: Kannan Rajah kra...@maprtech.com
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, May 5, 2015 7:06 PM
 Subject: Re: proposal to change time for Myriad biweekly meeting
   
So is this change of time for tomorrow or only from next week?

Kannan


On May 5, 2015 4:07 PM, Brandon Gulla gulla.bran...@gmail.com wrote:

 East coaster here. 9am pst is good with me.
 On May 5, 2015 4:26 PM, Danese Cooper dan...@gmail.com wrote:

  Fine by me.
 
 
   On May 4, 2015, at 11:14 AM, yuliya Feldman
 yufeld...@yahoo.com.INVALID
  wrote:
  
   Hello guys,
   I wanted to propose to change time for Myriad biweekly hangouts.Could
 we
  do 9 AM to 10 AM instead?
   As far as I remember we have people form China and for them it is very
  late and of course out of selfish reasons - I have recurring meeting I
 can
  not change from 10 to 11 AM.
   Thanks,Yuliya
 



  

Re: Hangout

[yuliya Feldman]

Wonder if the link is still valid
  From: Darin Johnson dbjohnson1...@gmail.com
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, July 1, 2015 8:58 AM
 Subject: Hangout
   
Are we doing a hangout today?
Cheers,
Darin


   

Re: Hangout

[yuliya Feldman]

Is there is a different link or what not - I don't think people can connect.
Thanks,Yuliya
  From: Darin Johnson dbjohnson1...@gmail.com
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, July 1, 2015 8:58 AM
 Subject: Hangout
   
Are we doing a hangout today?
Cheers,
Darin


   

Re: myriad scheduler startup with HDP2.7

[yuliya Feldman]

This method is part of JsonFactory class which is part of jackson-core jar
See if you have some other jars on the classpath (different versions) that 
precede jackson-core-2.5.1.jar
  From: Bill Sparks jspa...@cray.com
 To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org 
 Sent: Wednesday, August 19, 2015 7:08 AM
 Subject: myriad scheduler startup with HDP2.7
   
I'm sure this is been resolved, but I've been triaging why I'm getting the 
following error on resourcemanager startup. Everything on the configuration 
side looks correct, but I must have missed something.



2015-08-19 08:53:04,718 FATAL 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
ResourceManager

java.lang.NoSuchMethodError: 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/InputStream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream;

at 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:299)

at 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14)

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2011)

at com.ebay.myriad.Main.initialize(Main.java:70)

at 
com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:32)

at 
com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76)

at 
com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:572)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:972)

at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:259)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1202)


I have placed all the myriad jar in the hadoop-yarn/lib directory and the 
classpath reflect that.


cp /tmp/myriad/myriad-scheduler/build/libs/* 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib

cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar 
/usr/libexec/mesos/


[root@nid00037 myriad]# su - yarn

-bash-4.1$ yarn classpath

/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2.3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0-2557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/hadoop-yarn-client/lib/*



ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries


-bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad*

-rw-r--r-- 1 root root  3456 Aug 19 08:50 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar

-rw-r--r-- 1 root root 950687 Aug 19 08:50 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar

and


-bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson*

-rw-r--r-- 1 root root  39817 Aug 17 18:32 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar

-rw-r--r-- 1 root root  192699 Jul 14 08:22 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar

-rw-r--r-- 1 root root  229860 Aug 17 18:32 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar

-rw-r--r-- 1 root root  232248 Jul 14 08:22 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar

-rw-r--r-- 1 root root 1138921 Aug 17 18:32 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar

-rw-r--r-- 1 root root  321751 Aug 17 18:32 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar

-rw-r--r-- 1 root root  18336 Jul 14 08:22 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar

-rw-r--r-- 1 root root  780664 Jul 14 08:22 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar

-rw-r--r-- 1 root root  27084 Jul 14 08:22 
/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-xc-1.9.13.jar



-bash-4.1$ jar tf 
/usr/hdp/current/hadoop-yarn-client/lib/jackson-dataformat-yaml-2.5.1.jar  | 
grep YAMLFactory

Re: myriad scheduler startup with HDP2.7

[yuliya Feldman]

as you can imagine you need matching versions of jackson* jars otherwise you 
might get into issues of incompatibility
Easiest for you at the moment to put myriad dependency jars on classpath before 
others
  From: Bill Sparks jspa...@cray.com
 To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org 
Cc: yuliya Feldman yufeld...@yahoo.com 
 Sent: Wednesday, August 19, 2015 10:48 AM
 Subject: Re: myriad scheduler startup with HDP2.7
   
Well thats the point, I do have 2.2.3 installed as that's the version
shipped with HDP 2.3 and that gets loaded first in the classpath for YARN
resourcemanager. 

I guess I have three alternatives.

1) build myriad using 2.2.3, thus matching the HDP installed jar's
2) replace the HDP version with 2.5.1, not sure what's that going to do
for HDP compatibility
3) prepend a new classpath for yarn resourcemanager to pick up myriad
versioned jars first.

-- 
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.







On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote:

Myriad should be using jackson 2.5.1
https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6
743/myriad-scheduler/build.gradle#L13
Double-check your build.gradle, and make sure you don't have a jackson
2.2.3 preinstalled somewhere else on your system

On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote:

 Odd the class path reported in the yarn log contains jackson-core-2.2.3
 and not 2.5.1. Is there a way to build myriad to match the version
 supported by HDP - that being 2.2.3 ?


 --
 Jonathan (Bill) Sparks
 Software Architecture
 Cray Inc.





 On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote:

 Thanks I'll check..
 
 --
 Jonathan (Bill) Sparks
 Software Architecture
 Cray Inc.
 
 
 
 
 
 On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID
 wrote:
 
 This method is part of JsonFactory class which is part of jackson-core
 jar
 See if you have some other jars on the classpath (different versions)
 that precede jackson-core-2.5.1.jar
       From: Bill Sparks jspa...@cray.com
  To: dev@myriad.incubator.apache.org
dev@myriad.incubator.apache.org
  Sent: Wednesday, August 19, 2015 7:08 AM
  Subject: myriad scheduler startup with HDP2.7
 
 I'm sure this is been resolved, but I've been triaging why I'm getting
 the following error on resourcemanager startup. Everything on the
 configuration side looks correct, but I must have missed something.
 
 
 
 2015-08-19 08:53:04,718 FATAL
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
 starting ResourceManager
 
 java.lang.NoSuchMethodError:
 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In
pu
 t
 Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream;
 
 at
 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact
or
 y
 .java:299)
 
 at
 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact
or
 y
 .java:14)
 
 at
 
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java
:2
 0
 11)
 
 at com.ebay.myriad.Main.initialize(Main.java:70)
 
 at
 
com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep
to
 r
 .init(MyriadInitializationInterceptor.java:32)
 
 at
 
com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co
mp
 o
 siteInterceptor.java:76)
 
 at
 
com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa
ir
 S
 cheduler.java:50)
 
 at
 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163
)
 
 at
 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService
.j
 a
 va:107)
 
 at
 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS
er
 v
 ices.serviceInit(ResourceManager.java:572)
 
 at
 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163
)
 
 at
 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd
In
 i
 tActiveServices(ResourceManager.java:972)
 
 at
 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn
it
 (
 ResourceManager.java:259)
 
 at
 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163
)
 
 at
 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso
ur
 c
 eManager.java:1202)
 
 
 I have placed all the myriad jar in the hadoop-yarn/lib directory and
the
 classpath reflect that.
 
 
 cp /tmp/myriad/myriad-scheduler/build/libs/*
 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib
 
 cp
 
/tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j
ar
 /usr/libexec/mesos/
 
 
 [root@nid00037 myriad]# su - yarn
 
 -bash-4.1$ yarn classpath
 
 
/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us
r/
 h
 
dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp
/2
 .
 
3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.
3.
 0
 
.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h
dp
 /
 
2.3.0.0-2557/hadoop-yarn/lib

Re: Myriad 0.1 release scope

[yuliya Feldman]

mesos/myriad is the right one so far
  From: John Omernik j...@omernik.com
 To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com 
 Sent: Tuesday, August 18, 2015 1:44 PM
 Subject: Re: Myriad 0.1 release scope
   
(So if I clone that repo, am I cloning the right one?)



On Tue, Aug 18, 2015 at 3:43 PM, John Omernik j...@omernik.com wrote:

 Ok, I was going off
 https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md

 I will try it.

 John

 On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman 
 yufeld...@yahoo.com.invalid wrote:

 You actually do not need to rebuild even today - just keep this file in
 hadoop config directory that is on the classpath: like .../etc/hadoop
      From: John Omernik j...@omernik.com
  To: dev@myriad.incubator.apache.org
  Sent: Tuesday, August 18, 2015 1:35 PM
  Subject: Re: Myriad 0.1 release scope

 On the release scope, will having the myriad configuration file exist
 outside the jar (i.e. you can change configuration without rebuilding) be
 part of the .1 release scope?



 On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com
 wrote:

  Hello All,
 
   I've merged the FGS changes into phase1. Built and tested both coarse
  grained scaling and fine grained scaling, UI on a 4 node cluster.
 
   If anyone finds things are not working as expected, please let me know.
 
  Thanks,
  Santosh
 
  On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com
 
  wrote:
 
   Hello guys,
  
   I propose merging FGS into phase1. As I said before, I think it's at a
   point where the functionality works reasonably well.
   Any future improvements/fixes/UI changes can be done via separate
 JIRAs.
  
   Unless there are any major concerns, I'd like to merge FGS into phase1
   *EOD Monday* (PDT).
  
   Thanks,
   Santosh
  
   On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella 
 smare...@maprtech.com
   wrote:
  
   I feel FGS is very close to making it into 0.1. PR 116 addresses
 moving
   to hadoop 2.7 and making FGS and CGS coexist. This PR was recently
  reviewed
   by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and
  2.7.x
   clusters and it seemed to have worked as expected. Unless there are
 more
   reviews/feedback, it can be merged into issue_14. Once PR 116 is
 merged
   into issue_14, issue_14 can be merged into phase1.
  
   Thanks,
   Santosh
  
   On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io
  wrote:
  
   We do have a JIRA 0.1.0 fix version field, but none of our issues
 use
   it
   yet.
   I think the goal was just to take what we have and make it work
 under
   Apache infrastructure, then vote on that for 0.1.0.
   Although other features like HA or FGS would be great, let's try to
 get
   our
   first Apache release out ASAP.
   We can create 0.1.1 or 0.2.0 fix versions for subsequent releases
 with
   other issues/features. Roadmap would be great.
   (I'm just summarizing what we discussed a month or two ago. Feel
 free
  to
   correct me or disagree with this approach.)
  
   On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade 
   swapnil.daing...@gmail.com
wrote:
  
Hi all,
   
Was wondering what would be the scope for the Myriad 0.1 release.
It would be nice to have a roadmap page somewhere and target
features to releases (JIRA 'fix version' field perhaps)
   
Regards
Swapnil
   
  
  
  
  
 








  

making contributor

[yuliya Feldman]

Hello guys,
Could you make me contributor to Myriad, so I could assign JIRAs to myself?
Thanks,Yuliya

Licensing for new files

[yuliya Feldman]

Hello here,
What licensing should we use for newly created files in Myriad source code - 
pure Apache?
Thanks,Yuliya

Re: Complete Myriad HA implementation

[yuliya Feldman]

Cool!!!  Way to go
  From: Swapnil Daingade swapnil.daing...@gmail.com
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, August 19, 2015 8:52 PM
 Subject: Complete Myriad HA implementation
   
Hi All,

I have updated my pull request with the complete Myriad HA implementation
rebased
on top of the FGS changes here

https://github.com/mesos/myriad/pull/123

I am planning to send out another email with details on how to configure it.

Regards
Swapnil


  

Re: Trying to build/run Myriad

[yuliya Feldman]

I second Adam here.
Also - is your job failed to run or it is hanging  - AM is not starting?
  From: Adam Bordelon a...@mesosphere.io
 To: Evgeniy Chupriyanov t...@tchu.ru; dev@myriad.incubator.apache.org 
 Sent: Sunday, August 16, 2015 11:28 PM
 Subject: Re: Trying to build/run Myriad
   
++ dev@myriad mailing list

Evgeniy,

Thanks for trying out Myriad. Sorry for my delayed response. Just got back
from vacation.
Have you flexed up to add any NodeManagers to your YARN cluster?
See
https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup

On Fri, Aug 7, 2015 at 10:56 AM, Evgeniy Chupriyanov t...@tchu.ru wrote:

 Hello, Adam!

 Sorry to contact you directly, but I couldn’t find where is the better
 place to ask questions about Myriad.

 I am working for small russian startup Cybertonica (
 https://www.cyberonica.com) and we are useng Mesos as a base for our
 distributed system.
 Currently, I am exploring for better way to run Hadoop MR tasks on Mesos
 and Myriad looks very promising.

 I’ve tried to build Myriad/Mesos in local vagrant VM as described here:
 https://github.com/mesos/myriad/blob/phase1/docs/vagrant.md
 Everything went well and build/setup was successful. I can see Myriad
 framewrok registered in Mesos.

 Having that done, I have tried to small wordcount example like this:
 hduser@vagrant-ubuntu-trusty-64:/vagrant$ cat th.sh
 #!/bin/bash

 echo “I love UCSB”  /tmp/file0
 echo “Do you love UCSB?”  /tmp/file1
 hadoop fs -mkdir -p /tmp/foo/data
 hadoop fs -put /tmp/file? /tmp/foo/data
 yarn jar
 $YARN_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar
 wordcount /tmp/foo/data /tmp/foo/out
 hadoop fs -ls /tmp/foo/out
 hadoop fs -cat /tmp/foo/out/part*
 hduser@vagrant-ubuntu-trusty-64:/vagrant$

 But job fails to run and I see following message in resourcemanager log:
 2015-08-07 16:51:35,635 INFO
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated
 new applicationId: 1
 2015-08-07 16:51:36,533 INFO
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
 with id 1 submitted by user hduser
 2015-08-07 16:51:36,534 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing
 application with id application_1438966241928_0001
 2015-08-07 16:51:36,536 INFO
 org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hduser
 IP=10.0.2.15    OPERATION=Submit Application Request
 TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1438966241928_0001
 2015-08-07 16:51:36,537 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
 application_1438966241928_0001 State change from NEW to NEW_SAVING
 2015-08-07 16:51:36,541 INFO
 org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
 Storing info for app: application_1438966241928_0001
 2015-08-07 16:51:36,542 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
 application_1438966241928_0001 State change from NEW_SAVING to SUBMITTED
 2015-08-07 16:51:36,556 INFO
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
 Accepted application application_1438966241928_0001 from user: hduser, in
 queue: default, currently num of applications: 1
 2015-08-07 16:51:36,582 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
 application_1438966241928_0001 State change from SUBMITTED to ACCEPTED
 2015-08-07 16:51:36,582 INFO
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
 Registering app attempt : appattempt_1438966241928_0001_01
 2015-08-07 16:51:36,587 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
 appattempt_1438966241928_0001_01 State change from NEW to SUBMITTED
 2015-08-07 16:51:36,612 INFO
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
 Added Application Attempt appattempt_1438966241928_0001_01 to scheduler
 from user: hduser
 2015-08-07 16:51:36,614 INFO
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
 appattempt_1438966241928_0001_01 State change from SUBMITTED to
 SCHEDULED
 2015-08-07 16:51:37,002 INFO
 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
 Received offers 1
 2015-08-07 16:51:37,002 INFO
 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler: No
 pending tasks, declining all offers

 As I can see application gets status SCHEDULED and stays there. No any
 attempts are done to launch Mesos jobs are done. myriad scheduler reports
 it has no tasks pending.
 What I am doing wrong here? The only difference with the guide I did - I’m
 using Mesos 0.23.0, instead of 0.21 as described in guide (I changed
 version in build.gradle)


 Great thanks for a wonderful project.
 Sorry for any inconvenience

 Eugene







  

Re: JobHistoryServer running as MyriadTask

[yuliya Feldman]

OK
Thank you for  your feedback. 
This approach does not prohibit you from continuing running JHS, TLS, others 
from Marathon(s) as you do it today. It is in addition and not a replacement.
I would like to hear from more people before considering this feature a no-op.
Thanks,Yuliya
  From: Darin Johnson dbjohnson1...@gmail.com
 To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com 
 Sent: Thursday, July 23, 2015 9:19 AM
 Subject: Re: JobHistoryServer running as MyriadTask
   
I was planning to run the resource manager (hopefully two via marathon once
ha is done) and the history server via marathon.  The three cons listed
don't seem too severe.  Addressing the cons:



  1.

  In case of multiple Myriad clusters and single Marathon instance
  services will rely on FIFO within Marathon to get resources


  Not as bad as you'd think, FIFO is pretty safe for long running
  resources. You're not going to be hit that hard. Added benefit, the TLS,
  JHS, RM are running on a Marathon framework with a potentially better role.
  I do this already for one instance of Marathon for services I want to give
  high priority to. In general, I'd be inclined to leave a NM to a lower
  priority role. I want my M/R jobs to eventually run, I want the TLS to be
  running not eventually running.


  2.

  Some env. variables, settings that Myriad already has like RM host/ports
  may not be easily obtainable/available from outside of Myriad framework(s).


  If running the RM in Marathon the relevant configs should be obtainable
  via a URI already. I'm caveating the on the fact I already rely on a DNS
  style service discovery (Mesos-DNS and Consul-Mesos both work fine for
  this).


  3. When tearing down Myriad cluster those services will have to be
  stopped separately.


Possibly a bug, I might want to inspect the TLS and JHS after shutting down
YARN, or I may just want to update the RM (Myriad is actively being
developed).


Might be worth reinvestigating at a later date. One pro I could see is by
setting the JHS and TLS in Myriad, it may be easier to assign them dynamic
ports via Mesos, but this would be a lot of added complexity for updating
the configs of running NMs if a JHS or TLS was reassigned. I don't think
this is something to address right now as it adds complexity which I think
can be avoided.

Darin





Hello here,
As we discussed last meeting I have wrote a small doc on pros/cons and
design suggestions for running JHS as MyriadTask.

https://docs.google.com/document/d/1TGGU2bn1cVPSmHeriNrtkAEhdvsRKcFtuWRaaGp9bSY/edit?usp=sharing

Also commented on: https://github.com/mesos/myriad/issues/60
Comments, suggestions, objections are welcome.
I am trying to wrap up design this week, so I could go one way or the other
starting next week, as I will be gone for 2 weeks, starting Aug 1st.
Thanks,Yuliya


  

Re: Establish versions to target for first incubator release?

[yuliya Feldman]

+1 on 2.7
  From: Jim Klucar klu...@gmail.com
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, July 14, 2015 5:36 PM
 Subject: Establish versions to target for first incubator release?
   
The FGS discussion made me wonder if we've put a line in the sand about
what versions of YARN and Mesos we're going to target for the first Myriad
incubator release. Might be nice to start getting some kind of mileage on
specific versions. Perhaps 0.23 and 2.7? Is this premature?


   

Re: Question about build

[Yuliya Feldman]

Yes, 

If you rely just on fat jar. For dev it is great tool, but for production may 
not be so much.



On Jul 19, 2015, at 8:46 PM, Ken Sipe k...@mesosphere.io wrote:

 the point of a fat jar is to not have jars “outside”… it would make sense 
 that would cause challenges.   We have a fat jar now because 1) that is the 
 way it was original dev and 2) the ability to distribute all things the 
 needed by the executor in a runtime jar.Fat jars work great when they are 
 all inclusive, all stand alone.   if in a shared class path space then bad 
 things are sure to happen:)
 
 ken
 On Jul 19, 2015, at 10:08 PM, Yuliya Feldman yufeld...@yahoo.com.INVALID 
 wrote:
 
 I am wondering about usage of fat jar in principal - fat jar unless it has 
 the same versions of jars inside fat one and outside can create issues. slf 
 jars , mesos jars
 
 What is your experience? Ours so far was quite negative
 
 
 
 On Jul 19, 2015, at 6:46 PM, Ken Sipe k...@mesosphere.io wrote:
 
 
 On Jul 17, 2015, at 5:51 PM, Adam Bordelon a...@mesosphere.io wrote:
 
 Ken, Swapnil is looking at turning the executor into a YARN NM AuxService
 anyway, so it might end up without a main too. Then we wouldn't need the
 application plugin at all?
 
 actually I don’t think we need the application plugin right now.   I think 
 the capsule uses the application property for the main for configuration.  
 We can go without it I believe.  When we make the executor a NM AuxService, 
 we may need to look at new packaging.   I believe the capsule is a 
 “runnable” fat jar and we will need just a standard fat jar at that point.  
  I’ll look into it.
 
 ken
 
 On Fri, Jul 17, 2015 at 3:31 PM, Ken Sipe k...@mesosphere.io 
 mailto:k...@mesosphere.io wrote:
 
 Hey Patrick,
 
 I was the one that refactored myriad in to a multi-project structure.   I
 took what was originally there and tried to maintain the semantics into 
 the
 new structure (meaning it was my intention to not change anything other
 than the structure).  At first glance this appears to be a miss on my
 part.   There is no main class in the scheduler.  The scheduler is a YARN
 resource manager.
 
 Looking it over in more detail, I’m not sure we even need the application
 plugin for the executor.  It is required for the building of the capsule,
 but that could be declared in the capsule instead of using the
 configuration of the application.  It is worth more investigation.
 
 As for long, there are no other projects other than the executor that need
 the application plugin.  I have refactored this and have submitted it as
 PR-120:  https://github.com/mesos/myriad/pull/120 
 https://github.com/mesos/myriad/pull/120 
 https://github.com/mesos/myriad/pull/120
 
 Thanks for helping to identify this.
 
 Ken
 
 On Jul 17, 2015, at 3:50 PM, Patrick Wong pw...@maprtech.com wrote:
 
 Hello Myriad Team,
 
 I noticed that the application plugin is applied to all subprojects. Does
 this mean that you are supposed to be able to run gradle installDist to
 get a working local installation for each subproject, and gradle run to
 run them?
 
 --
 Patrick Wong
 510.386.7205
 
 mapr.com
 
 

Re: Question for Mesos gurus

[yuliya Feldman]

Thank you Adam
When you say:but before you upgrade Mesos to 0.23, you should upgrade
your scheduler (and executor) libmesos to 0.22.x

Do you mean - recompile?
Does this sentence from link with upgrade instructions you provided means the 
same? Rebuild and install any modules so that upgraded masters/slaves can 
use them
Thanks,Yuliya  From: Adam Bordelon a...@mesosphere.io
 To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com 
 Sent: Tuesday, August 25, 2015 10:06 AM
 Subject: Re: Question for Mesos gurus
   
Mesos guarantees forward and backward compatibility by one minor version.
It is expected that you upgrade the entire cluster to one consecutive
version before upgrading any component to the next. So, if your scheduler
jar's libmesos is from 0.21.x, you can upgrade your Mesos master/agents to
0.22.x safely, but before you upgrade Mesos to 0.23, you should upgrade
your scheduler (and executor) libmesos to 0.22.x. See
http://mesos.apache.org/documentation/latest/upgrades/ for other special
notes and recommended upgrade order.
Once we reach Mesos 1.0 (when the new HTTP API stabilizes), then we'll have
stronger guarantees about version compatibility within a major version.



On Tue, Aug 25, 2015 at 8:33 AM, yuliya Feldman yufeld...@yahoo.com.invalid
 wrote:

 Hello guys,
 I wonder about compatibility of Mesos protobuf for Myriad usage.
 If I complied Myriad with Mesos version 0.22.1/0.21.1 but on the cluster I
 have Mesos 0.23 - is it suppose to be compatible?
 Yesterday our guys came across an exception(see below).
 When switching jars to mesos-0.21.1 issue went away.
 Thanks,Yuliya
 15/08/24 10:57:40 INFO scheduler.TaskFactory$NMTaskFactoryImpl:
 yarn.resourcemanager.hostname is set to rm.marathon.mesos via
 YARN_RESOURCEMANAGER_OPTS. Passing it into YARN_NODEMANAGER_OPTS.
 Aug 24, 2015 10:57:40 AM com.lmax.disruptor.FatalExceptionHandler
 handleEventException
 SEVERE: Exception processing: 1
 com.ebay.myriad.scheduler.event.ResourceOffersEvent@74a1e0a5
 java.lang.NoSuchMethodError:

 org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
      at

 com.ebay.myriad.scheduler.TaskFactory$NMTaskFactoryImpl.createTask(TaskFactory.java:310)
      at

 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:98)
      at

 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:55)
      at
 com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
      at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

 15/08/24 10:57:40 ERROR yarn.YarnUncaughtExceptionHandler: Thread
 Thread[pool-2-thread-3,5,main] threw an Exception.
 java.lang.RuntimeException: java.lang.NoSuchMethodError:

 org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
      at

 com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
      at
 com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
      at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.NoSuchMethodError:

 org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
      at

 com.ebay.myriad.scheduler.TaskFactory$NMTaskFactoryImpl.createTask(TaskFactory.java:310)
      at

 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:98)
      at

 com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:55)
      at
 com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
      ... 3 more



  

Question for Mesos gurus

[yuliya Feldman]

Hello guys,
I wonder about compatibility of Mesos protobuf for Myriad usage.
If I complied Myriad with Mesos version 0.22.1/0.21.1 but on the cluster I have 
Mesos 0.23 - is it suppose to be compatible?
Yesterday our guys came across an exception(see below).
When switching jars to mesos-0.21.1 issue went away.
Thanks,Yuliya
15/08/24 10:57:40 INFO scheduler.TaskFactory$NMTaskFactoryImpl:
yarn.resourcemanager.hostname is set to rm.marathon.mesos via
YARN_RESOURCEMANAGER_OPTS. Passing it into YARN_NODEMANAGER_OPTS.
Aug 24, 2015 10:57:40 AM com.lmax.disruptor.FatalExceptionHandler
handleEventException
SEVERE: Exception processing: 1
com.ebay.myriad.scheduler.event.ResourceOffersEvent@74a1e0a5
java.lang.NoSuchMethodError:
org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
 at
com.ebay.myriad.scheduler.TaskFactory$NMTaskFactoryImpl.createTask(TaskFactory.java:310)
 at
com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:98)
 at
com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:55)
 at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

15/08/24 10:57:40 ERROR yarn.YarnUncaughtExceptionHandler: Thread
Thread[pool-2-thread-3,5,main] threw an Exception.
java.lang.RuntimeException: java.lang.NoSuchMethodError:
org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
 at
com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
 at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:
org.apache.mesos.Protos$TaskInfo$Builder.setData(Lcom/google/protobuf/ByteString;)Lorg/apache/mesos/Protos$TaskInfo$Builder;
 at
com.ebay.myriad.scheduler.TaskFactory$NMTaskFactoryImpl.createTask(TaskFactory.java:310)
 at
com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:98)
 at
com.ebay.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:55)
 at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
 ... 3 more

Re: JIRA work for 0.1.0

[yuliya Feldman]

Do you have other suggestions?
I understand that there might be clashes even with randomization, but it is 
much lower chance then consistently getting the same port w/o randomization.
Thanks,Yuliya  From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman 
<yufeld...@yahoo.com> 
 Sent: Friday, October 23, 2015 9:57 AM
 Subject: Re: JIRA work for 0.1.0
   
On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
outside of Mesos as other frameworks might select that port the probability
of that happening is 1-(number of ports requested by other
frameworks)/(number of ports in use).  This could get near 50% and cause
frameworks that are being good citizens to crash.

I'm also not convinced randomizing the port is in fact the correct fix for
this in the long term, as there is still a non-zero chance you'll get that
port again.

Darin





On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:

> Great list.
> I would include MYRIAD-160 to the list - I am working on that one. Of
> course as a workaround we could not use Mesos ports and let NM ports
> randomization kick in.I also almost done with MYRIAD-148 - it was really
> tricky. Should submit PR tonight.
> Thanks,Yuliya
>      From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>
>  Sent: Thursday, October 22, 2015 8:29 PM
>  Subject: Re: JIRA work for 0.1.0
>
> I think this sounds good about right.  A few Jim marked were new features,
> gotta leave something for the 0.2.0 release :).
>
>
>
>
> On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <smare...@maprtech.com>
> wrote:
>
> > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > There were 19 of them. I moved a few out. We are currently at 14.
> >
> > However, IMO the show stoppers are really the following:
> >
> > MYRIAD-43 Replace com.ebay namespace with org.apache
> > MYRIAD-44 Prepare for 0.1.0 release
> > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > MYRIAD-114 Automatic dashboard building
> > MYRIAD-145 Document Myriad Release Process
> > MYRIAD-150 Update NOTICE file
> > MYRIAD-159 Change default mesos version to 0.24
> >
> > Unless anyone thinks there are other JIRAs that are show stoppers, I
> think
> > we should stick to the above list
> > and cut a RC as soon as we address the above.
> >
> > **I'm positive the above can be fixed by early next week (10/27) and we
> can
> > have a RC out for voting mid next week (10/28).**
> >
> > If we can't fix the above JIRAs in time or if new ones come up as "show
> > stoppers", we will have a revised date.
> > And, of course, more fixes are welcome, as long as they can be merged
> > before 10/27.
> >
> > Just to let everyone know about the Apache release process (@Adam, feel
> > free to chime in):
> >  - Apache requires that a RC be put out for voting on
> dev@myriad.incubator
> > for 72 hrs or until 3 binding +1s and no binding -1s,
> >  - followed by a similar voting round on general@incubator.
> >
> > Now to run the last mile..!
> >
> > Cheers,
> > Santosh
> >
> > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon <a...@mesosphere.io>
> wrote:
> >
> > > Keep in mind that Santosh (as Release Manager) has final authority over
> > > pushing things out of 0.1.0.
> > >
> > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> JIRAs:
> > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > >
> > > Maybe we should create a dashboard too, like
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > >
> > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar <klu...@gmail.com> wrote:
> > >
> > > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > > 0.1.0
> > > > release. Some of them are really low hanging fruit, so I think we can
> > > knock
> > > > most of these out. As Adam said in the meeting, we'd rather have too
> > many
> > > > flagged for the release and pair down rather than miss some. To that
> > end,
> > > > please go through the tickets that are still unmarked and set the Fix
> > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> the
> > > > release.
> > > >
> > > > Jim
> > > >
> > >
> >
>
>
>
>


  

Re: JIRA work for 0.1.0

[yuliya Feldman]

Great list.
I would include MYRIAD-160 to the list - I am working on that one. Of course as 
a workaround we could not use Mesos ports and let NM ports randomization kick 
in.I also almost done with MYRIAD-148 - it was really tricky. Should submit PR 
tonight.
Thanks,Yuliya
  From: Darin Johnson 
 To: Dev  
 Sent: Thursday, October 22, 2015 8:29 PM
 Subject: Re: JIRA work for 0.1.0
   
I think this sounds good about right.  A few Jim marked were new features,
gotta leave something for the 0.2.0 release :).




On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella 
wrote:

> I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> There were 19 of them. I moved a few out. We are currently at 14.
>
> However, IMO the show stoppers are really the following:
>
> MYRIAD-43 Replace com.ebay namespace with org.apache
> MYRIAD-44 Prepare for 0.1.0 release
> MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> MYRIAD-114 Automatic dashboard building
> MYRIAD-145 Document Myriad Release Process
> MYRIAD-150 Update NOTICE file
> MYRIAD-159 Change default mesos version to 0.24
>
> Unless anyone thinks there are other JIRAs that are show stoppers, I think
> we should stick to the above list
> and cut a RC as soon as we address the above.
>
> **I'm positive the above can be fixed by early next week (10/27) and we can
> have a RC out for voting mid next week (10/28).**
>
> If we can't fix the above JIRAs in time or if new ones come up as "show
> stoppers", we will have a revised date.
> And, of course, more fixes are welcome, as long as they can be merged
> before 10/27.
>
> Just to let everyone know about the Apache release process (@Adam, feel
> free to chime in):
>  - Apache requires that a RC be put out for voting on dev@myriad.incubator
> for 72 hrs or until 3 binding +1s and no binding -1s,
>  - followed by a similar voting round on general@incubator.
>
> Now to run the last mile..!
>
> Cheers,
> Santosh
>
> On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon  wrote:
>
> > Keep in mind that Santosh (as Release Manager) has final authority over
> > pushing things out of 0.1.0.
> >
> > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad JIRAs:
> > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> >
> > Maybe we should create a dashboard too, like
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> >
> > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar  wrote:
> >
> > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > 0.1.0
> > > release. Some of them are really low hanging fruit, so I think we can
> > knock
> > > most of these out. As Adam said in the meeting, we'd rather have too
> many
> > > flagged for the release and pair down rather than miss some. To that
> end,
> > > please go through the tickets that are still unmarked and set the Fix
> > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by the
> > > release.
> > >
> > > Jim
> > >
> >
>


  

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 0)

[yuliya Feldman]

We have found an issue with MYRIAD-162
Executor values (cpu, mem) counted twice.  It is not a big issue, since it is 
only adding additional 0.2 CPU and some amount of memory.
I have a fix for this - will submit PR shortly.
Other then that 
+1 (non-binding)
Checked signatureDownloaded tar.gzBuilt from tar.gzDeployed on one node 
clusterTested NM/JHS flexup/downRun M/R job successfully
Thanks,Yuliya

  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, November 9, 2015 5:05 PM
 Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 
0)
   
Changed Myriad version to 0.1.0 in the code and github documentation.
 PR: https://github.com/apache/incubator-myriad/pull/37

Updated the Wiki documentation with hadoop version changed to 2.7.0 and
myriad version changed to 0.1.0.

@Adam, et al., any other feedback on rc0 ?

Unless more code changes are anticipated, I can cut a rc1 once the
outstanding PRs are reviewed.



Santosh



On Mon, Nov 9, 2015 at 1:29 AM, Adam Bordelon  wrote:

> Do we need to rev the version on the executor jars up to 0.1.0?
> Is there anywhere else we need to bump the version?
>
> $ grep -rin "myriad.*[^ \[\(0-9]0" . |grep -v Apache
> ./docker/Dockerfile:34:ADD /libs/myriad-executor-runnable-0.0.1.jar
> /usr/local/libexec/mesos/
> ./docker/ResourceManager.dockerfile:30:ADD
> /libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/
> ./docker/build-myriad.sh:40:cp -rf
> ../myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar libs/
> ./docs/vagrant.md:100
> :/vagrant/myriad-scheduler/build/libs/myriad-executor-0.0.1.jar
> ./docs/vagrant.md:107:cp
> /vagrant/myriad-executor/build/libs/myriad-executor-0.0.1.jar
> $YARN_HOME/share/hadoop/yarn/lib/
> ./docs/vagrant.md:150:  path:
> file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./docs/myriad-dev.md:67:cp
> myriad-executor/build/libs/myriad-executor-0.0.1.jar
> /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
> ./docs/myriad-remote-distribution-configuration.md:39:cp
> myriad-executor/build/libs/myriad-executor-0.0.1.jar
> /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
> ./myriad-scheduler/src/main/resources/myriad-config-default.yml:55:  path:
> file:///usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/main/resources/myriad-config-default.yml:58:  #path:
> hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar
>
> ./myriad-scheduler/src/main/java/org/apache/myriad/configuration/MyriadConfiguration.java:55:
> * path:
> file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/test/resources/myriad-config-test-default.yml:31:
> path: file:///usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/test/resources/myriad-config-test-default.yml:34:
> #path: hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar
>
>
> On Mon, Nov 9, 2015 at 1:20 AM, Santosh Marella 
> wrote:
>
> > I haven't created a git tag for the rc. I will try creating one in the
> > morning.
> >
> > --
> > Sent from mobile
> > On Nov 9, 2015 1:10 AM, "Adam Bordelon"  wrote:
> >
> > > Awesome! I'll try to build and run this myself this week.
> > > Is there a git tag I can sync to for 0.1.0-rc0?
> > >
> > > On Mon, Nov 9, 2015 at 1:01 AM, Santosh Marella  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a build for Apache Myriad 0.1.0-incubating, release
> > > > candidate 0.
> > > >
> > > > Thanks to everyone who has contributed to this release.
> > > >
> > > > Here’s the release notes:
> > > > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> > > >
> > > > The commit to be voted upon is:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-myriad/commit/343a11aceb0fc9b5c84f46bba43b2423643eff6f
> > > >
> > > > The artifacts to be voted on are located here:
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc0/
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/smarella.asc
> > > >
> > > > Please vote on releasing this package as Apache Myriad
> > 0.1.0-incubating.
> > > >
> > > > The vote is open for the next 72 hours and passes if a majority of
> > > > at least three +1 PPMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > > [ ] -1 Do not release this package because...
> > > >
> > > >
> > > > Here is my vote:
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Santosh
> > > >
> > >
> >
>

  

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 0)

[yuliya Feldman]

Are you saying we should have rc1 after you are done?
  From: Brandon Gulla 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, November 9, 2015 5:33 AM
 Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 
0)
   
I am adjusting the version on the Dockerfiles as we speak and will be
submitting a PR very shortly.

2.7.1. -> 2.7.0



On Mon, Nov 9, 2015 at 4:29 AM, Adam Bordelon  wrote:

> Do we need to rev the version on the executor jars up to 0.1.0?
> Is there anywhere else we need to bump the version?
>
> $ grep -rin "myriad.*[^ \[\(0-9]0" . |grep -v Apache
> ./docker/Dockerfile:34:ADD /libs/myriad-executor-runnable-0.0.1.jar
> /usr/local/libexec/mesos/
> ./docker/ResourceManager.dockerfile:30:ADD
> /libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/
> ./docker/build-myriad.sh:40:cp -rf
> ../myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar libs/
> ./docs/vagrant.md:100
> :/vagrant/myriad-scheduler/build/libs/myriad-executor-0.0.1.jar
> ./docs/vagrant.md:107:cp
> /vagrant/myriad-executor/build/libs/myriad-executor-0.0.1.jar
> $YARN_HOME/share/hadoop/yarn/lib/
> ./docs/vagrant.md:150:  path:
> file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./docs/myriad-dev.md:67:cp
> myriad-executor/build/libs/myriad-executor-0.0.1.jar
> /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
> ./docs/myriad-remote-distribution-configuration.md:39:cp
> myriad-executor/build/libs/myriad-executor-0.0.1.jar
> /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
> ./myriad-scheduler/src/main/resources/myriad-config-default.yml:55:  path:
> file:///usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/main/resources/myriad-config-default.yml:58:  #path:
> hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar
>
> ./myriad-scheduler/src/main/java/org/apache/myriad/configuration/MyriadConfiguration.java:55:
> * path:
> file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/test/resources/myriad-config-test-default.yml:31:
> path: file:///usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar
> ./myriad-scheduler/src/test/resources/myriad-config-test-default.yml:34:
> #path: hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar
>
>
> On Mon, Nov 9, 2015 at 1:20 AM, Santosh Marella 
> wrote:
>
> > I haven't created a git tag for the rc. I will try creating one in the
> > morning.
> >
> > --
> > Sent from mobile
> > On Nov 9, 2015 1:10 AM, "Adam Bordelon"  wrote:
> >
> > > Awesome! I'll try to build and run this myself this week.
> > > Is there a git tag I can sync to for 0.1.0-rc0?
> > >
> > > On Mon, Nov 9, 2015 at 1:01 AM, Santosh Marella  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a build for Apache Myriad 0.1.0-incubating, release
> > > > candidate 0.
> > > >
> > > > Thanks to everyone who has contributed to this release.
> > > >
> > > > Here’s the release notes:
> > > > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> > > >
> > > > The commit to be voted upon is:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-myriad/commit/343a11aceb0fc9b5c84f46bba43b2423643eff6f
> > > >
> > > > The artifacts to be voted on are located here:
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc0/
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/smarella.asc
> > > >
> > > > Please vote on releasing this package as Apache Myriad
> > 0.1.0-incubating.
> > > >
> > > > The vote is open for the next 72 hours and passes if a majority of
> > > > at least three +1 PPMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > > [ ] -1 Do not release this package because...
> > > >
> > > >
> > > > Here is my vote:
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Santosh
> > > >
> > >
> >
>



-- 
Brandon

  

Re: New Committer: Swapnil Daingade

[yuliya Feldman]

Congratulations Swapnil!!!
Well done
Yuliya
  From: Adam Bordelon 
 To: dev@myriad.incubator.apache.org 
 Sent: Thursday, November 5, 2015 4:38 AM
 Subject: New Committer: Swapnil Daingade
   
The Podling Project Management Committee (PPMC) for Apache Myriad has asked
Swapnil Daingade to become a committer and PPMC member and we are pleased
to announce that he has accepted.

Please join me in welcoming Swapnil as a Myriad committer, and let's thank
him for all his contributions so far. Looking forward to more!

Cheers,
-Adam-


   

Re: New committer: Darin J

[yuliya Feldman]

Congratulations Darrin!!!
Great news.
Yuliya
  From: Adam Bordelon 
 To: dev@myriad.incubator.apache.org 
 Sent: Thursday, November 5, 2015 4:36 AM
 Subject: New committer: Darin J
   
The Podling Project Management Committee (PPMC) for Apache Myriad has asked
Darin to become a committer and PPMC member and we are pleased to announce
that he has accepted.

Please join me in welcoming Darin as a Myriad committer, and let's thank
him for all his contributions so far. Looking forward to more!

Cheers,
-Adam-


   

PR 121 is ready for review

[yuliya Feldman]

Hello here,
I have updated PR: https://github.com/mesos/myriad/pull/121   (Myriad-Issue-60 
Added ability to launch JHS)
With latest set of changes based on prior feedback.
Will appreciate your review.
Thanks,Yuliya

Re: apache rat plugin

[yuliya Feldman]

Yes, we should do it - it forces not to forget license
  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Thursday, October 15, 2015 2:53 PM
 Subject: Re: apache rat plugin
   
Great idea Jim. Integrating it into the build would be very helpful.

Santosh



On Thu, Oct 15, 2015 at 2:44 PM, Adam Bordelon  wrote:

> +1 to automated checking. I don't want to have to remember to tell people
> to add licenses to every new file.
>
> On Thu, Oct 15, 2015 at 1:15 PM, Jim Klucar  wrote:
>
> > I assume gradle can use Maven plugins? If so, there's a rat (Release
> Audit
> > Tool) plugin that checks file headers for licenses. What do you think
> about
> > using it?
> >
> > http://creadur.apache.org/rat/apache-rat-plugin/
> >
> > Jim
> >
>


   

Re: Myriad 0.1.0 Release Planning

[yuliya Feldman]

I had quite a hard time rebasing all 15 commits on top of latest NM Constraints 
feature, so I end up squashing all my 15 commits into one.
On top of it I screwed up branch I had everything on, so had to create a 
separate branch and subsequently it created a new PR.
Codewise it is the same as was marked as "shippable", upreved on top of the 
latest Santosh's changes.
Tested after uprev/merge and what not.
New PR is: https://github.com/mesos/myriad/pull/132

Thanks,Yuliya

  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, October 14, 2015 11:49 PM
 Subject: Re: Myriad 0.1.0 Release Planning
   
Hello All,

 Thanks for actively reviewing the PRs. Most PRs seem to be close to
shipping! Awesome.

 On https://github.com/apache/incubator-myriad/pull/6, we felt that we
should get the "almost shippable"
 PRs merged in *before Monday*.

 Jim can then run perform the namespace changes to all the files in
 the master. This saves a bunch of rebases for the PRs currently
outstanding.

Thanks,
Santosh



On Wed, Oct 14, 2015 at 3:17 PM, Santosh Marella 
wrote:

> Super quick Jim! Thanks!
>
> Santosh
>
> On Wed, Oct 14, 2015 at 2:48 PM, Jim Klucar  wrote:
>
>> I just submitted a PR that fixes the source headers and changes the
>> namespace.
>>
>> The Apache header was taken from here
>> http://www.apache.org/legal/src-headers.html
>>
>> p.s.  I have mad vi skills
>>
>> On Wed, Oct 14, 2015 at 3:52 PM, Santosh Marella 
>> wrote:
>>
>> > Hello Myriad Community,
>> >
>> >  As we prepare for the first Myriad release, here is the status quo and
>> >  actions required from all of us.
>> >
>> > **PRs**
>> > 8 PRS outstanding: 5 to mesos/myriad + 3 to apache/incubator-myriad
>> >
>> > * Action(s) Required *
>> > 1.  We REALLY need to expedite reviews and conclusions on the
>> outstanding
>> > PRs.
>> >      Appreciate any help reviewing/committing the PRs.
>> >
>> > 2. We will be cutting a RC from apache/incubator-myriad. Thus, at the
>> > least,
>> >    all the current PRs targeted to mesos/myriad need to either be
>> merged
>> > into
>> >    apache git or should become PRs against apache git.
>> >
>> >
>> > **JIRAs**
>> > Here is the current list of JIRAs with "Fix Version" set to
>> myriad-0.1.0:
>> > http://tinyurl.com/ptttsq3
>> >
>> > * Action(s) Required *
>> > 1. There are some JIRAs related to Apache licensing, copyright, changing
>> >      com.ebay namespace to org.apache. These are blockers for the
>> release.
>> >      Any volunteers to take them up?
>> >
>> > 2.  If there are JIRAs in this list that can be moved out,
>> >      please remove the "Fix Version" or send a message to dev@..
>> >
>> > 3.  If there are JIRAs that are not in this list, but should be
>> targeted to
>> > myriad-0.1.0,
>> >      please add the "Fix Version" or send a message to dev@.
>> >
>> > **Apache INFRA**
>> >
>> > * Action(s) Required: From Mentors/IPMC members *
>> >
>> > 1. INFRA ticket to create a "myriad" folder under
>> >    "dev": https://dist.apache.org/repos/dist/dev/incubator/
>> >    "release": https://dist.apache.org/repos/dist/release/incubator/
>> > 2. INFRA ticket to have Nexus access for myriad in order to
>> >      publish release artificats to
>> > https://repository.apache.org/content/repositories
>> >    (Not sure if this is needed for the 0.1.0 release or not)
>> >
>> > Thanks,
>> > Santosh
>> >
>> > On Wed, Oct 7, 2015 at 3:20 PM, Adam Bordelon 
>> wrote:
>> >
>> > > Santosh has volunteered as the Myriad 0.1.0 release manager, and I
>> will
>> > > assist.
>> > >
>> > > As my first contribution, I submit these links:
>> > > https://incubator.apache.org/guides/releasemanagement.html
>> > >
>> https://incubator.apache.org/incubation/Incubation_Policy.html#Releases
>> > > https://incubator.apache.org/guides/release.html
>> > > as well as the Mesos release guide as a reference (what I'm familiar
>> > with):
>> > > http://mesos.apache.org/documentation/latest/release-guide/
>> > >
>> > > We are hoping to cut the first release candidate for 0.1.0 in a
>> couple of
>> > > weeks, so please tag any blocker JIRAs with the "Myriad 0.1.0"
>> version in
>> > > JIRA:
>> > https://issues.apache.org/jira/browse/MYRIAD/fixforversion/12333141
>> > >
>> > > Mentors, we'd love your help with this process, since we'll need to
>> get
>> > > this approved by the Incubator PMC before we can release.
>> > >
>> >
>>
>
>


  

Re: https://github.com/mesos/myriad/pull/121 ready for review (again)

[yuliya Feldman]

I have addressed majority of review comments.
Thanks,Yuliya
  From: yuliya Feldman <yufeld...@yahoo.com>
 To: Dev <dev@myriad.incubator.apache.org> 
 Sent: Thursday, October 1, 2015 1:14 AM
 Subject: https://github.com/mesos/myriad/pull/121 ready for review (again)
   
Hello here,
I have made couple of major refactoring to address concerns on:1. usage of 
NMProfile in JHS - now it is abstracted out to ServiceResourceProfile and 
ExtendedResourceProfile2. reuse of command line generation to download distro 
I really hope to wrap up this request soon: 
https://github.com/mesos/myriad/pull/121
@JimCulcar  - I would also like to discuss with you UI for other services 
flexup/down (besides NMs)
Thanks,Yuliya



  

Re: Struggling with Permissions

[yuliya Feldman]

Please change workdir directory for mesos slave to one that is not /tmp and 
make sure that dir is owned by root.
There is one more caveat with binary distro and MapR - in Myriad code for 
binary distro configuration is copied from RM to NMs - it doe snot work for 
MapR since we need hostname (yes for the sake of local volumes) to be unique.
MapR will have Myriad release to handle this situation.
  From: John Omernik 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, November 17, 2015 11:37 AM
 Subject: Re: Struggling with Permissions
   
Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
followed the instructions with the same errors. At this point do I still
need to have a place where the entire path is owned by root? That seems
like a an odd requirement (a changed of each node to facilitate a framework)





On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:

> Hey all, I am struggling with permissions on myriad, trying to get the
> right permissions in the tgz as well as who to run as.  I am running in
> MapR, which means I need to run as mapr or root (otherwise my volume
> creation scripts will fail on MapR, MapR folks, we should talk more about
> those scripts)
>
> But back to the code, I've had lots issues. When I run the Frameworkuser
> and Superuser as mapr, it unpacks everything as MapR and I get a
> "/bin/container-executor" must be owned by root but is owned by 700 (my
> mapr UID).
>
> So now I am running as root, and I am getting the error below as it
> relates to /tmp. I am not sure which /tmp this refers to. the /tmp that my
> slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> MaprFS /tmp directory (both of which are world writable, as /tmp typically
> is... or am I mistaken here?)
>
> Any thoughts on how to get this to resolve? This is when nodemanager is
> trying to start running as root and root for both of my Myriad users.
>
> Thanks!
>
>
> Caused by: ExitCodeException exitCode=24: File /tmp must not be world or 
> group writable, but is 1777
>
>
>
>


  

Re: [jira] [Commented] (MYRIAD-129) Creating custom profiles requires configurations changes on all nodes.

[yuliya Feldman]

There should be a line between what is stored in state store and what is stored 
in ZK (if it is not a state store), as we would need ot maintain two systems of 
record in such a case.
Definitely ZK has such advantages as "triggers" and MyriadScheduler can find 
out instantly if something was changed/removed/deleted and in case of multiple 
MyriadSchedulers running it would be an advantage. I just don't think we would 
have such a situation where we have multiple MyriadSchedulers acting as masters.
and BTW - those comments do not get on the JIRA, so we better continue 
conversation on the JIRA.

  From: John Omernik <j...@omernik.com>
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, September 8, 2015 11:27 AM
 Subject: Re: [jira] [Commented] (MYRIAD-129) Creating custom profiles requires 
configurations changes on all nodes.
   
Since the profile information is small, could this be done in Zookeeper?



On Tue, Sep 8, 2015 at 12:59 PM, Yuliya Feldman (JIRA) <j...@apache.org>
wrote:

>
>    [
> https://issues.apache.org/jira/browse/MYRIAD-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735288#comment-14735288
> ]
>
> Yuliya Feldman commented on MYRIAD-129:
> ---
>
> I think we should have API/UI to add/remove/modify profiles. Also keep all
> the configuration updates in state store in case of failover
>
> > Creating custom profiles requires configurations changes on all nodes.
> > --
> >
> >                Key: MYRIAD-129
> >                URL: https://issues.apache.org/jira/browse/MYRIAD-129
> >            Project: Myriad
> >          Issue Type: Bug
> >            Reporter: Aashreya Ravi Shankar
> >
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>


  

Re: Getting Nodes to be "Running" in Mesos

[yuliya Feldman]

Take a look at :   https://github.com/mesos/myriad/pull/128
for yarn-site.xml updates

  From: John Omernik 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, September 8, 2015 12:38 PM
 Subject: Getting Nodes to be "Running" in Mesos
   
So I am playing around with a recent build of Myriad, and I am using MapR
5.0 (hadoop-2.7.0) I hate to use the dev list as a "help Myriad won't run"
forum, so please forgive me if I am using the list wrong.

Basically, I seem to be able to get myriad running, and the things up, and
it tries to start a nodemanager.

In mesos, the status of the nodemanager task never gets past staging, and
eventually, fails.  The logs for both the node manager and myriad, seem to
look healthy, and I am not sure where I should look next to troubleshoot
what is happening. Basically you can see the registration of the
nodemanager, and then it fails with no error in the logs... Any thoughts
would be appreciated on where I can look next for troubleshooting.


Node Manager Logs (complete)

STARTUP_MSG:  build = g...@github.com:mapr/private-hadoop-common.git
-r fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on
2015-08-19T20:02Z
STARTUP_MSG:  java = 1.8.0_45-internal
/
15/09/08 14:35:23 INFO nodemanager.NodeManager: registered UNIX signal
handlers for [TERM, HUP, INT]
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType
for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for
class org.apache.hadoop.yarn.server.nodemanager.NodeManager
15/09/08 14:35:24 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
15/09/08 14:35:24 INFO impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
15/09/08 14:35:24 INFO impl.MetricsSystemImpl: NodeManager metrics
system started
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadService
15/09/08 14:35:24 INFO localizer.ResourceLocalizationService: per
directory file limit = 8192
15/09/08 14:35:24 INFO localizer.ResourceLocalizationService:
usercache path :
file:///tmp/hadoop-mapr/nm-local-dir/usercache_DEL_1441740924753
15/09/08 14:35:24 INFO event.AsyncDispatcher: Registering class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType
for class 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
15/09/08 14:35:24 WARN containermanager.AuxServices: The Auxilurary
Service named 'mapreduce_shuffle' in the configuration is for class
org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send
ServiceData and read Service 

Re: Getting Nodes to be "Running" in Mesos

[yuliya Feldman]

John,
It is a problem with permissions for container-executor.cfg - it requires whole 
path to it to be owned by root.
One step is to change work-dir for mesos-slave to point to a different 
directory (not tmp) that is writable only by root.
It still does not solve full issue since binary distro is changing permissions 
of the distro directory to a framework user.
If framework user is root and myriad is running as root it can be solved, 
otherwise we need changes to binary distro code.
I was planning to do it, but got distracted by other stuff. Will try to look at 
it this week.
Thanks,Yuliya
  From: John Omernik <j...@omernik.com>
 To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com> 
 Sent: Tuesday, September 8, 2015 1:31 PM
 Subject: Re: Getting Nodes to be "Running" in Mesos
   
interesting... when I did root as the framework user then I got this:

ExitCodeException exitCode=24: File /tmp must not be world or group
writable, but is 1777

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
15/09/08 15:30:38 INFO nodemanager.ContainerExecutor:
15/09/08 15:30:38 INFO service.AbstractService: Service NodeManager
failed in state INITED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to
initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to
initialize container executor
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:212)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
Caused by: java.io.IOException: Linux container executor not
configured properly (error=24)
    at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:188)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210)
    ... 3 more
Caused by: ExitCodeException exitCode=24: File /tmp must not be world
or group writable, but is 1777

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182)
    ... 4 more
15/09/08 15:30:38 WARN service.AbstractService: When stopping the
service NodeManager : java.lang.NullPointerException
java.lang.NullPointerException
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:162)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:274)
    at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
    at 
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
    at 
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
15/09/08 15:30:38 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to
initialize container executor
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:212)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
Caused by: java.io.IOException: Linux container executor not
configured properly (error=24)
    at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:188)
    at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210)
    ... 3 more
Caused by: ExitCodeException exitCode=24: File /tmp must not be world
or group

Marathon version 0.11 dependency on Java 8

[yuliya Feldman]

Hello guys,
Is it true that Java 8 is a requirement to run Marathon 0.11?
It feels a bit strange that it would not support Java 7.
Thanks,Yuliya

Re: Understanding Myriad workflows

[yuliya Feldman]

Myriad implementation does not depend on Marathon, so you don't need to start 
MyraidScheduler/RM using Marathon.
In this case you need to take care of MyriadScheduler/RM restart in case of 
it's failure yourself.
Myriad HA feature will take care of preserving state, but restart and 
discoverability will be in your hands.
Thanks,Yuliya 
  From: Haripriya Ayyalasomayajula 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, September 21, 2015 10:19 AM
 Subject: Re: Understanding Myriad workflows
   
Hi Adam,

Thanks. I was aware of the possibility of starting RM using Marathon -
thats for clarifying. I was wondering if any of the Myriad implementation
is based on that of Marathon..

On Mon, Sep 21, 2015 at 10:08 AM, Adam Bordelon  wrote:

> Haripriya,
>
> You may want to use something like Marathon to start the RM/scheduler and
> keep it up, so that Marathon can restart it somewhere else if the RM
> app/process, or even its entire machine fails. You can, of course, start
> the RM/scheduler manually on a node, but you'll have to manage its HA on
> your own.
>
> On Mon, Sep 21, 2015 at 10:06 AM, Haripriya Ayyalasomayajula <
> aharipriy...@gmail.com> wrote:
>
> > Hi all,
> >
> > https://cwiki.apache.org/confluence/display/MYRIAD/Administration
> >
> > I see that there is a mention of Marathon in this document - I'm not sure
> > where in the Myriad workflow marathon is being used - can anyone help me
> > understand this better - or if I'm missing something ?
> >
> > On Wed, Sep 16, 2015 at 11:07 AM, Ruth Harris 
> > wrote:
> >
> > > hi Haripriya,
> > >
> > > No problem. Let me know if you see any errors or inconsistencies.
> > >
> > > Regards,  --ruth
> > >
> > > Ruth Harris
> > > Sr. Tech. Writer, MapR
> > >
> > > On Wed, Sep 16, 2015 at 9:44 AM, Haripriya Ayyalasomayajula <
> > > aharipriy...@gmail.com> wrote:
> > >
> > > > Thanks Ruth! That was helpful!
> > > >
> > > > On Mon, Sep 14, 2015 at 1:06 PM, Ruth Harris 
> > > wrote:
> > > >
> > > > > The REST API is documented in the Apache wiki:
> > > > >
> > https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Cluster+API
> > > > >
> > > > > Also, in progress, documentation on fine-grained scaling:
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/MYRIAD/Fine-grained+Scaling
> > > > >
> > > > > --ruth
> > > > >
> > > > > On Mon, Sep 14, 2015 at 12:24 PM, Haripriya Ayyalasomayajula <
> > > > > aharipriy...@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Is there any architecture document describing the basic
> workflows -
> > > on
> > > > > flex
> > > > > > up the methods called etc? I'm going through the source code on
> git
> > > hub
> > > > > and
> > > > > > was wondering if there is any such source which would help me
> > > > understand
> > > > > > the internals more..
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Haripriya Ayyalasomayajula
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Ruth Harris
> > > > > Sr. Technical Writer, MapR


> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Haripriya Ayyalasomayajula
> > > >
> > >
> > >
> > >
> > > --
> > > Ruth Harris
> > > Sr. Technical Writer, MapR
> > >
> >
> >
> >
> > --
> > Regards,
> > Haripriya Ayyalasomayajula
> >
>



-- 
Regards,
Haripriya Ayyalasomayajula


  

Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete Branches

[yuliya Feldman]

I fully second Todd.
Thanks,Yuliya
  From: Todd Richmond 
 To: dev@myriad.incubator.apache.org 
 Sent: Thursday, December 3, 2015 8:59 AM
 Subject: Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete Branches
   
excellent pro/con list

+1 for either A or + .5 for Alt 1. A branch is easy to follow and allows for 
long term patch support. Each new 0.x.y patch release becomes the only 
“supported” 0.x version but more than one “x” can be supported simultaneously

Alt 2 tags work best when you release very often (such as monthly) to users who 
are willing to upgrade and so can quickly deprecate previous releases. 
Cherry-picking is more manual effort and possibly error prone as the committer 
count grows

+1 for proposal B. feature branches can usually be done on private forks 
instead and should definitely be removed once the feature is merged to develop

  Todd




> On Dec 3, 2015, at 12:36 AM, Adam Bordelon  wrote:
> 
> Proposal A: Use Release Branches
> I propose that we create a '0.1.x' branch that will contain all of the
> 0.1.0-rc tags, the final 0.1.0 release tag, and any tags related to hotfix
> releases on top (0.1.1, 0.1.2). A hotfix release like 0.1.1 can cherry-pick
> fixes from master directly on top of the 0.1.0 tag in the 0.1.x branch, and
> we'll iterate on its rc's in the 0.1.x branch. Development for
> features/fixes for 0.2.0 go into the master branch, and whenever 0.2.0 is
> feature-complete/ready, we'll cut the new '0.2.x' branch from master and
> tag a 0.2.0-rc1, then cherry pick any necessary fixes from master into
> 0.2.x, for future release candidates and hotfix releases.
> + Easy to create/review github PRs to merge fixes into release candidates
> and hotfix releases.
> + Release candidates and hotfixes are handled in the appropriate release
> branch, while normal development can continue in master.
> + Minimal additional branches/commands required; no need for ephemeral
> branches for each release (candidate).
> 
> Alternative 1: Follow
> http://nvie.com/posts/a-successful-git-branching-model/#release-branches
> My proposal is similar to the model described by nvie except that we use
> the myriad 'master' branch for what he calls the 'develop' branch, and we
> use our 0.1.x,0.2.x release branches as longer-lived branches for hotfix
> releases. (Note: Feature branches are a separate discussion, unrelated to
> release management.)
> + Easy to follow guide.
> + Good separation of concerns/responsibility.
> - Doesn't explain how release candidates are handled.
> - So many branches.
> 
> Alternative 2: Use tags for releases, no branches (like Mesos does)
> See the discussion at:
> http://stackoverflow.com/questions/9810050/git-tag-vs-release-beta-branches
> + No mess of branches in the repo; no merging between branches.
> + Since release candidates and releases are cherry-picked and tagged,
> normal development can continue on master without interruption/corruption.
> - Github PRs cannot use a tag (Dealbreaker?).
> http://stackoverflow.com/a/12279290/4056606
> 
> Please let me know your thoughts on release branches. I went ahead and
> created the '0.1.x' branch from the 0.1.0-rc3 tag so you can check it out
> and play around, and so you can push 0.2.0 features to master without
> worrying about messing up the 0.1.0 release. We can cherry-pick any
> rc4/0.1.1 patches out of master, and we can always delete/rename/reorg the
> release branch later if desired.
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/heads/0.1.x
> 
> 
> 
> Proposal B: Delete all these obsolete branches from the Apache git repo:
> 9/23/15 phase1 (72 behind master)
> 8/12/15 constraints (kensipe)
> 8/12/15 myriadha (kensipe)
> 8/10/15 issue_14 (smarella)
> 7/17/15 executor-only-application (kensipe)
> 6/11/15 multi-project (kensipe)
> 6/11/15 docker-image (kensipe)
> 3/04/15 issue_16 (mohit)
> If nobody speaks up to save any/all of these, I'll delete them next week.
> (Note that most of these still live on in the old github repo, if you look
> closely.)


 

Re: Myriad is 0.1.0

[yuliya Feldman]

Yes, indeed kudos to Santosh for managing this release!!!
Yuliya
  From: Swapnil Daingade 
 To: dev@myriad.incubator.apache.org 
 Sent: Thursday, December 10, 2015 7:14 PM
 Subject: Re: Myriad is 0.1.0
   
Congratulations and thank you everyone for all the hard work!!!
Special thanks to Santosh for being an awesome release manager!!!

Regards
Swapnil




On Fri, Dec 11, 2015 at 12:11 AM, Darin Johnson 
wrote:

> Sweet, thanks for all the work on the release guys!
>
> On Thu, Dec 10, 2015 at 1:31 PM, mohit soni 
> wrote:
>
> > WooHoo! Perfect holiday gift. Thanks everyone.
> >
> > On Thu, Dec 10, 2015 at 10:19 AM, Aashreya Shankar <
> ashan...@maprtech.com>
> > wrote:
> >
> > > This is great !
> > > Good job everyone.
> > >
> > > On Thu, Dec 10, 2015 at 8:02 AM, Yuliya 
> > > wrote:
> > >
> > > > Great news!!!
> > > >
> > > > We are finally there
> > > >
> > > >
> > > > > On Dec 10, 2015, at 1:13 AM, Santosh Marella <
> smare...@maprtech.com>
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > >  Congratulations on a the first Apache Myriad release..! Kudos to
> > > > everyone
> > > > > involved for making this happen.
> > > > >
> > > > >  As we now have IPMC's approval, there are a few things that I did
> to
> > > > wrap
> > > > > up the release:
> > > > >  - Make 0.1.0 artifacts available from the release SVN repo [1].
> > > > >  - Git tag the voted RC as the "0.1.0" release [2].
> > > > >  - Delete the previously marked git RC tags.
> > > > >  - Closed the remaining JIRAs marked for 0.1.0 version and marked
> the
> > > > > 0.1.0 version as "released" [3].
> > > > >  - Submitted a PR [4] with scripts to prepare a RC and release a RC
> > > > > (automates the above git and svn steps)
> > > > >  - Updated the release guide [5] with voting links to help future
> > > release
> > > > > managers.
> > > > >
> > > > >  Here are a couple of things still remaining:
> > > > >  - Update the downloads page on Myriad's website with links to
> 0.1.0
> > > > > artifacts on svn, git tag, release notes.
> > > > >  - Do an announcement blog post. Here is a draft [6]. Please
> suggest
> > > any
> > > > > changes.
> > > > >
> > > > >  If I may be missing something for 0.1.0, appreciate if you could
> > bring
> > > > to
> > > > > my notice.
> > > > >
> > > > >  1.
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/release/incubator/myriad/myriad-0.1.0-incubating/
> > > > >  2.
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-myriad/releases/tag/myriad-0.1.0-incubating
> > > > >  3.
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/MYRIAD/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel
> > > > >  4. https://github.com/apache/incubator-myriad/pull/53
> > > > >  5.
> > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Guide
> > > > >  6.
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1zCXnDlqzNhj0BL_CqRz5-poCap9QFah7R3tKkHdspYg/edit
> > > > >
> > > > > Thanks,
> > > > > Santosh
> > > >
> > >
> >
>


 

Re: Myriad is 0.1.0

[yuliya Feldman]

Great news!!!
The very first release
Thanks everybody,Yuliya
  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Sunday, December 13, 2015 10:35 PM
 Subject: Re: Myriad is 0.1.0
   
Updated the Apache Myriad website with 0.1.0 release news, download
information and an announcement blog:

http://myriad.incubator.apache.org/news/
http://myriad.incubator.apache.org/downloads/
http://myriad.incubator.apache.org/blogs/2015/12/09/myriad-0-1-0-release-announcement.html

Please spread the word!

Santosh



On Thu, Dec 10, 2015 at 10:41 AM, Darin Johnson 
wrote:

> Sweet, thanks for all the work on the release guys!
>
> On Thu, Dec 10, 2015 at 1:31 PM, mohit soni 
> wrote:
>
> > WooHoo! Perfect holiday gift. Thanks everyone.
> >
> > On Thu, Dec 10, 2015 at 10:19 AM, Aashreya Shankar <
> ashan...@maprtech.com>
> > wrote:
> >
> > > This is great !
> > > Good job everyone.
> > >
> > > On Thu, Dec 10, 2015 at 8:02 AM, Yuliya 
> > > wrote:
> > >
> > > > Great news!!!
> > > >
> > > > We are finally there
> > > >
> > > >
> > > > > On Dec 10, 2015, at 1:13 AM, Santosh Marella <
> smare...@maprtech.com>
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > >  Congratulations on a the first Apache Myriad release..! Kudos to
> > > > everyone
> > > > > involved for making this happen.
> > > > >
> > > > >  As we now have IPMC's approval, there are a few things that I did
> to
> > > > wrap
> > > > > up the release:
> > > > >  - Make 0.1.0 artifacts available from the release SVN repo [1].
> > > > >  - Git tag the voted RC as the "0.1.0" release [2].
> > > > >  - Delete the previously marked git RC tags.
> > > > >  - Closed the remaining JIRAs marked for 0.1.0 version and marked
> the
> > > > > 0.1.0 version as "released" [3].
> > > > >  - Submitted a PR [4] with scripts to prepare a RC and release a RC
> > > > > (automates the above git and svn steps)
> > > > >  - Updated the release guide [5] with voting links to help future
> > > release
> > > > > managers.
> > > > >
> > > > >  Here are a couple of things still remaining:
> > > > >  - Update the downloads page on Myriad's website with links to
> 0.1.0
> > > > > artifacts on svn, git tag, release notes.
> > > > >  - Do an announcement blog post. Here is a draft [6]. Please
> suggest
> > > any
> > > > > changes.
> > > > >
> > > > >  If I may be missing something for 0.1.0, appreciate if you could
> > bring
> > > > to
> > > > > my notice.
> > > > >
> > > > >  1.
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/release/incubator/myriad/myriad-0.1.0-incubating/
> > > > >  2.
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-myriad/releases/tag/myriad-0.1.0-incubating
> > > > >  3.
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/MYRIAD/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel
> > > > >  4. https://github.com/apache/incubator-myriad/pull/53
> > > > >  5.
> > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Guide
> > > > >  6.
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1zCXnDlqzNhj0BL_CqRz5-poCap9QFah7R3tKkHdspYg/edit
> > > > >
> > > > > Thanks,
> > > > > Santosh
> > > >
> > >
> >
>


 

Re: Next dev sync hangout will be on 1/6/2016

[yuliya Feldman]

Do we have a sync today?
 

  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, December 16, 2015 9:47 AM
 Subject: Next dev sync hangout will be on 1/6/2016
   
We have decided to hold the next dev sync on 1/6/2016 (instead of
12/30/2015).

Meeting notes from today's hangout:
https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#

Thanks,
Santosh


   

Re: Struggling with Permissions

[yuliya Feldman]

You mean that if you enable cgroups in yarn /tmp as a slave temp dir works just 
fine?
  From: John Omernik <j...@omernik.com>
 To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com> 
 Sent: Thursday, November 19, 2015 8:02 AM
 Subject: Re: Struggling with Permissions
   
Ok all I now have a real answer and a solution to the needing to change
/tmp locations in Mesos Slaves.

Basically, it wasn't a difference between Redhat and Ubuntu, it was a
mistake I made in my yarn-site.xml for the resource manager.  Basically I
copied the yarn-site from the wiki but on one I had uncommented the cgroups
section. My hypothesis: when running with Cgroups information, Yarn doesn't
see the /tmp and it's permissions, therefore creating the tgz with the
script I posted earlier works.  The permissions are what Yarn requires.
Once I set that on my Redhat cluster, all was well.  I realized that in
comparing configs.  That addresses most of my concerns, and I recommend
people use that where possible to avoid the Mesos Slave changes.  We also
may want to do some notes in the wiki around this, but I found it through
hacking, and I'd welcome more "from a point of view of understanding"
comments, I just made it work :)

I'll send another email shortly on my last issue.

On Wed, Nov 18, 2015 at 3:16 PM, John Omernik <j...@omernik.com> wrote:

> Baffling... so as I said, I fixed the mapr.host that works well (as long
> as I have one NM per physical node) and I got my Ubuntu based Mesos cluster
> to work with Myriad.  At this point I am baffled on why Myriad/Yarn would
> complain about permissions on Redhat but not Ubuntu, below I've pulled
> files in /, /hadoop-2.7.0, /hadoop-2.7.0/bin and down the etc tree so you
> can see how they are setup the same. I am running as the same users in
> marathon etc. Same myriad configs... but I am getting the permissions issue
> for Redhat, and Ubuntu is working fine (and more than just getting the NMs
> running, I flexed up 6 NMs and was running hive queries with no issues!)
>
> The only differences I can see is that the java versions are different
> (1.8.0_45 for Ubuntu and 1.8.0_65 for Redhat) and the apparently the builds
> of hadoop-2.7.0 are different from MapR. (They are different in size as
> well, but that may be the difference between a RPM Redhat install and a
> Debian Ubuntu install, I guess I didn't expect different hadoop builds
> between platforms though...)
>
> If you can see anything else, please let me know, I'd love to have this
> working.
>
> John
>
>
> Here are the dumps:  Ubuntu... this is working:
>
> STARTUP_MSG:  build = g...@github.com:mapr/private-hadoop-common.git -r
> fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on
> 2015-08-19T20:02Z
> STARTUP_MSG:  java = 1.8.0_45-internal
>
> No Error
> Ubuntu
>
> marathon user: mapr
> myriadframeuser: mapr
> myriadsuperuser: mapr
>
> /
> drwxr-xr-x 10 mapr mapr 4 KB Sep 10 14:40 hadoop-2.7.0
> -rw-r--r-- 1 mapr mapr 77 KB Nov 18 12:48 conf
> -rw-r--r-- 1 mapr mapr 282 MB Nov 18 12:48 hadoop-2.7.0.tgz
> -rw-r--r-- 1 mapr mapr 156 KB Nov 18 14:04 stderr
> -rw-r--r-- 1 mapr mapr 743 B Nov 18 12:48 stdout
>
> /hadoop-2.7.0/
>
> drwxrwxrwx 3 mapr root 4 KB Nov 18 12:48 logs
> drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 bin
> drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 etc
> drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 include
> drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 lib
> drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 libexec
> drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 sbin
> drwxr-xr-x 4 mapr root 4 KB Sep 10 14:40 share
> -rw-r--r-- 1 mapr root 15 KB Jul 09 04:36 LICENSE.txt
> -rw-r--r-- 1 mapr root 101 B Jul 09 04:36 NOTICE.txt
> -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36
>
> /hadoop-2.7.0/bin
>
> -rwxr-xr-x 1 mapr root 9 KB Jul 09 04:36 hadoop
> -rwxr-xr-x 1 mapr root 10 KB Jul 09 04:36 hadoop.cmd
> -rwxr-xr-x 1 mapr root 12 KB Jul 09 04:36 hdfs
> -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 hdfs.cmd
> -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 mapred
> -rwxr-xr-x 1 mapr root 6 KB Jul 09 04:36 mapred.cmd
> -rwxr-xr-x 1 mapr root 2 KB Jul 09 04:36 rcc
> -rwxr-xr-x 1 mapr root 172 KB Jul 09 04:36 test-container-executor
> -rwxr-xr-x 1 mapr root 14 KB Jul 09 04:36 yarn
> -rwxr-xr-x 1 mapr root 11 KB Jul 09 04:36 yarn.cmd
> r-x--- 1 root mapr 140 KB Jul 09 04:36 container-executor
>
> /hadoop-2.7.0/etc
>
> drwxr-xr-x 2 mapr root 4 KB Nov 18 12:48 hadoop
>
> /hadoop-2.7.0/etc/hadoop/
>
> rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 capacity-scheduler.xml
> -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 configuration.xsl
> -rw-r--r-- 1 root root 168 B Oct 06 08:37 container-executor.cfg
> -rw-r--r-- 1 mapr root 775 B Oct 06 08:37 core-site.xml
> -rw-r--r--

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 3)

[yuliya Feldman]

When I download release gradlew file is not there, so I can not build
Am I missing anything?
Thanks,Yuliya
  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, November 23, 2015 12:36 PM
 Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 
3)
   
+1 (binding)

- Downloaded the RC
- Verified signatures
- Built binaries by following instructions from the README page [1]
- Deployed binaries on a 2.7.0 MapR cluster as described in the
documentation [2]
- Verified the following:
  - flexup/flexdown NMs of zero/low/medium profiles from Myriad's Web UI.
  - Successfully ran a 10G terasort M/R job with 60 mappers and 10 reducers
  - Myriad/RM HA: Killed RM and restarted it while a M/R job is running.
      - Verified the job completed successfully.
      - Verified that the Myriad UI showed the active NM tasks correctly.
  - Verified "Shutdown Framework"

[1] https://github.com/apache/incubator-myriad#build-myriad
[2]
https://github.com/apache/incubator-myriad/blob/master/docs/myriad-dev.md#step-2-deploy-the-myriad-binaries

Santosh



On Thu, Nov 19, 2015 at 10:37 PM, Santosh Marella 
wrote:

> Hi All,
>
> Firstly, thanks everyone for the valuable contributions to the project and
> for holding on tight as we move along the release process. We're almost
> home!
>
> I have created a source tar ball for Apache Myriad 0.1.0-incubating,
> release candidate 3. This includes the feedback from the recent IPMC
> voting.
> Here’s the release notes:
> https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
>
> The commit to be voted upon is tagged with "myriad-0.1.0-incubating-rc3"
> and is available here:
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/tags/myriad-0.1.0-incubating-rc3
>
> The artifacts to be voted upon are located below. Please note that this
> is a source release:
>
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc3/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/smarella.asc
>
> **Please note that the release tar ball does not include the gradlew
> script to build. You need to generate one in order to build.**
>
> Please try out the release candidate and vote. The vote is open for a
> minimum of 72 hours or until the necessary number of votes (3 binding
> +1s) is reached.
>
> If/when this vote succeeds, I will call for a vote with IPMC seeking
> permission to release RC3 as Apache Myriad 0.1.0 (incubating).
>
> [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thanks,
> Santosh
>

  

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 3)

[yuliya Feldman]

Thank you Santosh
Probably question to Ken:
Just wonder why do we need to download gradle to use it to generate gradlew and 
then gradlew downloads gradle again?
Thanks,Yuliya
  From: Santosh Marella <smare...@maprtech.com>
 To: yuliya Feldman <yufeld...@yahoo.com>; dev@myriad.incubator.apache.org 
 Sent: Monday, November 23, 2015 2:19 PM
 Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 
3)
   
Yes. The gradlew file is excluded from the tarball. We have to generate the
gradlew file using "gradle wrapper" command. Please see the updated build
instructions
https://github.com/apache/incubator-myriad/blob/master/docs/myriad-dev.md



--
Sent from mobile
On Nov 23, 2015 1:56 PM, "yuliya Feldman" <yufeld...@yahoo.com.invalid>
wrote:

> When I download release gradlew file is not there, so I can not build
> Am I missing anything?
> Thanks,Yuliya
>      From: Santosh Marella <smare...@maprtech.com>
>  To: dev@myriad.incubator.apache.org
>  Sent: Monday, November 23, 2015 12:36 PM
>  Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release
> candidate 3)
>
> +1 (binding)
>
> - Downloaded the RC
> - Verified signatures
> - Built binaries by following instructions from the README page [1]
> - Deployed binaries on a 2.7.0 MapR cluster as described in the
> documentation [2]
> - Verified the following:
>  - flexup/flexdown NMs of zero/low/medium profiles from Myriad's Web UI.
>  - Successfully ran a 10G terasort M/R job with 60 mappers and 10 reducers
>  - Myriad/RM HA: Killed RM and restarted it while a M/R job is running.
>      - Verified the job completed successfully.
>      - Verified that the Myriad UI showed the active NM tasks correctly.
>  - Verified "Shutdown Framework"
>
> [1] https://github.com/apache/incubator-myriad#build-myriad
> [2]
>
> https://github.com/apache/incubator-myriad/blob/master/docs/myriad-dev.md#step-2-deploy-the-myriad-binaries
>
> Santosh
>
>
>
> On Thu, Nov 19, 2015 at 10:37 PM, Santosh Marella <smare...@maprtech.com>
> wrote:
>
> > Hi All,
> >
> > Firstly, thanks everyone for the valuable contributions to the project
> and
> > for holding on tight as we move along the release process. We're almost
> > home!
> >
> > I have created a source tar ball for Apache Myriad 0.1.0-incubating,
> > release candidate 3. This includes the feedback from the recent IPMC
> > voting.
> > Here’s the release notes:
> > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> >
> > The commit to be voted upon is tagged with "myriad-0.1.0-incubating-rc3"
> > and is available here:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/tags/myriad-0.1.0-incubating-rc3
> >
> > The artifacts to be voted upon are located below. Please note that this
> > is a source release:
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc3/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/smarella.asc
> >
> > **Please note that the release tar ball does not include the gradlew
> > script to build. You need to generate one in order to build.**
> >
> > Please try out the release candidate and vote. The vote is open for a
> > minimum of 72 hours or until the necessary number of votes (3 binding
> > +1s) is reached.
> >
> > If/when this vote succeeds, I will call for a vote with IPMC seeking
> > permission to release RC3 as Apache Myriad 0.1.0 (incubating).
> >
> > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Thanks,
> > Santosh
> >
>
>

  

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 3)

[yuliya Feldman]

+1 (non-binding)
Checked signature
Downloaded tar.gz
Built from tar.gz
Deployed on a one node cluster
Tested NM/JHS flexup/down
Run M/R job successfully
Tested RM HA
Thanks,Yuliya
  From: Santosh Marella 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, November 23, 2015 12:36 PM
 Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 
3)
   
+1 (binding)

- Downloaded the RC
- Verified signatures
- Built binaries by following instructions from the README page [1]
- Deployed binaries on a 2.7.0 MapR cluster as described in the
documentation [2]
- Verified the following:
  - flexup/flexdown NMs of zero/low/medium profiles from Myriad's Web UI.
  - Successfully ran a 10G terasort M/R job with 60 mappers and 10 reducers
  - Myriad/RM HA: Killed RM and restarted it while a M/R job is running.
      - Verified the job completed successfully.
      - Verified that the Myriad UI showed the active NM tasks correctly.
  - Verified "Shutdown Framework"

[1] https://github.com/apache/incubator-myriad#build-myriad
[2]
https://github.com/apache/incubator-myriad/blob/master/docs/myriad-dev.md#step-2-deploy-the-myriad-binaries

Santosh



On Thu, Nov 19, 2015 at 10:37 PM, Santosh Marella 
wrote:

> Hi All,
>
> Firstly, thanks everyone for the valuable contributions to the project and
> for holding on tight as we move along the release process. We're almost
> home!
>
> I have created a source tar ball for Apache Myriad 0.1.0-incubating,
> release candidate 3. This includes the feedback from the recent IPMC
> voting.
> Here’s the release notes:
> https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
>
> The commit to be voted upon is tagged with "myriad-0.1.0-incubating-rc3"
> and is available here:
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/tags/myriad-0.1.0-incubating-rc3
>
> The artifacts to be voted upon are located below. Please note that this
> is a source release:
>
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc3/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/smarella.asc
>
> **Please note that the release tar ball does not include the gradlew
> script to build. You need to generate one in order to build.**
>
> Please try out the release candidate and vote. The vote is open for a
> minimum of 72 hours or until the necessary number of votes (3 binding
> +1s) is reached.
>
> If/when this vote succeeds, I will call for a vote with IPMC seeking
> permission to release RC3 as Apache Myriad 0.1.0 (incubating).
>
> [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thanks,
> Santosh
>

 

Re: Issues while deploying Myriad

[yuliya Feldman]

Hello Jonatan,
You are welcome to bring up issues on this email thread or open JIRA if you 
feel like it.


  From: "jonatan.e...@rai.usc.es" 
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, February 3, 2016 8:44 AM
 Subject: Issues while deploying Myriad
   
Hi,

I am Jonatan and I am currently deploying Myriad as part of a project
on a local datacenter. While testing this technology I found a few issues,
not bugs on the code (yet), but strange behavior and trouble while
deploying  due to what I believe are missing steps or confusing documentation.
(Here:  
https://cwiki.apache.org/confluence/display/MYRIAD/Installation+and+Configuration)

Do I report the issues I found and the ones I am bound to find in the future
to this email?, considering that most of them will probably not be identified
source code bugs.

Thanks for your time, regards

Jonatan.



  

Re: Not able to launch Node Managers via Myriad

[yuliya Feldman]

Could you provide your yarn-site.xml and myriad-config-default.yml ?
Let's make sure you have following in your yarn-site.xml:
      yarn.resourcemanager.scheduler.class    
com.ebay.myriad.scheduler.yarn.MyriadFairScheduler    
One can configure other scehdulers as well from following list: 
com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, 
com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler    
    yarn.nodemanager.resource.cpu-vcores    
${nodemanager.resource.cpu-vcores}        
yarn.nodemanager.resource.memory-mb    
${nodemanager.resource.memory-mb}        
yarn.nodemanager.address    
${myriad.yarn.nodemanager.address}        
yarn.nodemanager.webapp.address    
${myriad.yarn.nodemanager.webapp.address}    
    yarn.nodemanager.webapp.https.address    
${myriad.yarn.nodemanager.webapp.address}    
    yarn.nodemanager.localizer.address    
${myriad.yarn.nodemanager.localizer.address}    
   mapreduce.shuffle.port   
${myriad.mapreduce.shuffle.port}        
yarn.nodemanager.aux-services    
mapreduce_shuffle,myriad_executor        
yarn.nodemanager.aux-services.myriad_executor.class    
com.ebay.myriad.executor.MyriadExecutorAuxService  
And in your yml file (should be at least linked to */etc/hadoop place)You have 
following line defined correctly:
mesosMaster: zk:///mesos
Thanks,Yuliya

  From: Björn Hagemeier 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, February 23, 2016 6:21 AM
 Subject: Re: Not able to launch Node Managers via Myriad
   
Hi all,

Am 23.02.2016 um 12:55 schrieb Björn Hagemeier:
> Hi Santosh,
> 
> Am 23.02.2016 um 01:21 schrieb Santosh Marella:
>> Hey guys.. the flow of things is as below:
>>
>> 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos
>> UI should show this)
does Myriad register itself as a Mesos framework automatically, just by
starting the RM as one would do it without Mesos? Or do I need to start
the RM as a Mesos framework to make the magic happen, i.e. by "mesos
execute"?

In the latter case, I think I found out why my Myriad does not seem to
do anything. I did not start the resource manager as a Mesos framework.
I somehow reckoned, the myriad integration would take care of that by
itself. When I do this, I do see the framework, but with classpath
issues (missing myriad for some reason).

I'll keep digging in that direction.


Cheers,
Björn

> I think this is where it is already failing for me. I do not see the
> Myriad framework in the UI.
>> 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM
>> process is up.
>>    This is queued up until Myriad receives an offer from Mesos that is big
>> enough to launch a medium profle NM.
> Yes, I do see the medium profile NM in the Myriad UI.
>>    (Mesos' UI should show a task such as "nm.medium." if Myriad
>> attempted to launch a NM).
> Nope, unfortunately it does not.
> 
> What makes me wonder is that I do have a running Myriad, i.e. I can
> access the Myriad UI on port 8192, but it does not show up in Mesos.
> 
> I guess my other problem, not seeing any offers, is a follow-on of this.
> 
>>
>> Mayank - in your cluster, does Mesos UI show Myriad as a registered
>> framework? Does it show if Myriad launched NM tasks? Perhaps a good
>> starting point is to check what are the sizes of the offers Myriad is
>> receiving (Mesos' master log should show this). You can also enable DEBUG
>> logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's
>> log4j.properties.
> I will try and increase the log level to DEBUG.
> 
> 
> Cheers,
> Björn
>>
>>
>>
>> Santosh
>>
>> On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier >> wrote:
>>
>>> Hi Mayank,
>>>
>>> it seems you and I are having similar issues. Let's see whether they can
>>> be traced to the same causes.
>>>
>>>
>>> Best regards,
>>> Björn
>>>
>>> Am 22.02.2016 um 19:33 schrieb Mayank Bansal:
 Hi,

 I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x


 I have followed all the instructions in the wiki

 So the resource manager is up and i am able to launch the web UI of
>>> myriad
 as well however whenever I try to use flex up it launches some tasks and
 put them in to pending queue and after that nothing launches.

 I am attaching below the resourcemanager.out and log snippets.

 Please let me know if I am missing something.

 Thanks,


 *I see some exceptions in resourcemanager.out files those are as follows*


 Feb 22, 2016 6:21:42 PM
 com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
 resolve

 SEVERE: null

 java.lang.IllegalAccessException: Class
 com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
 can not access a member of class javax.ws.rs.core.Response with modifiers
 "protected"

        at 

Re: Pending Flex Up Tasks

[yuliya Feldman]

When you say:
>>> I do see all my expected 8 slaves with all their cores, RAM, and disk.
Do you see it on Mesos Master main UI?  What are those per slave - meaning how 
many CPUs and how much RAM?

  From: Björn Hagemeier <b.hageme...@fz-juelich.de>
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, February 22, 2016 12:35 AM
 Subject: Re: Pending Flex Up Tasks
   
Dear Yuliya,

thank you for the warm welcome.

Am 19.02.2016 um 18:22 schrieb yuliya Feldman:
> Hello Bjorn,
> Welcome to Myriad.
> Few questions that could help to help you.
> 1. Do you just have pending tasks and you never get any active
> ones?
There are only pending tasks, never any active ones. The RM log does not
seem very helpful to me. It records the flexup/flexdown requests and
also the actual killing of flexdown tasks including the fact that I
requested to flexdown more instances than are pending. This is all in
line with what I'd expected from my actions.

> 2. Do you have some active and some pending tasks?3. Are your pending tasks
all NMs?
> If [1] - could you look at RM log (should be accessible from Mesos
console) whether there is enough resources to start the tasks - because
if not they will remain pending.
I do not see any available resources in the Mesos log. This is something
I also noticed in the Mesos Web interface, which does not list any
outstanding offers. Is this related?

I do see all my expected 8 slaves with all their cores, RAM, and disk.

If [2] - we start one NM per hostname
per Myriad framework, so second NM will not start unless first one goes away
> In any case it is best to look at RM log for clues.
Thank you for this hint. It may be worth knowing once I've made it past [1].


Best regards,
Björn
> Thanks,Yuliya
> 
>      From: Björn Hagemeier <b.hageme...@fz-juelich.de>
>  To: Myriad Dev <dev@myriad.incubator.apache.org> 
>  Sent: Friday, February 19, 2016 2:01 AM
>  Subject: Pending Flex Up Tasks
>    
> Hi all,
> 
> I am very new to Myriad, but also to Mesos and Yarn. I am having trouble
> with pending flex up tasks, for which I cannot see any further
> information. Thus, I do not even have the faintest idea where to start
> debugging. I can easily run frameworks in my Mesos cluster, but Yarn NMs
> are a different issue.
> 
> My idea was to use package installations of NM on the slave nodes, if
> that is possible (?). The documentation mentions sth. about remote
> distribution, but from the wording it seems to be more of an option than
> a requirement. Packages have been installed and customized (myriad
> configuration) via Puppet and I'd very much like to stay with this.
> 
> Anyhow, please let me know the possible causes of flex up tasks
> remaining in the pending state. I can easily flex them down and thus
> remove the pending tasks, but I never get them into active.
> 
> Any hint is very much appreciated.
> 
> 
> Best regards,
> Björn
> 


-- 
Dipl.-Inform. Björn Hagemeier
Federated Systems and Data
Juelich Supercomputing Centre
Institute for Advanced Simulation

Phone: +49 2461 61 1584
Fax  : +49 2461 61 6656
Email: b.hageme...@fz-juelich.de
Skype: bhagemeier
WWW  : http://www.fz-juelich.de/jsc

JSC is the coordinator of the
John von Neumann Institute for Computing
and member of the
Gauss Centre for Supercomputing

-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-
-


  

Re: Challenges after MapR 5.1 Upgrade.

[yuliya Feldman]

Hello John,
Did you upgrade to 5.1 or installed new one?
Feels like MapR default properties were not loaded - I need to poke around and 
then I will ask you for additional info
Thanks,Yuliya

  From: John Omernik 
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, April 4, 2016 12:29 PM
 Subject: Challenges after MapR 5.1 Upgrade.
   
I had at one point Myriad working fine in MapR 5.0.  I updated to 5.1, and
repackaged my hadoop tgz for remote distribution and now I have two
problems occurring.

1. At first when I had the mapr direct shuffle enabled per the
yarn-site.xml on the myriad documentaion, node managers would not start,
and would fail with the error below.

2. Once I removed the mapr shuffle from the yarn-site, I got node managers
started however, when I tried to launch a size 0, I got the other error
below. Not sure what's happening here.

Any thoughts would be appreciated. Like I said, this was working with 5.0,
and now doesn't work in 5.1.

Thanks!

John

SHuffle Error

16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed
in state INITED; cause: java.lang.RuntimeException: No class defined for
mapr_direct_shuffle
java.lang.RuntimeException: No class defined for mapr_direct_shuffle
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics
system...


Zero Sized Node Manager Error:

16/04/04 14:22:49 INFO service.AbstractService: Service
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in
state STARTED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from Resourcemanager ,Registration of NodeManager failed, Message
from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from Resourcemanager ,Registration of NodeManager failed, Message
from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed,
Message from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224)
... 6 more


  

Re: Challenges after MapR 5.1 Upgrade.

[yuliya Feldman]

YarnDefaultProperties.java that defines class for mapr_direct_shuffle should be 
there even in 5.0, so nothing new there even if maprfs jar is outdated - could 
you also check that?
Also could you paste content of your yarn-site.xml here?
Thanks,Yuliya

  From: yuliya Feldman <yufeld...@yahoo.com.INVALID>
 To: "dev@myriad.incubator.apache.org" <dev@myriad.incubator.apache.org> 
 Sent: Monday, April 4, 2016 1:43 PM
 Subject: Re: Challenges after MapR 5.1 Upgrade.
   
Hello John,
Did you upgrade to 5.1 or installed new one?
Feels like MapR default properties were not loaded - I need to poke around and 
then I will ask you for additional info
Thanks,Yuliya

      From: John Omernik <j...@omernik.com>
 To: dev@myriad.incubator.apache.org 
 Sent: Monday, April 4, 2016 12:29 PM
 Subject: Challenges after MapR 5.1 Upgrade.
  
I had at one point Myriad working fine in MapR 5.0.  I updated to 5.1, and
repackaged my hadoop tgz for remote distribution and now I have two
problems occurring.

1. At first when I had the mapr direct shuffle enabled per the
yarn-site.xml on the myriad documentaion, node managers would not start,
and would fail with the error below.

2. Once I removed the mapr shuffle from the yarn-site, I got node managers
started however, when I tried to launch a size 0, I got the other error
below. Not sure what's happening here.

Any thoughts would be appreciated. Like I said, this was working with 5.0,
and now doesn't work in 5.1.

Thanks!

John

SHuffle Error

16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed
in state INITED; cause: java.lang.RuntimeException: No class defined for
mapr_direct_shuffle
java.lang.RuntimeException: No class defined for mapr_direct_shuffle
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics
system...


Zero Sized Node Manager Error:

16/04/04 14:22:49 INFO service.AbstractService: Service
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in
state STARTED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from Resourcemanager ,Registration of NodeManager failed, Message
from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN
signal from Resourcemanager ,Registration of NodeManager failed, Message
from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved
SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed,
Message from ResourceManager: NodeManager from  hadoopmapr4.brewingintel.com
doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the
NodeManager.
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224)
... 6 more




  

Re: Challenges after MapR 5.1 Upgrade.

[yuliya Feldman]

Hmmm...
You did not have to specify 

        yarn.nodemanager.aux-services.mapreduce_shuffle.class
        org.apache.hadoop.mapred.ShuffleHandler
    

either
Can you check if maprfs*.jar is on the classpath (should be - otherwise you 
would probably get more weird errors) and it has following class: 
YarnDefaultProperties

Thanks,Yuliya


  From: John Omernik <j...@omernik.com>
 To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com> 
 Sent: Monday, April 4, 2016 2:18 PM
 Subject: Re: Challenges after MapR 5.1 Upgrade.
   
This was a Upgrade from 5.0.  I will post here, note: I have removed the
mapr_shuffle to get node managers to work, however, I am seeing other odd
things, so any help would be appreciated.





    
        yarn.nodemanager.aux-services
        mapreduce_shuffle,myriad_executor
        
    
    
        yarn.resourcemanager.hostname
        myriadprod.marathonprod.mesos
    
    
        yarn.nodemanager.aux-services.mapreduce_shuffle.class
        org.apache.hadoop.mapred.ShuffleHandler
    
    
        yarn.nodemanager.aux-services.myriad_executor.class
        org.apache.myriad.executor.MyriadExecutorAuxService
    
    
        yarn.nm.liveness-monitor.expiry-interval-ms
        2000
    
    
        yarn.am.liveness-monitor.expiry-interval-ms
        1
    
    
        yarn.resourcemanager.nm.liveness-monitor.interval-ms
        1000
    

    
        yarn.nodemanager.resource.cpu-vcores
        ${nodemanager.resource.cpu-vcores}
    
    
        yarn.nodemanager.resource.memory-mb
        ${nodemanager.resource.memory-mb}
    


    
        yarn.scheduler.minimum-allocation-mb
        512
    

    
        yarn.scheduler.minimum-allocation-vcores
        1
    



  

        yarn.nodemanager.address
        ${myriad.yarn.nodemanager.address}
    
    
        yarn.nodemanager.webapp.address
        ${myriad.yarn.nodemanager.webapp.address}
    
    
        yarn.nodemanager.webapp.https.address
        ${myriad.yarn.nodemanager.webapp.address}
    
    
        yarn.nodemanager.localizer.address
        ${myriad.yarn.nodemanager.localizer.address}
    


    
        yarn.resourcemanager.scheduler.class
        org.apache.myriad.scheduler.yarn.MyriadFairScheduler
    


  
    yarn.scheduler.minimum-allocation-vcores
        0
    
    
        yarn.scheduler.minimum-allocation-vcores
        0
    

    
        Who will execute(launch) the containers.
        yarn.nodemanager.container-executor.class
        ${yarn.nodemanager.container-executor.class}
    
    
        The class which should help the LCE handle
resources.

yarn.nodemanager.linux-container-executor.resources-handler.class

${yarn.nodemanager.linux-container-executor.resources-handler.class}
    
    

yarn.nodemanager.linux-container-executor.cgroups.hierarchy

${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}
    
    
        yarn.nodemanager.linux-container-executor.cgroups.mount

${yarn.nodemanager.linux-container-executor.cgroups.mount}
    
    

yarn.nodemanager.linux-container-executor.cgroups.mount-path

${yarn.nodemanager.linux-container-executor.cgroups.mount-path}
    
    
        yarn.nodemanager.linux-container-executor.group
        ${yarn.nodemanager.linux-container-executor.group}
    
    
        yarn.nodemanager.linux-container-executor.path
        ${yarn.home}/bin/container-executor
    

    yarn.http.policy
    HTTP_ONLY



On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman <yufeld...@yahoo.com.invalid>
wrote:

> YarnDefaultProperties.java that defines class for mapr_direct_shuffle
> should be there even in 5.0, so nothing new there even if maprfs jar is
> outdated - could you also check that?
> Also could you paste content of your yarn-site.xml here?
> Thanks,Yuliya
>
>      From: yuliya Feldman <yufeld...@yahoo.com.INVALID>
>  To: "dev@myriad.incubator.apache.org" <dev@myriad.incubator.apache.org>
>  Sent: Monday, April 4, 2016 1:43 PM
>  Subject: Re: Challenges after MapR 5.1 Upgrade.
>
> Hello John,
> Did you upgrade to 5.1 or installed new one?
> Feels like MapR default properties were not loaded - I need to poke around
> and then I will ask you for additional info
> Thanks,Yuliya
>
>      From: John Omernik <j...@omernik.com>
>  To: dev@myriad.incubator.apache.org
>  Sent: Monday, April 4, 2016 12:29 PM
>  Subject: Challenges after MapR 5.1 Upgrade.
>
> I had at one point Myriad working fine in MapR 5.0.  I updated to 5.1, and
> repackaged my hadoop tgz for remote distribution and now I have two
> problems occurring.
>
> 1. At first when I had the mapr direct shuffle enabled per the
> yarn-site.xml on the myriad documentaion, node managers would not start,
> and would fail with the error below.
>
> 2. Once I removed the mapr shuffle from the yarn-site, I got node managers
> started however, when

Re: Observations on Fine Grained Scaling

[yuliya Feldman]

>>> Noticed with jobs with short map tasks (8-12 secs), I rarely got morethan 
>>>two containers per node, I'm curious if I'm not consuming resources
fast enough
Yes, we have noticed that too. I think because heartbeat between Mesos and RM 
is 5 sec (I think) we may not get enough resources with short tasks. It is 
worth investigation though.
Thanks,Yuliya

  From: Darin Johnson 
 To: Dev  
 Sent: Wednesday, April 13, 2016 8:34 AM
 Subject: Observations on Fine Grained Scaling
   
I've been running a number of tests on the Fine Grained scaling aspect on
Myriad.  Here's a few notes:

1. After the patches it seems stable, I'm able to run multiple terasort/pi
jobs and a few scalding jobs without difficulty.
2. Noticed with jobs with short map tasks (8-12 secs), I rarely got more
than two containers per node, I'm curious if I'm not consuming resources
fast enough.  The issue goes away on the reduce side (able to get far
better utilization of offers).  The issue can be lessened by increasing
mapred.splits.min.size and mapred.splits.max.size.  This may be an issue
for things like Hive.

Darin


  

Re: Sync today?

[yuliya Feldman]

same - can not connect

  From: Darin Johnson 
 To: Dev  
 Sent: Wednesday, July 13, 2016 9:09 AM
 Subject: Sync today?
   
Couldn't connect


   

Re: Sync today?

[yuliya Feldman]

I guess nobody from Mesosphere today to start hangout.

  From: yuliya Feldman <yufeld...@yahoo.com.INVALID>
 To: "dev@myriad.incubator.apache.org" <dev@myriad.incubator.apache.org> 
 Sent: Wednesday, July 13, 2016 9:10 AM
 Subject: Re: Sync today?
   
same - can not connect

      From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org> 
 Sent: Wednesday, July 13, 2016 9:09 AM
 Subject: Sync today?
  
Couldn't connect


  

  

Re: Sync tomorrow?

[yuliya Feldman]

I won't be there tomorrow - will have a meeting

  From: Adam Bordelon 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, August 9, 2016 5:32 PM
 Subject: Re: Sync tomorrow?
   
Yeah, I'll be there. Don't have a lot of news though.

On Tue, Aug 9, 2016 at 5:21 PM, Darin Johnson 
wrote:

> Trying to plan my day tomorrow.
>


   

Re: Myraid Slack

[yuliya Feldman]

no luck joining so far

  From: Ken Sipe 
 To: dev@myriad.incubator.apache.org 
 Sent: Wednesday, June 29, 2016 9:04 AM
 Subject: Re: Myraid Slack
   
I am on
> On Jun 29, 2016, at 11:04 AM, Darin Johnson  wrote:
> 
> Having issues getting on, is anybody else able to connect?
> On Jun 28, 2016 10:34 PM, "Adam Bordelon"  wrote:
> 
>> (Next dev sync is tomorrow, 9am Pacific time)
>> 
>> On Tue, Jun 28, 2016 at 12:32 PM, Darin Johnson 
>> wrote:
>> 
>>> We also have a dev sync every other Wednesday via Google Hangouts:
>>> https://plus.google.com/hangouts/_/mesosphere.io/myriad
>>> 
>>> Darin
>>> 
>>> On Thu, Jun 23, 2016 at 3:01 AM, Swapnil Daingade <
>>> swapnil.daing...@gmail.com> wrote:
>>> 
 Hi Sam,
 
 Myriad is a fairly new project. The IPMC vote for Myriad 0.2 just
>> passed
 this week.
 Given we are early in the incubation stage, its not uncommon for one or
 two vendors
 to back the project.
 
 I'll let other community members talk about their experiences deploying
 Myriad
 but Its really great that you are considering deploying Myriad in
 production.
 Your feedback will definitely help shape the road map for Myriad going
 forward.
 
 Regards
 Swapnil
 
 
 
 On 06/22/2016 11:24 PM, Sam Chen wrote:
 
> Hi Swapnil,
> MapR is one company to give Myriad support, right?  Any reference ?
> 
> Regards,
> Sam
> 
> 
> Sent from my iPhone
> 
> On Jun 23, 2016, at 10:39 AM, Swapnil Daingade <
>> swapnil.daing...@gmail.com> wrote:
>> 
>> MapR supports Myriad 0.1 currently
>> 
>> https://www.mapr.com/products/whats-included
>> https://www.mapr.com/products/product-overview/apache-myriad
>> 
>> Regards
>> Swapnil
>> 
>> 
>> On Wed, Jun 22, 2016 at 6:51 PM, Sam Chen 
>>> wrote:
>>> 
>>> Hi Darin,
>>> Thanks for you reply. Makes sense to use Slack. Btw, we are going to
>>> use
>>> Myriad in production, any company have capability to support this ?
>>> And
>>> is
>>> there any reference in production ?
>>> 
>>> Regards,
>>> Sam
>>> 
>>> Sent from my iPhone
>>> 
>>> On Jun 23, 2016, at 2:30 AM, Darin Johnson >> 
> 
 wrote:
 
 Sam,
 
 I don't believe so.  But we do have an IRC channel #myriad on
>>> FreeNode.
 
>>> I
>>> 
 know the mesosphere guys set up slackbots to interact with it.  I'm
 only
 there occasionally or by appointment. I did notice Kudu now uses
>>> slack,
 
>>> so
>>> 
 maybe slack makes more sense than IRC these days, or Gitter Chat.
 
 Darin
 
 On Wed, Jun 22, 2016 at 1:55 AM, Sam Chen <
>> rc...@linkernetworks.com>
> 
 wrote:
>>> 
 Guys,
> Do we have Slack for Myraid?
> 
> Regards ,
> Sam
> 
> Sent from my iPhone
> 
 
>>> 
>>> 
> 
> 
 
>>> 
>> 


  

Re: Myraid Slack

[yuliya Feldman]

nope - hanging in "Requesting to join" :(

  From: Ken Sipe <k...@mesosphere.io>
 To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com> 
 Sent: Wednesday, June 29, 2016 9:11 AM
 Subject: Re: Myraid Slack
   
https://plus.google.com/hangouts/_/mesosphere.io/myriad 
<https://plus.google.com/hangouts/_/mesosphere.io/myriad>


> On Jun 29, 2016, at 11:08 AM, yuliya Feldman <yufeld...@yahoo.com.INVALID> 
> wrote:
> 
> no luck joining so far
> 
>      From: Ken Sipe <k...@mesosphere.io>
> To: dev@myriad.incubator.apache.org 
> Sent: Wednesday, June 29, 2016 9:04 AM
> Subject: Re: Myraid Slack
> 
> I am on
>> On Jun 29, 2016, at 11:04 AM, Darin Johnson <dbjohnson1...@gmail.com> wrote:
>> 
>> Having issues getting on, is anybody else able to connect?
>> On Jun 28, 2016 10:34 PM, "Adam Bordelon" <a...@mesosphere.io> wrote:
>> 
>>> (Next dev sync is tomorrow, 9am Pacific time)
>>> 
>>> On Tue, Jun 28, 2016 at 12:32 PM, Darin Johnson <dbjohnson1...@gmail.com>
>>> wrote:
>>> 
>>>> We also have a dev sync every other Wednesday via Google Hangouts:
>>>> https://plus.google.com/hangouts/_/mesosphere.io/myriad
>>>> 
>>>> Darin
>>>> 
>>>> On Thu, Jun 23, 2016 at 3:01 AM, Swapnil Daingade <
>>>> swapnil.daing...@gmail.com> wrote:
>>>> 
>>>>> Hi Sam,
>>>>> 
>>>>> Myriad is a fairly new project. The IPMC vote for Myriad 0.2 just
>>> passed
>>>>> this week.
>>>>> Given we are early in the incubation stage, its not uncommon for one or
>>>>> two vendors
>>>>> to back the project.
>>>>> 
>>>>> I'll let other community members talk about their experiences deploying
>>>>> Myriad
>>>>> but Its really great that you are considering deploying Myriad in
>>>>> production.
>>>>> Your feedback will definitely help shape the road map for Myriad going
>>>>> forward.
>>>>> 
>>>>> Regards
>>>>> Swapnil
>>>>> 
>>>>> 
>>>>> 
>>>>> On 06/22/2016 11:24 PM, Sam Chen wrote:
>>>>> 
>>>>>> Hi Swapnil,
>>>>>> MapR is one company to give Myriad support, right?  Any reference ?
>>>>>> 
>>>>>> Regards,
>>>>>> Sam
>>>>>> 
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>> On Jun 23, 2016, at 10:39 AM, Swapnil Daingade <
>>>>>>> swapnil.daing...@gmail.com> wrote:
>>>>>>> 
>>>>>>> MapR supports Myriad 0.1 currently
>>>>>>> 
>>>>>>> https://www.mapr.com/products/whats-included
>>>>>>> https://www.mapr.com/products/product-overview/apache-myriad
>>>>>>> 
>>>>>>> Regards
>>>>>>> Swapnil
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jun 22, 2016 at 6:51 PM, Sam Chen <rc...@linkernetworks.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Darin,
>>>>>>>> Thanks for you reply. Makes sense to use Slack. Btw, we are going to
>>>> use
>>>>>>>> Myriad in production, any company have capability to support this ?
>>>> And
>>>>>>>> is
>>>>>>>> there any reference in production ?
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Sam
>>>>>>>> 
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>> On Jun 23, 2016, at 2:30 AM, Darin Johnson <dbjohnson1...@gmail.com
>>>> 
>>>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Sam,
>>>>>>>>> 
>>>>>>>>> I don't believe so.  But we do have an IRC channel #myriad on
>>>> FreeNode.
>>>>>>>>> 
>>>>>>>> I
>>>>>>>> 
>>>>>>>>> know the mesosphere guys set up slackbots to interact with it.  I'm
>>>>>>>>> only
>>>>>>>>> there occasionally or by appointment. I did notice Kudu now uses
>>>> slack,
>>>>>>>>> 
>>>>>>>> so
>>>>>>>> 
>>>>>>>>> maybe slack makes more sense than IRC these days, or Gitter Chat.
>>>>>>>>> 
>>>>>>>>> Darin
>>>>>>>>> 
>>>>>>>>> On Wed, Jun 22, 2016 at 1:55 AM, Sam Chen <
>>> rc...@linkernetworks.com>
>>>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Guys,
>>>>>>>>>> Do we have Slack for Myraid?
>>>>>>>>>> 
>>>>>>>>>> Regards ,
>>>>>>>>>> Sam
>>>>>>>>>> 
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
> 
> 


  

Re: Website is updated, 0.2.0 is official!

[yuliya Feldman]

Great news.
Thank you Darrin and everybody for the 2.0 release!!!

  From: Darin Johnson 
 To: Dev  
 Sent: Wednesday, June 29, 2016 2:05 PM
 Subject: Website is updated, 0.2.0 is official!
   
http://myriad.apache.org/

Tell your friends!


   

Do we have a call today? I am not able to get it

[yuliya Feldman]

Re: Do we have a call today? I am not able to get it

[yuliya Feldman]

ok
Unfortunately I have to leave now :(

  From: Adam Bordelon <a...@mesosphere.io>
 To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com> 
 Sent: Wednesday, September 21, 2016 9:14 AM
 Subject: Re: Do we have a call today? I am not able to get it 
   
I'm stuck in another meeting, but Miguel is waiting in
https://hangouts.google.com/hangouts/_/mesosphere.io/myriad

On Wed, Sep 21, 2016 at 9:12 AM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

>
>


   

Do we have sync up today, or I am too late?

[yuliya Feldman]

[DISCUSS] handling roles in Myriad code

[yuliya Feldman]

Hello there,
I wanted to discuss current handling of roles in Myriad code. Specifically on 
"master" branch. Most likely due to heavy refactoring.
As far as I can see we try to handle presence or absence of a role on a 
resource(s) based on the fact that framework may or may not have a role.On the 
other hand we always set framework role to "*" - which means it will always 
have a role, just that role will be "default".
So far I encountered couple of bugs related to roles in RangeResource related 
to ports and inability to spin up NodeManagers, as no ports were assigned 
because of the fact how we handle roles.
I would like @Adam and other Mesosphere folks to comment on how should we 
handle relationship between frameworkRole and resource role(s)
Thanks,Yuliya

Re: [DISCUSS] Using defined ports versus provided by Mesos

[yuliya Feldman]

To add to the previous - in case of work preserving NodeManager we would also 
need consistent port assignment in case NM restarts.

  From: yuliya Feldman <yufeld...@yahoo.com>
 To: Dev <dev@myriad.incubator.apache.org> 
 Sent: Wednesday, October 19, 2016 12:34 PM
 Subject: [DISCUSS] Using defined ports versus provided by Mesos
   
Hello there,
I wanted to discuss usage of hardcoded ports instead of ones provided by Mesos 
as a resource.On master branch - probably due to significant code refactoring 
we pretty much do NOT allow any ports (range resource) that is NOT provided by 
Mesos.
Earlier on we could have "reserved" ports (of course they might be taken for 
whatever reason) and do not reserve them via resource "materialization" to 
Mesos, but now we don't.It is quite inconvenient sometimes for at least JHS.
Wanted to get everybody's opinion on what we should allow.
Thanks,Yuliya   

   

[DISCUSS] Using defined ports versus provided by Mesos

[yuliya Feldman]

Hello there,
I wanted to discuss usage of hardcoded ports instead of ones provided by Mesos 
as a resource.On master branch - probably due to significant code refactoring 
we pretty much do NOT allow any ports (range resource) that is NOT provided by 
Mesos.
Earlier on we could have "reserved" ports (of course they might be taken for 
whatever reason) and do not reserve them via resource "materialization" to 
Mesos, but now we don't.It is quite inconvenient sometimes for at least JHS.
Wanted to get everybody's opinion on what we should allow.
Thanks,Yuliya   

Re: [DISCUSS] handling roles in Myriad code

[yuliya Feldman]

I am not sure we should care about role being set or not, what if in the future 
we will have multiple rolesNot even sure if presence/absence of role should 
play role (no pun intended :) ).

  From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman 
<yufeld...@yahoo.com> 
 Sent: Wednesday, October 19, 2016 7:17 PM
 Subject: Re: [DISCUSS] handling roles in Myriad code
   
Ah so if I understand correctly, if frameworkRole='*' is present in the
config, it's handled as thought it's the framework role.  I believe when I
was testing I was using frameworkRole="test" or commenting out
frameworkRole="test".  It looks as though in MyriadConfiguration,
getFrameworkRole now returns "*" even if not set.

Seems like we should be able to add a check like r.hasRole() &&
r.getRole().equals(role)
&& !role.equals("*") in a few places. Though it may be better
to pass think about a better approach here.

Darin

On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> Hello Darrin,
> I kind of see the point regarding JHS ports. May be there is truth to it.
> Regarding my issues with role/no role.
> I had this issue for NMs with random ports (not hardcoded), as it has
> different code path when role is present and when it is not. My impression
> those are bugs.
> I am happy to point you to the places in the code that caused issues on
> master (at least for me).[1] does not increment numDefaultValues if role is
> set (which is always set), subsequently [2] has issues[3] same thing -
> fills out list only if there is no role, but again it is always there, just
> set to "*"
>
>
> Regarding:>>> To handle nodemanager persistence I think we should work
> with Klaus's PR's to get thecorrect ports, though we'll need to use some
> disk persistence as well to
> keep the NM state.
> Disk persistence won't help here (not even sure NM has much state to
> persist - even if it does it should be taken care by YARN), as containers
> have to reconnect to NM after it restarts, so they have to know RPC port.
> Thanks,Yuliya
> [1] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L85
> [2] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L128
>
> [3] https://github.com/apache/incubator-myriad/blob/master/
> myriad-scheduler/src/main/java/org/apache/myriad/scheduler/resource/
> RangeResource.java#L140
>
>
>      From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> yufeld...@yahoo.com>
>  Sent: Wednesday, October 19, 2016 6:04 PM
>  Subject: Re: [DISCUSS] handling roles in Myriad code
>
> Yuyiya,
>
> Yes on master a lot of refactoring was done, in particular you specify
> ports other than 0 in the myriad-default.yaml, it will only return those
> ports (not random ones).  This was done in part because the we were
> attempting the use the JHS on a port like 32001, but it the port was
> already in use by another app and hence the port wasn't offered myriad was
> still launching the JHS only to have it crash.
>
> If you want to use static ports you can just not put anything in the
> myriad-default.yaml and configure the yarn-site.xml and mapred-site.xml as
> usual (they should be outside the range mesos offers).  To handle
> nodemanager persistence I think we should work with Klaus's PR's to get the
> correct ports, though we'll need to use some disk persistance as well to
> keep the NM state.
>
> As for a bug in NM's getting zero ports could you send a copy of your
> configuration and I'll try to recreate the problem?
>
> On Wed, Oct 19, 2016 at 3:29 PM, yuliya Feldman
> <yufeld...@yahoo.com.invalid
> > wrote:
>
> > Hello there,
> > I wanted to discuss current handling of roles in Myriad code.
> Specifically
> > on "master" branch. Most likely due to heavy refactoring.
> > As far as I can see we try to handle presence or absence of a role on a
> > resource(s) based on the fact that framework may or may not have a
> role.On
> > the other hand we always set framework role to "*" - which means it will
> > always have a role, just that role will be "default".
> > So far I encountered couple of bugs related to roles in RangeResource
> > related to ports and inability to spin up NodeManagers, as no ports were
> > assigned because of the fact how we handle roles.
> > I would like @Adam and other Mesosphere folks to comment on how should we
> > handle relationship between frameworkRole and resource role(s)
> > Thanks,Yuliya
>
>
>
>


   

Re: Hangout today

[yuliya Feldman]

Sorry - could not attend - was on the road

  From: Darin Johnson 
 To: Dev  
 Sent: Wednesday, November 16, 2016 9:33 AM
 Subject: Hangout today
   
Missed due to another meeting going long.  Anyone else attend?


   

Re: [DISCUSS] handling roles in Myriad code

[yuliya Feldman]

We clearly need a word from Adam, Mohit, Ken
My impression is that Myriad will not get any resources that are not specific 
to the role it has (or entitled to), so we may not need much roles manipulation 
in Myriad code.
Just my 2c of gut feelings :)
Thanks,Yuliya

  From: Darin Johnson <dbjohnson1...@gmail.com>
 To: Dev <dev@myriad.incubator.apache.org> 
 Sent: Friday, October 28, 2016 2:54 PM
 Subject: Re: [DISCUSS] handling roles in Myriad code
   
Any word from Adam or Mohit?

On Oct 20, 2016 12:36 AM, "Klaus Ma" <klaus1982...@gmail.com> wrote:

> I can help on this discussion; I used to be Mesos contributor for a year
> :).
>
> Mesos allocate regular resources based on role by DRF; and role is also
> used for reservation & quotas. So, the framework (like Myriad), may get two
> kind of resources: "*" or "myriad-s role". When Myriad launch tasks, it can
> not overuse any kind of resources: for example, if Myarid got offers:
> cpu(*):1;cpu(myriad):1, Myriad can not launch tasks by cpu(*):2 which will
> be rejected by Mesos master.
>
> Thanks
> Klaus
>
>
> On Thu, Oct 20, 2016 at 12:10 PM Yuliya <yufeld...@yahoo.com.invalid>
> wrote:
>
> > I really would like Mesosphere guys to comment here. I had a chat with
> > Adam today morning and I did not get the same impression
> >
> > Thanks,
> > Yuliya
> >
> > > On Oct 19, 2016, at 8:50 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> > >
> > > We use roles extensively to ensure different frameworks can (or can't)
> > get
> > > resources via mechanisms such as reserved resorces and quotas.  Also if
> > you
> > > don't pay attention you can miss a lot of the resources you're given.
> I
> > > wish it was we didn't have to do all the book keeping our selves, but I
> > > suppose there are good reasons for delegating it to the framework, for
> > > instance we can choose when to fave a reserved vs a default resource.
> > >
> > > On Wed, Oct 19, 2016 at 11:30 PM, yuliya Feldman <
> > > yufeld...@yahoo.com.invalid> wrote:
> > >
> > >> I am not sure we should care about role being set or not, what if in
> the
> > >> future we will have multiple rolesNot even sure if presence/absence of
> > role
> > >> should play role (no pun intended :) ).
> > >>
> > >>      From: Darin Johnson <dbjohnson1...@gmail.com>
> > >> To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> > >> yufeld...@yahoo.com>
> > >> Sent: Wednesday, October 19, 2016 7:17 PM
> > >> Subject: Re: [DISCUSS] handling roles in Myriad code
> > >>
> > >> Ah so if I understand correctly, if frameworkRole='*' is present in
> the
> > >> config, it's handled as thought it's the framework role.  I believe
> > when I
> > >> was testing I was using frameworkRole="test" or commenting out
> > >> frameworkRole="test".  It looks as though in MyriadConfiguration,
> > >> getFrameworkRole now returns "*" even if not set.
> > >>
> > >> Seems like we should be able to add a check like r.hasRole() &&
> > >> r.getRole().equals(role)
> > >> && !role.equals("*") in a few places. Though it may be better
> > >> to pass think about a better approach here.
> > >>
> > >> Darin
> > >>
> > >> On Wed, Oct 19, 2016 at 9:28 PM, yuliya Feldman
> > >> <yufeld...@yahoo.com.invalid
> > >>> wrote:
> > >>
> > >>> Hello Darrin,
> > >>> I kind of see the point regarding JHS ports. May be there is truth to
> > it.
> > >>> Regarding my issues with role/no role.
> > >>> I had this issue for NMs with random ports (not hardcoded), as it has
> > >>> different code path when role is present and when it is not. My
> > >> impression
> > >>> those are bugs.
> > >>> I am happy to point you to the places in the code that caused issues
> on
> > >>> master (at least for me).[1] does not increment numDefaultValues if
> > role
> > >> is
> > >>> set (which is always set), subsequently [2] has issues[3] same thing
> -
> > >>> fills out list only if there is no role, but again it is always
> there,
> > >> just
> > >>> set to "*"
> > >>>
> > >>>
> > >>> Regarding:&

will miss standup today - doc appt.

[yuliya Feldman]

Re: Sync today?

[yuliya Feldman]

sorry - forgot

  From: Darin Johnson 
 To: Dev  
 Sent: Wednesday, March 8, 2017 9:06 AM
 Subject: Sync today?
   
Tried joining.


   

Re: [Vote] Retire Myriad

[yuliya Feldman]

Sorry for chiming in late. I was out of town.
I think we should keep the project going if there is an activity on the project.
I can definitely contribute time to vet the release, review and merge. I may 
not be able to actively work on the project as it is not inline with my current 
work schedule.
Thanks,Yuliya

  From: Darin Johnson 
 To: Dev  
 Sent: Tuesday, June 20, 2017 11:19 AM
 Subject: Re: [Vote] Retire Myriad
   
Swapnil: I only counted myself and Adam, if there's two additional
committers who are willing to vet the release AND merge commits I'll
consider changing my vote.

Darin

On Tue, Jun 20, 2017 at 12:54 PM, Swapnil Daingade <
swapnil.daing...@gmail.com> wrote:

> I am trying to understand what changed since the last discussion.
>
> If I remember correctly, Ted asked for 3-5 committers to vet the next
> release.
> 4 committers said they were willing. Did I miss something ?
>
> Regards
> Swapnil
>
>
>
> On Tue, Jun 20, 2017 at 7:35 AM, John Yost  wrote:
>
> > +1
> >
> > On Tue, Jun 20, 2017 at 10:22 AM, Klaus Ma 
> wrote:
> >
> > > +1
> > >
> > > On Tue, Jun 20, 2017 at 10:13 PM Brandon Gulla <
> gulla.bran...@gmail.com>
> > > wrote:
> > >
> > > > +1 (if the vote is open to non-comitters)
> > > >
> > > >
> > > >
> > > > On Tue, Jun 20, 2017 at 9:22 AM, Darin Johnson 
> > > wrote:
> > > >
> > > > > Based on previous discussions it seems the best course of action.
> > I'm
> > > > > holding the vote open for 3 business days.
> > > > >
> > > > > I'm +1 binding.
> > > > >
> > > > > Darin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Brandon
> > > >
> > > --
> > >
> > > Regards,
> > > 
> > > Da (Klaus), Ma (马达), PMP® | Software Architect
> > > IBM Platform Development & Support, STG, IBM GCG
> > > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> > >
> >
>

   

Re: Report

[yuliya Feldman]

 Feels like project is dead, Wonder if anybody at MapR and/or Mesosphere 
interested in continuing the project?
On Tuesday, October 10, 2017, 4:42:56 AM PDT, Ted Dunning 
 wrote:  
 
 The incubator report for myriad seems to be missing.

Is anybody up for filing it? (probably too late for this month's report)
  

Re: Myriad Sync - 11/15/17

[yuliya Feldman]

 Could it be a bit more advance notice? Or I missed that notice?
On Wednesday, November 15, 2017, 10:03:37 AM PST, mohit soni 
 wrote:  
 
 Hi

Please join:
https://hangouts.google.com/hangouts/_/uiq2idcclngybc3hlifznfdxi4e for
today's Myriad Sync.

Best
Mohit
  

Re: Apache Slider and Apache Myriad

[yuliya Feldman]

 Are you suggesting using Myriad as meta-scheduler for Mesos?
If yes, then:1. Why would one need Mesos or Myriad at all - use YARN for 
everything?2. Mesos scheduler is much more robust then YARN scheduler as it's 
based on the offers - it was done having multiple subsystems in mind unlike YARN
AFAIK Marathon serves very thin purpose in Myriad - literally just to spin up 
Myriad Master
Just my 2c.
Thanks,Yuliya
On Tuesday, April 24, 2018, 10:56:37 AM PDT, Javi Roman 
 wrote:  
 
 Hi!

Thinking a little more deeply this YARN feature (Hadoop YARN Service
framework) could affect Apache Myriad regarding new development goals.
Apache Myriad based on YARN 3.x could be a possible substitute for
general purpose meta-scheduling Mesos frameworks such as Marathon or
Apache Aurora.  We can rewrite the Myriad UI in order to allow run
YARN native applications, Docker containers or regular applications.

It's just a crazy idea: DC/OS + Apache Myriad (natively installed as
systemd unit) instead of Marathon, working as meta-scheduler for
Mesos.

What do you think?
--
Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info


On Mon, Apr 23, 2018 at 8:02 AM, Javi Roman  wrote:
> Hi Darin,
>
> You are right Slider is a kind of framework for long running
> applications into YARN, and it's a way of running applications on YARN
> without changing the application code, there is no need to develop a
> custom Application Master or other YARN code. Apparently you can run
> unmodified applications on YARN via Slider.
>
> This shift towards Hadoop code (instead of separate project, incubator
> slider) doesn't affect Myriad at all. However I was thinking any kind
> of collaboration with Slider project, and this collaboration is
> unclear right now, because of fuzzy continuity of Slider.
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson  
> wrote:
>> Javi can you explain how this effects myriad?  My understanding is that
>> slider is similar to marathon but for yarn.
>>
>> On Sat, Apr 21, 2018, 2:23 AM Javi Roman  wrote:
>>
>>> FYI
>>>
>>> Regarding the recent conversation about YARN applications development
>>> via Apache Slider, as a different approach for creating data services
>>> applications on Apache Mesos (via Myriad,) "article 4" here [1], will
>>> probably have to wait until the resolution of this discussion at
>>> Apache Slider list [2]. Basically the Slider community is discussing
>>> about the retirement of the project, because the Slider core is being
>>> included in Hadoop 3 [3].
>>>
>>> [1] https://goo.gl/RVkFUF
>>> [2] https://goo.gl/CPR2TF
>>> [3] https://issues.apache.org/jira/browse/YARN-5079
>>>
>>> --
>>> Javi Roman
>>>
>>> Twitter: @javiromanrh
>>> GitHub: github.com/javiroman
>>> Linkedin: es.linkedin.com/in/javiroman
>>> Big Data Blog: dataintensive.info
>>>
  

Re: [Proposal] Migrate to gitbox

[yuliya Feldman]

 Thank you Javi for the information.
+1 from me.
On Thursday, September 13, 2018, 10:45:04 PM PDT, Javi Roman 
 wrote:  
 
 Yuliya closing PRs is only possible by the owner of the PR or by means
a commit by other author. When a PR is abandoned by the author (we
have some of them) the procedure is open a JIRA ticket to INFRA for
closing them. With GitBox we can use GitHub interface for this kind of
tasks among others in guess.

IMHO changing to GitBox is an enhance in our workflow, even our
"father" project Apache Mesos migrated to GitBox some days ago.

We can wait a few days a give a result vote for this change, we have 3
+1 votes, and one -1 (your vote I understand) by now.

Greetings.
--
Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
Apache Id: javiroman

On Thu, Sep 13, 2018 at 7:30 PM yuliya Feldman
 wrote:
>
>  Hello there,
> I am trying to figure out what are the real advantages for Myriad in moving 
> it to gitbox
>  - JIRA linking can be enabled on request to the Apache Infra team, which
>  will automatically link a PR with its corresponding JIRA in
> I believe it is already there
>  - It will make our code review process cleaner and more efficient when
>  compared to the Review Board.
> I believe we are using github itself for review.
> - Opens up a lot of opportunities on leveraging the rich set of github
>  APIs and web hooks to develop tools that can help monitor the open PRs,
>  trigger a jenkins job to run unit tests, etc..
> Which ones?
> Thanks,Yuliya
>    On Thursday, September 13, 2018, 7:26:35 AM PDT, JP Gilaberte 
> wrote:
>
>  Hi all,
>
> I am starting this vote (copy from Ambary folks) to migrate Myriad
> project from git-wip
> repository to gitbox, which allows a deeper integration with github
> features. Moving to gitbox will allow committers to merge, close or edit
> pull requests in the github UI.
>
> Other advantages:
>
>  - JIRA linking can be enabled on request to the Apache Infra team, which
>  will automatically link a PR with its corresponding JIRA in
>  https://issues.apache.org/jira
>  - It will make our code review process cleaner and more efficient when
>  compared to the Review Board.
>  - Opens up a lot of opportunities on leveraging the rich set of github
>  APIs and web hooks to develop tools that can help monitor the open PRs,
>  trigger a jenkins job to run unit tests, etc..
>
>  [ ] +1, Migrate Myriad repository to gitbox
>  [ ] -1, Keep git-wip + github read-only mirror
>
> Regards,
> JP Gilaberte
>
  

Re: [Proposal] Migrate to gitbox

[yuliya Feldman]

 Hello there,
I am trying to figure out what are the real advantages for Myriad in moving it 
to gitbox
  - JIRA linking can be enabled on request to the Apache Infra team, which
  will automatically link a PR with its corresponding JIRA in
I believe it is already there 
  - It will make our code review process cleaner and more efficient when
  compared to the Review Board.
I believe we are using github itself for review.
- Opens up a lot of opportunities on leveraging the rich set of github
  APIs and web hooks to develop tools that can help monitor the open PRs,
  trigger a jenkins job to run unit tests, etc..
Which ones?
Thanks,Yuliya
On Thursday, September 13, 2018, 7:26:35 AM PDT, JP Gilaberte 
 wrote:  
 
 Hi all,

I am starting this vote (copy from Ambary folks) to migrate Myriad
project from git-wip
repository to gitbox, which allows a deeper integration with github
features. Moving to gitbox will allow committers to merge, close or edit
pull requests in the github UI.

Other advantages:

  - JIRA linking can be enabled on request to the Apache Infra team, which
  will automatically link a PR with its corresponding JIRA in
  https://issues.apache.org/jira
  - It will make our code review process cleaner and more efficient when
  compared to the Review Board.
  - Opens up a lot of opportunities on leveraging the rich set of github
  APIs and web hooks to develop tools that can help monitor the open PRs,
  trigger a jenkins job to run unit tests, etc..

  [ ] +1, Migrate Myriad repository to gitbox
  [ ] -1, Keep git-wip + github read-only mirror

Regards,
JP Gilaberte
  

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 Hello,
Please see inline.
Thanks,Yuliya
On Thursday, September 26, 2019, 12:54:47 AM PDT, Oscar Fernandez 
 wrote:  
 
 Hi,

*1. *Myriad Scheduler would be responsible to register with Mesos and, on
demand, bring up Yarn clusters (RMs and NMs) and manage its resources.
>>> OK 

*2. *Yes, the idea is that Myriad will control NMs for all YARN clusters
that the user wants to deploy. Obviously the web UI should be updated and
the logic to handle the state of several clusters implemented.
>>> UI is the least of a concern here. I am sure UI and API would be fine with 
>>> multiple clusters.
* a.* NMs should come and go on demand, from the UI and API. In the future,
maybe we can implement some auto-scaling with the available resources in
the Mesos cluster, which is on the roadmap.
* b.* IMO NMs shouldn't be permanent, else, we miss the scaling feature.
>>> Sure
* c*. RMs will be permanent until YARN cluster shutdown, as the RM is
needed for the YARN cluster to run properly. Also, Myriad should keep track
of where the RM for each Yarn cluster is running in order to configure the
NM for that cluster.

*3. *I'm not sure I understand this question, what you mean with "isn't it
too much.."? This feature should be implemented and defined, as the current
state of Myriad doesn't allow any of this.
>>> What I meant is that your MyriadScheduler may need to deal with a lot of 
>>> resources it's schedulingyou will need to make sure it's not a bottleneck 
>>> and you don't starve one cluster in favor of the other.It will have to be 
>>> really good 2nd level scheduler - that's why I was saying it's not as 
>>> trivial as getting offers from Mesos, keeping track of them and start NMs 
>>> as you please. Even today I believe we need to make sure locality is taken 
>>> care of.

We have enabled comments in the doc, maybe you can help us make this pros
and cons list with the new design.
>>> Thanks

Thank you, all this help is appreciated

On Wed, Sep 25, 2019 at 6:11 PM yuliya Feldman 
wrote:

>  Hello,
> Thank you for the diagrams - it helps. Could you also enable comments in
> your doc?
> Few thoughts:1.  Myriad Scheduler is wonderful - but it's yet another
> scheduler you need to deal with - or you plan to have current
> MyriadScheduler that sits in RM and use it instead?2. Is Myriad scheduler
> going to control NMs for all YARN clusters?    a. How NMs will come and
> go?    b. Are they going to be permanent?    c. I assume RMs will be
> permanent until cluster shutdown, right?3. If NMs will not be permanent -
> isn't it too much for upper level Myriad Scheduler to deal with all of them?
>
> Also could you please list cons - pros are great, but it's better to have
> cons as well.
> Thanks,Yuliya
>
>
>    On Wednesday, September 25, 2019, 12:30:20 AM PDT, Oscar Fernandez <
> oscarf...@apache.org> wrote:
>
>  Hi,
>
> I've made a diagram to represent the new proposed design in order to
> support Yarn as a service with some of the pros:
>
> https://docs.google.com/document/d/15X0-zSu0G0BDpWyndRhbvAJCXLtAbkNA45wQ_xVKOKQ
>
> Thank you for all your comments and help
>
> On Tue, Sep 24, 2019 at 8:57 PM Javi Roman 
> wrote:
>
> > Honestly your opinion is welcome, this kind of discussions are great
> > in this small traffic dev list ;-)
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> > Apache Id: javiroman
> >
> > On Tue, Sep 24, 2019 at 8:55 PM yuliya Feldman
> >  wrote:
> > >
> > >  I am not saying it's crazy. I was voicing my opinion. Isn't it what
> was
> > the purpose of the discussion?
> > > It's definitely great to have UI that manages all the YARN clusters,
> but
> > it's not like UI/Web service has to be coupled/collocated with any of the
> > Myriad particular YARN version daemons.
> > > It's great if you would provide write up with pros and cons for your
> > approach or any alternative approaches.
> > >
> > >
> > >    On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman <
> > jroman.espi...@gmail.com> wrote:
> > >
> > >  On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman
> > >  wrote:
> > > >
> > > >  Hello,
> > > > Again I apologize for the late reply.
> > > > I think I replied to the thread, but will add more direct notes here
> > > > What you are proposing is to have yet another daemon that would start
> > Yarn Clusters on demand within Mesos framework.
> > > > Meaning - it would be another laye

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 It's true with one "but" you need a yet another scheduler in your service and 
it's not a trivial feat - if you want it to be fulfilling the purpose, 
otherwise Marathon as good as it is. As far as I remember Marathon has simple 
FIFO one (may be it evolved though).
In any case - it was my opinion :). You guys know better and closer to it.
Thanks,Yuliya


On Tuesday, September 24, 2019, 11:32:32 AM PDT, Javi Roman 
 wrote:  
 
 On Tue, Sep 24, 2019 at 7:48 PM yuliya Feldman
 wrote:
>
>  >>> I guess we are talking about the concept of multi-service scheduler

Are you proposing to have a multi service scheduler? I thought that's
what Mesos is for?What am I missing here?

Yes is common in Mesos (IMHO): Marathon is a multi-service scheduler,
Apache Aurora too. Please take a look here:

https://mesosphere.github.io/dcos-commons/multi-service/


> On Tue, Sep 24, 2019 at 5:40 PM Yuliya  wrote:
> >
> > Hello there,
> >
> > Sorry for late reply.
> >
> > Frankly speaking I don’t know original motivation , I would probably say 
> > that it started organically as a need.
> >
> > What do you mean by starting yarn from myriad? I believe myriad daemons 
> > encompass some functionality of yarn daemons, namely rm and nm, so it’s not 
> > yarn that starts myriad, but myriad daemons that play role of yarn daemons.
> >  Unless I am missing something here.
> >
> > Are you proposing to have the same myriad daemons starting different 
> > versions of yarn?
> >  I would consider different set of docker containers built with different 
> >versions of yarn would be better decoupling. I am open for discussion though.
> >
> > Thanks,
> > Yuliya
> >
> >
> >
> > > On Sep 15, 2019, at 10:35 PM, Javi Roman  wrote:
> > >
> > > Hi Oscar,
> > >
> > > I have to say I don't know the initial motivation of this design. You
> > > are right the way of starting Myriad, strongly coupled to YARN is a
> > > little bit weird.
> > > Because of lack of activity of the initial committers, this is a
> > > question that probably we never get a clear answer.
> > >
> > > By the way, your proposal, according with MYRIAD-295 is, from my
> > > understanding, the right way to go ahead with the project.
> > >
> > > This new design is totally aligned with the further Myriad UI design
> > > (https://issues.apache.org/jira/browse/MYRIAD-279).
> > >
> > > The document design of this new UI here:
> > > https://docs.google.com/document/d/16gA67RXoPK24OIxDMNNhuYS8ioScI1eOBR-XMMPjWQE/edit?usp=sharing
> > > --
> > > Javi Roman
> > >
> > > Twitter: @javiromanrh
> > > GitHub: github.com/javiroman
> > > Linkedin: es.linkedin.com/in/javiroman
> > > Big Data Blog: dataintensive.info
> > > Apache Id: javiroman
> > >
> > >> On Thu, Sep 12, 2019 at 8:55 AM Oscar Fernandez  
> > >> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 
> > >> -
> > >> Multiple versions of Apache Hadoop YARN as a Service.
> > >>
> > >> In order to implement this, we should avoid starting the Myriad framework
> > >> from Yarn and instead starting Yarn(s) from Myriad on demand.
> > >>
> > >> I wanted to ask the Myriad community if this design was intended for a
> > >> reason or if you think it's a good idea to decouple the execution of 
> > >> Myriad
> > >> from the Yarn RM. With the new design, the Myriad Framework would 
> > >> register
> > >> on Mesos, and then, start on demand the RM and NM that the user wants,
> > >> allowing several Yarn clusters to run in he same Mesos, even with 
> > >> different
> > >> versions.
> > >>
> > >> Thank you
> >  

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 I am not saying it's crazy. I was voicing my opinion. Isn't it what was the 
purpose of the discussion?
It's definitely great to have UI that manages all the YARN clusters, but it's 
not like UI/Web service has to be coupled/collocated with any of the Myriad 
particular YARN version daemons.
It's great if you would provide write up with pros and cons for your approach 
or any alternative approaches.


On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman 
 wrote:  
 
 On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman
 wrote:
>
>  Hello,
> Again I apologize for the late reply.
> I think I replied to the thread, but will add more direct notes here
> What you are proposing is to have yet another daemon that would start Yarn 
> Clusters on demand within Mesos framework.
> Meaning - it would be another layer of abstraction.  In this case that new 
> layer would need to behave as second level scheduler and deal with third 
> level scheduler(s) (RMs) to propagate offers from Mesos and keep track, etc.
> I am sure you can somehow use concept of Capacity and/or FairShare scheduler 
> in your new layer to do the job. I am just not very much convinced that 3 
> layers of scheduling will be easy to maintain/reconcile/etc.
> Again - if I understand your design correctly.
> Would be great if you do a small write up with the proposal and have some 
> simple diagram of services interactions.
> Just my 2c.
> Thanks,Yuliya

Great, I wil do a diagram!

Only for clarify:

Myriad is registered as framework in Mesos master. The same thread
start the API server and the user interface. By means the user
interface you select the YARN version to run, and the scheduler get
resources from master for running RM and NMs. So you con manage as
many YARN schedulers you want. YARN as a Service.

Maybe I am missing the point, bu I don't feel this is something so
strange, or so crazy!


>    On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez 
> wrote:
>
>  Hi,
>
> I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 -
> Multiple versions of Apache Hadoop YARN as a Service.
>
> In order to implement this, we should avoid starting the Myriad framework
> from Yarn and instead starting Yarn(s) from Myriad on demand.
>
> I wanted to ask the Myriad community if this design was intended for a
> reason or if you think it's a good idea to decouple the execution of Myriad
> from the Yarn RM. With the new design, the Myriad Framework would register
> on Mesos, and then, start on demand the RM and NM that the user wants,
> allowing several Yarn clusters to run in he same Mesos, even with different
> versions.
>
> Thank you
>
  

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 Hello,
Again I apologize for the late reply.
I think I replied to the thread, but will add more direct notes here
What you are proposing is to have yet another daemon that would start Yarn 
Clusters on demand within Mesos framework.
Meaning - it would be another layer of abstraction.  In this case that new 
layer would need to behave as second level scheduler and deal with third level 
scheduler(s) (RMs) to propagate offers from Mesos and keep track, etc.
I am sure you can somehow use concept of Capacity and/or FairShare scheduler in 
your new layer to do the job. I am just not very much convinced that 3 layers 
of scheduling will be easy to maintain/reconcile/etc.
Again - if I understand your design correctly.
Would be great if you do a small write up with the proposal and have some 
simple diagram of services interactions.
Just my 2c.
Thanks,Yuliya
On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez 
 wrote:  
 
 Hi,

I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 -
Multiple versions of Apache Hadoop YARN as a Service.

In order to implement this, we should avoid starting the Myriad framework
from Yarn and instead starting Yarn(s) from Myriad on demand.

I wanted to ask the Myriad community if this design was intended for a
reason or if you think it's a good idea to decouple the execution of Myriad
from the Yarn RM. With the new design, the Myriad Framework would register
on Mesos, and then, start on demand the RM and NM that the user wants,
allowing several Yarn clusters to run in he same Mesos, even with different
versions.

Thank you
  

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 >>> I guess we are talking about the concept of multi-service schedulerAre you 
 >>>proposing to have a multi service scheduler? I thought that's what Mesos is 
 >>>for?What am I missing here?
Thanks,Yuliya
On Tuesday, September 24, 2019, 10:00:39 AM PDT, Javi Roman 
 wrote:  
 
 I guess we are talking about the concept of multi-service scheduler, or one
framework for multiple services, in our case multiple YARN services
(different versions, and so on).

Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
Apache Id: javiroman
On Tue, Sep 24, 2019 at 5:40 PM Yuliya  wrote:
>
> Hello there,
>
> Sorry for late reply.
>
> Frankly speaking I don’t know original motivation , I would probably say that 
> it started organically as a need.
>
> What do you mean by starting yarn from myriad? I believe myriad daemons 
> encompass some functionality of yarn daemons, namely rm and nm, so it’s not 
> yarn that starts myriad, but myriad daemons that play role of yarn daemons.
>  Unless I am missing something here.
>
> Are you proposing to have the same myriad daemons starting different versions 
> of yarn?
>  I would consider different set of docker containers built with different 
>versions of yarn would be better decoupling. I am open for discussion though.
>
> Thanks,
> Yuliya
>
>
>
> > On Sep 15, 2019, at 10:35 PM, Javi Roman  wrote:
> >
> > Hi Oscar,
> >
> > I have to say I don't know the initial motivation of this design. You
> > are right the way of starting Myriad, strongly coupled to YARN is a
> > little bit weird.
> > Because of lack of activity of the initial committers, this is a
> > question that probably we never get a clear answer.
> >
> > By the way, your proposal, according with MYRIAD-295 is, from my
> > understanding, the right way to go ahead with the project.
> >
> > This new design is totally aligned with the further Myriad UI design
> > (https://issues.apache.org/jira/browse/MYRIAD-279).
> >
> > The document design of this new UI here:
> > https://docs.google.com/document/d/16gA67RXoPK24OIxDMNNhuYS8ioScI1eOBR-XMMPjWQE/edit?usp=sharing
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> > Apache Id: javiroman
> >
> >> On Thu, Sep 12, 2019 at 8:55 AM Oscar Fernandez  
> >> wrote:
> >>
> >> Hi,
> >>
> >> I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 -
> >> Multiple versions of Apache Hadoop YARN as a Service.
> >>
> >> In order to implement this, we should avoid starting the Myriad framework
> >> from Yarn and instead starting Yarn(s) from Myriad on demand.
> >>
> >> I wanted to ask the Myriad community if this design was intended for a
> >> reason or if you think it's a good idea to decouple the execution of Myriad
> >> from the Yarn RM. With the new design, the Myriad Framework would register
> >> on Mesos, and then, start on demand the RM and NM that the user wants,
> >> allowing several Yarn clusters to run in he same Mesos, even with different
> >> versions.
> >>
> >> Thank you
>  

Re: Multiple versions of Apache Hadoop YARN as a Service

[yuliya Feldman]

 Hello,
Thank you for the diagrams - it helps. Could you also enable comments in your 
doc?
Few thoughts:1.  Myriad Scheduler is wonderful - but it's yet another scheduler 
you need to deal with - or you plan to have current MyriadScheduler that sits 
in RM and use it instead?2. Is Myriad scheduler going to control NMs for all 
YARN clusters?     a. How NMs will come and go?     b. Are they going to be 
permanent?    c. I assume RMs will be permanent until cluster shutdown, 
right?3. If NMs will not be permanent - isn't it too much for upper level 
Myriad Scheduler to deal with all of them?

Also could you please list cons - pros are great, but it's better to have cons 
as well.
Thanks,Yuliya


On Wednesday, September 25, 2019, 12:30:20 AM PDT, Oscar Fernandez 
 wrote:  
 
 Hi,

I've made a diagram to represent the new proposed design in order to
support Yarn as a service with some of the pros:
https://docs.google.com/document/d/15X0-zSu0G0BDpWyndRhbvAJCXLtAbkNA45wQ_xVKOKQ

Thank you for all your comments and help

On Tue, Sep 24, 2019 at 8:57 PM Javi Roman  wrote:

> Honestly your opinion is welcome, this kind of discussions are great
> in this small traffic dev list ;-)
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
> Apache Id: javiroman
>
> On Tue, Sep 24, 2019 at 8:55 PM yuliya Feldman
>  wrote:
> >
> >  I am not saying it's crazy. I was voicing my opinion. Isn't it what was
> the purpose of the discussion?
> > It's definitely great to have UI that manages all the YARN clusters, but
> it's not like UI/Web service has to be coupled/collocated with any of the
> Myriad particular YARN version daemons.
> > It's great if you would provide write up with pros and cons for your
> approach or any alternative approaches.
> >
> >
> >    On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman <
> jroman.espi...@gmail.com> wrote:
> >
> >  On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman
> >  wrote:
> > >
> > >  Hello,
> > > Again I apologize for the late reply.
> > > I think I replied to the thread, but will add more direct notes here
> > > What you are proposing is to have yet another daemon that would start
> Yarn Clusters on demand within Mesos framework.
> > > Meaning - it would be another layer of abstraction.  In this case that
> new layer would need to behave as second level scheduler and deal with
> third level scheduler(s) (RMs) to propagate offers from Mesos and keep
> track, etc.
> > > I am sure you can somehow use concept of Capacity and/or FairShare
> scheduler in your new layer to do the job. I am just not very much
> convinced that 3 layers of scheduling will be easy to
> maintain/reconcile/etc.
> > > Again - if I understand your design correctly.
> > > Would be great if you do a small write up with the proposal and have
> some simple diagram of services interactions.
> > > Just my 2c.
> > > Thanks,Yuliya
> >
> > Great, I wil do a diagram!
> >
> > Only for clarify:
> >
> > Myriad is registered as framework in Mesos master. The same thread
> > start the API server and the user interface. By means the user
> > interface you select the YARN version to run, and the scheduler get
> > resources from master for running RM and NMs. So you con manage as
> > many YARN schedulers you want. YARN as a Service.
> >
> > Maybe I am missing the point, bu I don't feel this is something so
> > strange, or so crazy!
> >
> >
> > >    On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez <
> oscarf...@apache.org> wrote:
> > >
> > >  Hi,
> > >
> > > I've started working on
> https://issues.apache.org/jira/browse/MYRIAD-295 -
> > > Multiple versions of Apache Hadoop YARN as a Service.
> > >
> > > In order to implement this, we should avoid starting the Myriad
> framework
> > > from Yarn and instead starting Yarn(s) from Myriad on demand.
> > >
> > > I wanted to ask the Myriad community if this design was intended for a
> > > reason or if you think it's a good idea to decouple the execution of
> Myriad
> > > from the Yarn RM. With the new design, the Myriad Framework would
> register
> > > on Mesos, and then, start on demand the RM and NM that the user wants,
> > > allowing several Yarn clusters to run in he same Mesos, even with
> different
> > > versions.
> > >
> > > Thank you
> > >
> >
>
  

[jira] [Commented] (MYRIAD-121) Myriad-Issue-60 Added ability to launch JHS ...

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703996#comment-14703996
 ] 

Yuliya Feldman commented on MYRIAD-121:
---

All the comments are going on https://github.com/mesos/myriad/pull/121

 Myriad-Issue-60 Added ability to launch JHS ...
 ---

 Key: MYRIAD-121
 URL: https://issues.apache.org/jira/browse/MYRIAD-121
 Project: Myriad
  Issue Type: Bug
Reporter: Yuliya Feldman

 ... and other services that fit under Myriad umbrella from MyriadScheduler 
 using default Mesos slave executor. Will update following doc 
 https://docs.google.com/document/d/1TGGU2bn1cVPSmHeriNrtkAEhdvsRKcFtuWRaaGp9bSY/edit?usp=sharing
 with more details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-18) staging - pending loop

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967852#comment-14967852
 ] 

Yuliya Feldman commented on MYRIAD-18:
--

I believe now you can flex down those, since we changed the order and added 
profile of NM you are flexing down to flex down pending, staging and only then 
active tasks.

> staging - pending loop
> --
>
> Key: MYRIAD-18
> URL: https://issues.apache.org/jira/browse/MYRIAD-18
> Project: Myriad
>  Issue Type: Bug
>Reporter: Maysam Yabandeh
> Fix For: Myriad 0.1.0
>
>
> if staging task is lost for any reason it gets stuck in a staging-pending 
> loop.
> case TASK_LOST:
> schedulerState.makeTaskPending(taskId);
> break;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (MYRIAD-148) /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's param key value.

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman reassigned MYRIAD-148:
-

Assignee: Yuliya Feldman

> /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's 
> param key value.
> -
>
> Key: MYRIAD-148
> URL: https://issues.apache.org/jira/browse/MYRIAD-148
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
>    Assignee: Yuliya Feldman
>
> When executed the /api/config to get myriad configs, the api output doesn't 
> have values for nodeManagerConfiguration & myriadExecutorConfiguration's 
> parameter keys.
> e.g. /api/config json (snippet) output:
> "myriadExecutorConfiguration": {
> "jvmMaxMemoryMB": {
> "present": true<--
> }, 
> "nodeManagerUri": {
> "present": false   <--
> }, 
> "path": 
> "file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar"
> }, 
> "nativeLibrary": "/usr/local/lib/libmesos.so", 
> "nmInstances": {
> "medium": 1
> }, 
> "nodeManagerConfiguration": {
> "cgroups": {
> "present": true<--
> }, 
> "cpus": {
> "present": true  <--
> }, 
> "jvmMaxMemoryMB": {
> "present": true <--
> }, 
> "jvmOpts": {
> "present": false<--
> }
> }, 
> From myriad-config-default.yml:
> nodemanager:
>   jvmMaxMemoryMB: 1024
>   cpus: 0.2
>   cgroups: false
> executor:
>   jvmMaxMemoryMB: 256
>   path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
>   #The following should be used for a remotely distributed URI, hdfs assumed 
> but other URI types valid.
>   #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.5.0.tar.gz
>   #path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
> As the parameter values are specified in *.yml, /api/config json output 
> should contain the values for the param keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-148) /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's param key value.

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970520#comment-14970520
 ] 

Yuliya Feldman commented on MYRIAD-148:
---

Submitted PR: https://github.com/apache/incubator-myriad/pull/19

Problem is that Optional does not serialize correctly . It shows state 
"present" instead of value.
Created custom serializers to produce value if present and string "value is 
absent" when value is absent.

> /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's 
> param key value.
> -
>
> Key: MYRIAD-148
> URL: https://issues.apache.org/jira/browse/MYRIAD-148
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
>Assignee: Yuliya Feldman
>
> When executed the /api/config to get myriad configs, the api output doesn't 
> have values for nodeManagerConfiguration & myriadExecutorConfiguration's 
> parameter keys.
> e.g. /api/config json (snippet) output:
> "myriadExecutorConfiguration": {
> "jvmMaxMemoryMB": {
> "present": true<--
> }, 
> "nodeManagerUri": {
> "present": false   <--
> }, 
> "path": 
> "file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar"
> }, 
> "nativeLibrary": "/usr/local/lib/libmesos.so", 
> "nmInstances": {
> "medium": 1
> }, 
> "nodeManagerConfiguration": {
> "cgroups": {
> "present": true<--
> }, 
> "cpus": {
> "present": true  <--
> }, 
> "jvmMaxMemoryMB": {
> "present": true <--
> }, 
> "jvmOpts": {
> "present": false<--
> }
> }, 
> From myriad-config-default.yml:
> nodemanager:
>   jvmMaxMemoryMB: 1024
>   cpus: 0.2
>   cgroups: false
> executor:
>   jvmMaxMemoryMB: 256
>   path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
>   #The following should be used for a remotely distributed URI, hdfs assumed 
> but other URI types valid.
>   #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.5.0.tar.gz
>   #path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
> As the parameter values are specified in *.yml, /api/config json output 
> should contain the values for the param keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-172) More resources assigned to executor

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman updated MYRIAD-172:
--
Assignee: DarinJ  (was: Yuliya Feldman)

> More resources assigned to executor
> ---
>
> Key: MYRIAD-172
> URL: https://issues.apache.org/jira/browse/MYRIAD-172
> Project: Myriad
>  Issue Type: Bug
>  Components: Executor, Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: DarinJ
>
> Twice the number of resources for CPU and memory is being assigned for the 
> executor.
> Once the Node Manager task is launched I see the following resources in mesos:
> Zero Profile NM : Mem : 1.4 GB , CPU : 0.4
> Medium Profile NM: Mem : 5.4 GB CPU :  4.4 
> Earlier it was 
> Zero : 1.2 GB and CPU : 0.2   ( Mem : 1 GB + 10% , CPU static value of 0.2)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (MYRIAD-172) More resources assigned to executor

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman reassigned MYRIAD-172:
-

Assignee: Yuliya Feldman

> More resources assigned to executor
> ---
>
> Key: MYRIAD-172
> URL: https://issues.apache.org/jira/browse/MYRIAD-172
> Project: Myriad
>  Issue Type: Bug
>  Components: Executor, Scheduler
>Reporter: Aashreya Ravi Shankar
>    Assignee: Yuliya Feldman
>
> Twice the number of resources for CPU and memory is being assigned for the 
> executor.
> Once the Node Manager task is launched I see the following resources in mesos:
> Zero Profile NM : Mem : 1.4 GB , CPU : 0.4
> Medium Profile NM: Mem : 5.4 GB CPU :  4.4 
> Earlier it was 
> Zero : 1.2 GB and CPU : 0.2   ( Mem : 1 GB + 10% , CPU static value of 0.2)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-170) Myriad initialization fails with "parameter 5 of org.apache.myriad.scheduler.MyriadOperations.() is not @Nullable"

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993193#comment-14993193
 ] 

Yuliya Feldman commented on MYRIAD-170:
---

Thank you for reporting the issue. We will try to provide patch tomorrow. 
Meanwhile can you try to set

haEnabled: true

in myriad-config-default.yml

> Myriad initialization fails with "parameter 5 of 
> org.apache.myriad.scheduler.MyriadOperations.() is not @Nullable"
> 
>
> Key: MYRIAD-170
> URL: https://issues.apache.org/jira/browse/MYRIAD-170
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.2.0
> Environment: CentOS7
>Reporter: Zhongyue Luo
>Assignee: Swapnil Daingade
>
> After the framework gets registered, scheduler initialization complains that 
> " null returned by binding at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore()"
> I have build Myriad according to the remote binary distribution document.
> Additional steps I did out of the document was setting up the http_proxy 
> environment value.
> I've search if this problem was brought up before but failed to find answers.
> Below is the log output of the scheduler.
> ==
> I1106 10:42:01.039393 11243 sched.cpp:641] Framework registered with 
> c70248ae-62a9-4a02-82b9-46a5e10fd15f-0016
> 15/11/06 10:42:01 INFO service.AbstractService: Service 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 
> failed in state STARTED; cause: java.lang.RuntimeException: Failed to 
> initialize myriad
> java.lang.RuntimeException: Failed to initialize myriad
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:52)
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:92)
>   at 
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.serviceStart(MyriadFairScheduler.java:75)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:503)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:898)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:938)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:935)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:935)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:979)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1104)
> Caused by: com.google.inject.ProvisionException: Guice provision errors:
> 1) null returned by binding at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore()
>  but parameter 5 of org.apache.myriad.scheduler.MyriadOperations.() is 
> not @Nullable
>   at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore(MyriadModule.java:154)
>   while locating org.apache.myriad.state.MyriadStateStore
> for parameter 5 at 
> org.apache.myriad.scheduler.MyriadOperations.(MyriadOperations.java:59)
>   while locating org.apache.myriad.scheduler.MyriadOperations
> 1 error
>   at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
>   at 
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
>   at org.apache.myriad.Main.startNMInstances(Main.java:202)
>   at org.apache.myriad.Main.run(Main.java:113)
>   at org.apache.myriad.Main.initialize(Main.java:88)
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationI

[jira] [Commented] (MYRIAD-170) Myriad initialization fails with "parameter 5 of org.apache.myriad.scheduler.MyriadOperations.() is not @Nullable"

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993007#comment-14993007
 ] 

Yuliya Feldman commented on MYRIAD-170:
---

Thanks for reporting the issue.

Are you using apache incubator-myriad repo master branch? If yes, when did you 
build it?

> Myriad initialization fails with "parameter 5 of 
> org.apache.myriad.scheduler.MyriadOperations.() is not @Nullable"
> 
>
> Key: MYRIAD-170
> URL: https://issues.apache.org/jira/browse/MYRIAD-170
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.2.0
> Environment: CentOS7
>Reporter: Zhongyue Luo
>
> After the framework gets registered, scheduler initialization complains that 
> " null returned by binding at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore()"
> I have build Myriad according to the remote binary distribution document.
> Additional steps I did out of the document was setting up the http_proxy 
> environment value.
> I've search if this problem was brought up before but failed to find answers.
> Below is the log output of the scheduler.
> ==
> I1106 10:42:01.039393 11243 sched.cpp:641] Framework registered with 
> c70248ae-62a9-4a02-82b9-46a5e10fd15f-0016
> 15/11/06 10:42:01 INFO service.AbstractService: Service 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 
> failed in state STARTED; cause: java.lang.RuntimeException: Failed to 
> initialize myriad
> java.lang.RuntimeException: Failed to initialize myriad
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:52)
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:92)
>   at 
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.serviceStart(MyriadFairScheduler.java:75)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:503)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:898)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:938)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:935)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:935)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:979)
>   at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1104)
> Caused by: com.google.inject.ProvisionException: Guice provision errors:
> 1) null returned by binding at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore()
>  but parameter 5 of org.apache.myriad.scheduler.MyriadOperations.() is 
> not @Nullable
>   at 
> org.apache.myriad.MyriadModule.providesMyriadStateStore(MyriadModule.java:154)
>   while locating org.apache.myriad.state.MyriadStateStore
> for parameter 5 at 
> org.apache.myriad.scheduler.MyriadOperations.(MyriadOperations.java:59)
>   while locating org.apache.myriad.scheduler.MyriadOperations
> 1 error
>   at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
>   at 
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
>   at org.apache.myriad.Main.startNMInstances(Main.java:202)
>   at org.apache.myriad.Main.run(Main.java:113)
>   at org.apache.myriad.Main.initialize(Main.java:88)
>   at 
> org.apache.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:49)
>   ... 16 more
> 15/11/06 10:42:01 INFO service.AbstractService: Ser

[jira] [Resolved] (MYRIAD-60) Integrate YARN timeline server/job history server with Myriad.

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman resolved MYRIAD-60.
--
Resolution: Duplicate

MYRIAD-121 supersedes this one

> Integrate YARN timeline server/job history server with Myriad.
> --
>
> Key: MYRIAD-60
> URL: https://issues.apache.org/jira/browse/MYRIAD-60
> Project: Myriad
>  Issue Type: Improvement
>Reporter: Santosh Marella
>        Assignee: Yuliya Feldman
>
> With Myriad, RM launches NMs via Mesos. Perhaps Myriad should be responsible 
> to launch YARN's timeline/job history server as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MYRIAD-158) License string <-- Licensed ...--> is showing up on myriad UI.

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964485#comment-14964485
 ] 

Yuliya Feldman edited comment on MYRIAD-158 at 10/20/15 4:15 AM:
-

It is important to fix - it does not look pretty.

Looks like UI issues


was (Author: yufeldman):
It is important to fix - it doe snot look pretty.

Looks like UI issues

> License string <-- Licensed ...--> is showing up on myriad UI.
> --
>
> Key: MYRIAD-158
> URL: https://issues.apache.org/jira/browse/MYRIAD-158
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
> Fix For: Myriad 0.1.0
>
> Attachments: Screen Shot 2015-10-19 at 5.45.14 PM.png
>
>
> The following is showing up on myriad UI:
> <-- Licensed to the Apache Software Foundation (ASF) under one or more 
> contributor license agreements. See the NOTICE file distributed with this 
> work for additional information regarding copyright ownership. The ASF 
> licenses this file to you under the Apache License, Version 2.0 (the 
> "License"); you may not use this file except in compliance with the License. 
> You may obtain a copy of the License at 
> http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law 
> or agreed to in writing, software distributed under the License is 
> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
> KIND, either express or implied. See the License for the specific language 
> governing permissions and limitations under the License. -->
> See the screenshot



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-158) License string <-- Licensed ...--> is showing up on myriad UI.

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964485#comment-14964485
 ] 

Yuliya Feldman commented on MYRIAD-158:
---

It is important to fix - it doe snot look pretty.

Looks like UI issues

> License string <-- Licensed ...--> is showing up on myriad UI.
> --
>
> Key: MYRIAD-158
> URL: https://issues.apache.org/jira/browse/MYRIAD-158
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
> Fix For: Myriad 0.1.0
>
> Attachments: Screen Shot 2015-10-19 at 5.45.14 PM.png
>
>
> The following is showing up on myriad UI:
> <-- Licensed to the Apache Software Foundation (ASF) under one or more 
> contributor license agreements. See the NOTICE file distributed with this 
> work for additional information regarding copyright ownership. The ASF 
> licenses this file to you under the Apache License, Version 2.0 (the 
> "License"); you may not use this file except in compliance with the License. 
> You may obtain a copy of the License at 
> http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law 
> or agreed to in writing, software distributed under the License is 
> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
> KIND, either express or implied. See the License for the specific language 
> governing permissions and limitations under the License. -->
> See the screenshot



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-159) Change default mesos version to 0.24

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-159:
-

 Summary: Change default mesos version to 0.24
 Key: MYRIAD-159
 URL: https://issues.apache.org/jira/browse/MYRIAD-159
 Project: Myriad
  Issue Type: Improvement
  Components: Scheduler
Reporter: Yuliya Feldman
 Fix For: Myriad 0.1.0


Since Mesos moved to 0.25 version by now it would be beneficial to change Mesos 
version with which myriad is built to 0.24 at least



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-140) NullPointerException on NM flex up/down API when no params passed to HTTP PUT request.

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965604#comment-14965604
 ] 

Yuliya Feldman commented on MYRIAD-140:
---

Submitted PR: https://github.com/apache/incubator-myriad/pull/16  to fix this 
issue

> NullPointerException on NM flex up/down API when no params passed to HTTP PUT 
> request.
> --
>
> Key: MYRIAD-140
> URL: https://issues.apache.org/jira/browse/MYRIAD-140
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Sarjeet Singh
>    Assignee: Yuliya Feldman
>
> Observed the NullPointerException when tried to flex up using curl without 
> any params passed to http request. (didn't use myriad UI for flexing)
> 15/09/22 14:54:50 INFO api.ClustersResource: Received Flexup Cluster Request
> 15/09/22 14:54:50 INFO api.ClustersResource: Instances: null
> 15/09/22 14:54:50 INFO api.ClustersResource: Profile: null
> Sep 22, 2015 2:54:50 PM com.sun.jersey.spi.container.ContainerResponse 
> mapMappableContainerException
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing 
> to the HTTP container
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
>   at 
> java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:1016)
>   at 
> com.ebay.myriad.scheduler.NMProfileManager.exists(NMProfileManager.java:44)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:84)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>   at 
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>   at 
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>   at 
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>   at 
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>   at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>   at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>   at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>   at 
> org.mortba

[jira] [Commented] (MYRIAD-129) Creating custom profiles requires configurations changes on all nodes.

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735288#comment-14735288
 ] 

Yuliya Feldman commented on MYRIAD-129:
---

I think we should have API/UI to add/remove/modify profiles. Also keep all the 
configuration updates in state store in case of failover

> Creating custom profiles requires configurations changes on all nodes.
> --
>
> Key: MYRIAD-129
> URL: https://issues.apache.org/jira/browse/MYRIAD-129
> Project: Myriad
>  Issue Type: Bug
>Reporter: Aashreya Ravi Shankar
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-136) Need to handle TASK_ERROR in StatusUpdateEventHandler

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-136:
-

 Summary: Need to handle TASK_ERROR in StatusUpdateEventHandler
 Key: MYRIAD-136
 URL: https://issues.apache.org/jira/browse/MYRIAD-136
 Project: Myriad
  Issue Type: Bug
Reporter: Yuliya Feldman


In StatusUpdateEventHandler we do not handle TASK_ERROR at all.

Need to act on it, otherwise nothing happens when launched task returns with 
TASK_ERROR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-131) Timeout for tasks in Pending or Staging state

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876073#comment-14876073
 ] 

Yuliya Feldman commented on MYRIAD-131:
---

Feature Santosh is working on to enhance flexup/flexdown APIs with constraints 
could be a vehicle to address this JIRA - flexdown only pending/staging tasks

> Timeout for tasks in Pending or Staging state
> -
>
> Key: MYRIAD-131
> URL: https://issues.apache.org/jira/browse/MYRIAD-131
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>
> Currently tasks in the pending or staging state never times out. I think we 
> need to set a timeout if a process is unable to get launched for any reason.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-136) Need to handle TASK_ERROR in StatusUpdateEventHandler

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876062#comment-14876062
 ] 

Yuliya Feldman commented on MYRIAD-136:
---

I wanted to get opinion on how to handle TASK_ERROR. Simplest/obvious would be 
to remove the task (with recording the reason), but wanted to get people 
opinion.

> Need to handle TASK_ERROR in StatusUpdateEventHandler
> -
>
> Key: MYRIAD-136
> URL: https://issues.apache.org/jira/browse/MYRIAD-136
> Project: Myriad
>  Issue Type: Bug
>    Reporter: Yuliya Feldman
>
> In StatusUpdateEventHandler we do not handle TASK_ERROR at all.
> Need to act on it, otherwise nothing happens when launched task returns with 
> TASK_ERROR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-141) Include myriad framework name into task name

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-141:
-

 Summary: Include myriad framework name into task name
 Key: MYRIAD-141
 URL: https://issues.apache.org/jira/browse/MYRIAD-141
 Project: Myriad
  Issue Type: Improvement
  Components: Scheduler
Affects Versions: Myriad 0.1.0
Reporter: Yuliya Feldman


when people plan to run multiple myriad clusters under same Mesos it is very 
hard to figure out which task belongs to which framework when getting list of 
active tasks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-142) NMExecutorCLGenImpl has to define myriad.mapreduce.shuffle.port just as port, not address

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-142:
-

 Summary: NMExecutorCLGenImpl has to define 
myriad.mapreduce.shuffle.port just as port, not address
 Key: MYRIAD-142
 URL: https://issues.apache.org/jira/browse/MYRIAD-142
 Project: Myriad
  Issue Type: Bug
  Components: Scheduler
Affects Versions: Myriad 0.1.0
Reporter: Yuliya Feldman


NMExecutorCLGenImpl has to define myriad.mapreduce.shuffle.port just as port, 
not address

Otherwise there will be thrown NumberFormatException if 
myriad.mapreduce.shuffle.port provided in any config file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-140) NullPointerException on NM flex up/down API when no params passed to HTTP PUT request.

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman updated MYRIAD-140:
--
Fix Version/s: Myriad 0.1.0

> NullPointerException on NM flex up/down API when no params passed to HTTP PUT 
> request.
> --
>
> Key: MYRIAD-140
> URL: https://issues.apache.org/jira/browse/MYRIAD-140
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Sarjeet Singh
>    Assignee: Yuliya Feldman
> Fix For: Myriad 0.1.0
>
>
> Observed the NullPointerException when tried to flex up using curl without 
> any params passed to http request. (didn't use myriad UI for flexing)
> 15/09/22 14:54:50 INFO api.ClustersResource: Received Flexup Cluster Request
> 15/09/22 14:54:50 INFO api.ClustersResource: Instances: null
> 15/09/22 14:54:50 INFO api.ClustersResource: Profile: null
> Sep 22, 2015 2:54:50 PM com.sun.jersey.spi.container.ContainerResponse 
> mapMappableContainerException
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing 
> to the HTTP container
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
>   at 
> java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:1016)
>   at 
> com.ebay.myriad.scheduler.NMProfileManager.exists(NMProfileManager.java:44)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:84)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>   at 
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>   at 
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>   at 
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>   at 
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>   at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>   at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>   at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>   at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(Han

[jira] [Created] (MYRIAD-178) Add UI to support flexup/down services in Myriad

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-178:
-

 Summary: Add UI to support flexup/down services in Myriad
 Key: MYRIAD-178
 URL: https://issues.apache.org/jira/browse/MYRIAD-178
 Project: Myriad
  Issue Type: Improvement
  Components: Scheduler
Affects Versions: Myriad 0.1.0
Reporter: Yuliya Feldman


As of now we only support REST APIs to flex up/down services in Myriad.

We need to have UI for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MYRIAD-140) NullPointerException on NM flex up/down API when no params passed to HTTP PUT request.

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman resolved MYRIAD-140.
---
Resolution: Fixed

https://github.com/apache/incubator-myriad/pull/16

> NullPointerException on NM flex up/down API when no params passed to HTTP PUT 
> request.
> --
>
> Key: MYRIAD-140
> URL: https://issues.apache.org/jira/browse/MYRIAD-140
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Sarjeet Singh
>    Assignee: Yuliya Feldman
> Fix For: Myriad 0.1.0
>
>
> Observed the NullPointerException when tried to flex up using curl without 
> any params passed to http request. (didn't use myriad UI for flexing)
> 15/09/22 14:54:50 INFO api.ClustersResource: Received Flexup Cluster Request
> 15/09/22 14:54:50 INFO api.ClustersResource: Instances: null
> 15/09/22 14:54:50 INFO api.ClustersResource: Profile: null
> Sep 22, 2015 2:54:50 PM com.sun.jersey.spi.container.ContainerResponse 
> mapMappableContainerException
> SEVERE: The RuntimeException could not be mapped to a response, re-throwing 
> to the HTTP container
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
>   at 
> java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:1016)
>   at 
> com.ebay.myriad.scheduler.NMProfileManager.exists(NMProfileManager.java:44)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:84)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
>   at 
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
>   at 
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
>   at 
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
>   at 
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>   at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>   at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>   at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>   at 
> org.mortbay.jetty.handler.H

[jira] [Commented] (MYRIAD-178) Add UI to support flexup/down services in Myriad

[Yuliya Feldman (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160243#comment-15160243
 ] 

Yuliya Feldman commented on MYRIAD-178:
---

I would imagine that tools we are using should be similar to the ones we use 
for NM flexup/down UI.
Considering API has service name, # instances, flexup/down I imagine it should 
be dropdown for service names (as they are configured in yml file) and the rest 
would look similar to NM flexup/down

> Add UI to support flexup/down services in Myriad
> 
>
> Key: MYRIAD-178
> URL: https://issues.apache.org/jira/browse/MYRIAD-178
> Project: Myriad
>  Issue Type: Improvement
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
>    Reporter: Yuliya Feldman
>
> As of now we only support REST APIs to flex up/down services in Myriad.
> We need to have UI for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-243) support authentication for Myriad REST and UI

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-243:
-

 Summary: support authentication for Myriad REST and UI
 Key: MYRIAD-243
 URL: https://issues.apache.org/jira/browse/MYRIAD-243
 Project: Myriad
  Issue Type: New Feature
  Components: Scheduler
Reporter: Yuliya Feldman
Assignee: Yuliya Feldman
 Fix For: Myriad 0.3.0


Today we do not have any authentication mechanism to prevent unauthenticated 
users from using Myriad REST APIs and UI to flex up/down NMs and other 
configured services as well as shutting down framework

This JIRA is to add authentication to Myriad UI and REST APIs

Most likely BasicAuth as well as SPNEGO support should be added - as those are 
standards that browsers and standard REST API tools support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-245) Add ability to set secure communication between Myriad Scheduler and Mesos

[Yuliya Feldman (JIRA)]

Yuliya Feldman created MYRIAD-245:
-

 Summary: Add ability to set secure communication between Myriad 
Scheduler and Mesos
 Key: MYRIAD-245
 URL: https://issues.apache.org/jira/browse/MYRIAD-245
 Project: Myriad
  Issue Type: New Feature
  Components: Scheduler
Reporter: Yuliya Feldman
Assignee: Yuliya Feldman
 Fix For: Myriad 0.3.0


It may end up just configuration task, but at least for bookkeeping purposes - 
make sure Myriad Scheduler can communicate securely 
(authentication,authorization, encryption) with Mesos Master.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)