Re: compliments on the report

2015-10-23 Thread Jim Scott 
That is great feedback. Kudo's to those who worked on that!

On Fri, Oct 23, 2015 at 1:33 AM, Ted Dunning  wrote:

> A board member commented on the Myriad section in the Incubator report:
>
> > Myriad
> > I really like the detail and thought in this report, covering both the
> > activities they're doing as well as questions/issues they have learning
> > about various bits of Apache policies or practices (showing they're
> > paying attention and adapting in a thoughtful way).
> >
>
>
> Well done, indeed!
>
> Not just the report, but the thought behind it.
>



-- 
*Jim Scott*
Director, Enterprise Strategy & Architecture
+1 (347) 746-9281
@kingmesal 


[image: MapR Technologies] 

Now Available - Free Hadoop On-Demand Training

Re: JIRA work for 0.1.0

2015-10-23 Thread Darin Johnson 
yuliya, jim's right this belongs on JIRA, I only commented here as you
mentioned it.  Let's continue there.

On Fri, Oct 23, 2015 at 1:38 PM, yuliya Feldman  wrote:

> Do you have other suggestions?
> I understand that there might be clashes even with randomization, but it
> is much lower chance then consistently getting the same port w/o
> randomization.
> Thanks,Yuliya  From: Darin Johnson 
>  To: Dev ; yuliya Feldman <
> yufeld...@yahoo.com>
>  Sent: Friday, October 23, 2015 9:57 AM
>  Subject: Re: JIRA work for 0.1.0
>
> On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
> outside of Mesos as other frameworks might select that port the probability
> of that happening is 1-(number of ports requested by other
> frameworks)/(number of ports in use).  This could get near 50% and cause
> frameworks that are being good citizens to crash.
>
> I'm also not convinced randomizing the port is in fact the correct fix for
> this in the long term, as there is still a non-zero chance you'll get that
> port again.
>
> Darin
>
>
>
>
>
> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> yufeld...@yahoo.com.invalid> wrote:
>
> > Great list.
> > I would include MYRIAD-160 to the list - I am working on that one. Of
> > course as a workaround we could not use Mesos ports and let NM ports
> > randomization kick in.I also almost done with MYRIAD-148 - it was really
> > tricky. Should submit PR tonight.
> > Thanks,Yuliya
> >  From: Darin Johnson 
> >  To: Dev 
> >  Sent: Thursday, October 22, 2015 8:29 PM
> >  Subject: Re: JIRA work for 0.1.0
> >
> > I think this sounds good about right.  A few Jim marked were new
> features,
> > gotta leave something for the 0.2.0 release :).
> >
> >
> >
> >
> > On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella 
> > wrote:
> >
> > > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > > There were 19 of them. I moved a few out. We are currently at 14.
> > >
> > > However, IMO the show stoppers are really the following:
> > >
> > > MYRIAD-43 Replace com.ebay namespace with org.apache
> > > MYRIAD-44 Prepare for 0.1.0 release
> > > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > > MYRIAD-114 Automatic dashboard building
> > > MYRIAD-145 Document Myriad Release Process
> > > MYRIAD-150 Update NOTICE file
> > > MYRIAD-159 Change default mesos version to 0.24
> > >
> > > Unless anyone thinks there are other JIRAs that are show stoppers, I
> > think
> > > we should stick to the above list
> > > and cut a RC as soon as we address the above.
> > >
> > > **I'm positive the above can be fixed by early next week (10/27) and we
> > can
> > > have a RC out for voting mid next week (10/28).**
> > >
> > > If we can't fix the above JIRAs in time or if new ones come up as "show
> > > stoppers", we will have a revised date.
> > > And, of course, more fixes are welcome, as long as they can be merged
> > > before 10/27.
> > >
> > > Just to let everyone know about the Apache release process (@Adam, feel
> > > free to chime in):
> > >  - Apache requires that a RC be put out for voting on
> > dev@myriad.incubator
> > > for 72 hrs or until 3 binding +1s and no binding -1s,
> > >  - followed by a similar voting round on general@incubator.
> > >
> > > Now to run the last mile..!
> > >
> > > Cheers,
> > > Santosh
> > >
> > > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon 
> > wrote:
> > >
> > > > Keep in mind that Santosh (as Release Manager) has final authority
> over
> > > > pushing things out of 0.1.0.
> > > >
> > > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> > JIRAs:
> > > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > > >
> > > > Maybe we should create a dashboard too, like
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > > >
> > > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar 
> wrote:
> > > >
> > > > > I went through JIRA and assigned a bunch of the tickets to the
> Myriad
> > > > 0.1.0
> > > > > release. Some of them are really low hanging fruit, so I think we
> can
> > > > knock
> > > > > most of these out. As Adam said in the meeting, we'd rather have
> too
> > > many
> > > > > flagged for the release and pair down rather than miss some. To
> that
> > > end,
> > > > > please go through the tickets that are still unmarked and set the
> Fix
> > > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> > the
> > > > > release.
> > > > >
> > > > > Jim
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
>
>
>

Re: JIRA work for 0.1.0

2015-10-23 Thread Darin Johnson 
On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
outside of Mesos as other frameworks might select that port the probability
of that happening is 1-(number of ports requested by other
frameworks)/(number of ports in use).  This could get near 50% and cause
frameworks that are being good citizens to crash.

I'm also not convinced randomizing the port is in fact the correct fix for
this in the long term, as there is still a non-zero chance you'll get that
port again.

Darin



On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:

> Great list.
> I would include MYRIAD-160 to the list - I am working on that one. Of
> course as a workaround we could not use Mesos ports and let NM ports
> randomization kick in.I also almost done with MYRIAD-148 - it was really
> tricky. Should submit PR tonight.
> Thanks,Yuliya
>   From: Darin Johnson 
>  To: Dev 
>  Sent: Thursday, October 22, 2015 8:29 PM
>  Subject: Re: JIRA work for 0.1.0
>
> I think this sounds good about right.  A few Jim marked were new features,
> gotta leave something for the 0.2.0 release :).
>
>
>
>
> On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella 
> wrote:
>
> > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > There were 19 of them. I moved a few out. We are currently at 14.
> >
> > However, IMO the show stoppers are really the following:
> >
> > MYRIAD-43 Replace com.ebay namespace with org.apache
> > MYRIAD-44 Prepare for 0.1.0 release
> > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > MYRIAD-114 Automatic dashboard building
> > MYRIAD-145 Document Myriad Release Process
> > MYRIAD-150 Update NOTICE file
> > MYRIAD-159 Change default mesos version to 0.24
> >
> > Unless anyone thinks there are other JIRAs that are show stoppers, I
> think
> > we should stick to the above list
> > and cut a RC as soon as we address the above.
> >
> > **I'm positive the above can be fixed by early next week (10/27) and we
> can
> > have a RC out for voting mid next week (10/28).**
> >
> > If we can't fix the above JIRAs in time or if new ones come up as "show
> > stoppers", we will have a revised date.
> > And, of course, more fixes are welcome, as long as they can be merged
> > before 10/27.
> >
> > Just to let everyone know about the Apache release process (@Adam, feel
> > free to chime in):
> >  - Apache requires that a RC be put out for voting on
> dev@myriad.incubator
> > for 72 hrs or until 3 binding +1s and no binding -1s,
> >  - followed by a similar voting round on general@incubator.
> >
> > Now to run the last mile..!
> >
> > Cheers,
> > Santosh
> >
> > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon 
> wrote:
> >
> > > Keep in mind that Santosh (as Release Manager) has final authority over
> > > pushing things out of 0.1.0.
> > >
> > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> JIRAs:
> > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > >
> > > Maybe we should create a dashboard too, like
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > >
> > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar  wrote:
> > >
> > > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > > 0.1.0
> > > > release. Some of them are really low hanging fruit, so I think we can
> > > knock
> > > > most of these out. As Adam said in the meeting, we'd rather have too
> > many
> > > > flagged for the release and pair down rather than miss some. To that
> > end,
> > > > please go through the tickets that are still unmarked and set the Fix
> > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> the
> > > > release.
> > > >
> > > > Jim
> > > >
> > >
> >
>
>
>
>

Re: JIRA work for 0.1.0

2015-10-23 Thread yuliya Feldman 
Do you have other suggestions?
I understand that there might be clashes even with randomization, but it is 
much lower chance then consistently getting the same port w/o randomization.
Thanks,Yuliya  From: Darin Johnson 
 To: Dev ; yuliya Feldman 
 
 Sent: Friday, October 23, 2015 9:57 AM
 Subject: Re: JIRA work for 0.1.0
   
On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
outside of Mesos as other frameworks might select that port the probability
of that happening is 1-(number of ports requested by other
frameworks)/(number of ports in use).  This could get near 50% and cause
frameworks that are being good citizens to crash.

I'm also not convinced randomizing the port is in fact the correct fix for
this in the long term, as there is still a non-zero chance you'll get that
port again.

Darin





On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:

> Great list.
> I would include MYRIAD-160 to the list - I am working on that one. Of
> course as a workaround we could not use Mesos ports and let NM ports
> randomization kick in.I also almost done with MYRIAD-148 - it was really
> tricky. Should submit PR tonight.
> Thanks,Yuliya
>      From: Darin Johnson 
>  To: Dev 
>  Sent: Thursday, October 22, 2015 8:29 PM
>  Subject: Re: JIRA work for 0.1.0
>
> I think this sounds good about right.  A few Jim marked were new features,
> gotta leave something for the 0.2.0 release :).
>
>
>
>
> On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella 
> wrote:
>
> > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > There were 19 of them. I moved a few out. We are currently at 14.
> >
> > However, IMO the show stoppers are really the following:
> >
> > MYRIAD-43 Replace com.ebay namespace with org.apache
> > MYRIAD-44 Prepare for 0.1.0 release
> > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > MYRIAD-114 Automatic dashboard building
> > MYRIAD-145 Document Myriad Release Process
> > MYRIAD-150 Update NOTICE file
> > MYRIAD-159 Change default mesos version to 0.24
> >
> > Unless anyone thinks there are other JIRAs that are show stoppers, I
> think
> > we should stick to the above list
> > and cut a RC as soon as we address the above.
> >
> > **I'm positive the above can be fixed by early next week (10/27) and we
> can
> > have a RC out for voting mid next week (10/28).**
> >
> > If we can't fix the above JIRAs in time or if new ones come up as "show
> > stoppers", we will have a revised date.
> > And, of course, more fixes are welcome, as long as they can be merged
> > before 10/27.
> >
> > Just to let everyone know about the Apache release process (@Adam, feel
> > free to chime in):
> >  - Apache requires that a RC be put out for voting on
> dev@myriad.incubator
> > for 72 hrs or until 3 binding +1s and no binding -1s,
> >  - followed by a similar voting round on general@incubator.
> >
> > Now to run the last mile..!
> >
> > Cheers,
> > Santosh
> >
> > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon 
> wrote:
> >
> > > Keep in mind that Santosh (as Release Manager) has final authority over
> > > pushing things out of 0.1.0.
> > >
> > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> JIRAs:
> > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > >
> > > Maybe we should create a dashboard too, like
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > >
> > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar  wrote:
> > >
> > > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > > 0.1.0
> > > > release. Some of them are really low hanging fruit, so I think we can
> > > knock
> > > > most of these out. As Adam said in the meeting, we'd rather have too
> > many
> > > > flagged for the release and pair down rather than miss some. To that
> > end,
> > > > please go through the tickets that are still unmarked and set the Fix
> > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> the
> > > > release.
> > > >
> > > > Jim
> > > >
> > >
> >
>
>
>
>

Remove .md files from github

2015-10-23 Thread Ruth Harris 
Hi All,

Hi Team,

I'd like have the following files removed from the docs section in GitHub
because the equivalent information is in the Wiki (updated and reviewed).
It causes confusion when people use them, plus we really shouldn't have two
set of the same info.

The duplicate docs are:
API.md
cgroups.md
how-it-works.md
myriad-configuration.md
myriad-dev.md
myriad-fine-grained-scaling.md
myriad-remote-distribution-configuration.md
vagrant.md

myriad-scheduler-architecture.md should be renamed and updated. OR, it
could be deleted and the updated version can be added to the repository.

Are we good with removing the files?


Thanks, Ruth

-- 
Ruth Harris
Sr. Technical Writer, MapR

[jira] [Assigned] (MYRIAD-148) /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's param key value.

2015-10-23 Thread Yuliya Feldman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MYRIAD-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman reassigned MYRIAD-148:
-

Assignee: Yuliya Feldman

> /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's 
> param key value.
> -
>
> Key: MYRIAD-148
> URL: https://issues.apache.org/jira/browse/MYRIAD-148
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
>Assignee: Yuliya Feldman
>
> When executed the /api/config to get myriad configs, the api output doesn't 
> have values for nodeManagerConfiguration & myriadExecutorConfiguration's 
> parameter keys.
> e.g. /api/config json (snippet) output:
> "myriadExecutorConfiguration": {
> "jvmMaxMemoryMB": {
> "present": true<--
> }, 
> "nodeManagerUri": {
> "present": false   <--
> }, 
> "path": 
> "file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar"
> }, 
> "nativeLibrary": "/usr/local/lib/libmesos.so", 
> "nmInstances": {
> "medium": 1
> }, 
> "nodeManagerConfiguration": {
> "cgroups": {
> "present": true<--
> }, 
> "cpus": {
> "present": true  <--
> }, 
> "jvmMaxMemoryMB": {
> "present": true <--
> }, 
> "jvmOpts": {
> "present": false<--
> }
> }, 
> From myriad-config-default.yml:
> nodemanager:
>   jvmMaxMemoryMB: 1024
>   cpus: 0.2
>   cgroups: false
> executor:
>   jvmMaxMemoryMB: 256
>   path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
>   #The following should be used for a remotely distributed URI, hdfs assumed 
> but other URI types valid.
>   #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.5.0.tar.gz
>   #path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
> As the parameter values are specified in *.yml, /api/config json output 
> should contain the values for the param keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-148) /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's param key value.

2015-10-23 Thread Yuliya Feldman (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970520#comment-14970520
 ] 

Yuliya Feldman commented on MYRIAD-148:
---

Submitted PR: https://github.com/apache/incubator-myriad/pull/19

Problem is that Optional does not serialize correctly . It shows state 
"present" instead of value.
Created custom serializers to produce value if present and string "value is 
absent" when value is absent.

> /api/config missing myriadExecutorConfiguration & nodeManagerConfiguration's 
> param key value.
> -
>
> Key: MYRIAD-148
> URL: https://issues.apache.org/jira/browse/MYRIAD-148
> Project: Myriad
>  Issue Type: Bug
>Reporter: Sarjeet Singh
>Assignee: Yuliya Feldman
>
> When executed the /api/config to get myriad configs, the api output doesn't 
> have values for nodeManagerConfiguration & myriadExecutorConfiguration's 
> parameter keys.
> e.g. /api/config json (snippet) output:
> "myriadExecutorConfiguration": {
> "jvmMaxMemoryMB": {
> "present": true<--
> }, 
> "nodeManagerUri": {
> "present": false   <--
> }, 
> "path": 
> "file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar"
> }, 
> "nativeLibrary": "/usr/local/lib/libmesos.so", 
> "nmInstances": {
> "medium": 1
> }, 
> "nodeManagerConfiguration": {
> "cgroups": {
> "present": true<--
> }, 
> "cpus": {
> "present": true  <--
> }, 
> "jvmMaxMemoryMB": {
> "present": true <--
> }, 
> "jvmOpts": {
> "present": false<--
> }
> }, 
> From myriad-config-default.yml:
> nodemanager:
>   jvmMaxMemoryMB: 1024
>   cpus: 0.2
>   cgroups: false
> executor:
>   jvmMaxMemoryMB: 256
>   path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
>   #The following should be used for a remotely distributed URI, hdfs assumed 
> but other URI types valid.
>   #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.5.0.tar.gz
>   #path: 
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.0.1.jar
> As the parameter values are specified in *.yml, /api/config json output 
> should contain the values for the param keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

compliments on the report

2015-10-23 Thread Ted Dunning 
A board member commented on the Myriad section in the Incubator report:

> Myriad
> I really like the detail and thought in this report, covering both the
> activities they're doing as well as questions/issues they have learning
> about various bits of Apache policies or practices (showing they're
> paying attention and adapting in a thoughtful way).
>


Well done, indeed!

Not just the report, but the thought behind it.