Re: JIRA work for 0.1.0

[Jim Klucar]

Santosh,

How about deleting some old/dead branches from git before the release?

Jim

On Tue, Oct 27, 2015 at 5:57 PM, Santosh Marella 
wrote:

> Hello everyone,
>
>   Just a quick update on the release status. We still need the following
> JIRAs to be resolved.
>
>   * MYRIAD-43 Replace com.ebay namespace with org.apache
>   * MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
>  - Jim is working on both of them
>
>   * MYRIAD-114 Automatic dashboard building
>  - Not sure if anyone is working on this one.
>The last comment on JIRA indicates Jim and Ken are collaborating to
> resolve this.
>
>   * MYRIAD-150 Update NOTICE file
>  - I think this is done. The JIRA has to be just marked as Resolved.
>
>   * MYRIAD-159 Change default mesos version to 0.24
>  - A minor one line change, but someone still needs to do it.
>
>   * MYRIAD-145 Document Myriad Release Process
>  - Created a draft on the wiki. Will update it as we tag the RC and
> publish it for voting.
>   * MYRIAD-44 Prepare for 0.1.0 release
>  - Just an Umbrella JIRA keeping track of the individual items.
>
>   In addition, Yuliya submitted a PR for MYRIAD-160.
>
> Thanks,
> Santosh
>
> On Fri, Oct 23, 2015 at 11:32 AM, Santosh Marella 
> wrote:
>
> > > let Myriad pick a random port outside of Mesos
> >
> > Agree with Darin. Picking a port outside of Mesos is prohibitive.
> >
> > I felt the simplest solution would be to pick a random port from the list
> > of ports offered to Myriad.
> >
> > The right solution, as Darin mentioned in the JIRA, would be to remove NM
> > from RM's memory.
> > I haven't looked at YARN's code deep enough to see if that's possible.
> But
> > that's something worth exploring.
> >
> > Thanks,
> > Santosh
> >
> > On Fri, Oct 23, 2015 at 9:57 AM, Darin Johnson 
> > wrote:
> >
> >> On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
> >> outside of Mesos as other frameworks might select that port the
> >> probability
> >> of that happening is 1-(number of ports requested by other
> >> frameworks)/(number of ports in use).  This could get near 50% and cause
> >> frameworks that are being good citizens to crash.
> >>
> >> I'm also not convinced randomizing the port is in fact the correct fix
> for
> >> this in the long term, as there is still a non-zero chance you'll get
> that
> >> port again.
> >>
> >> Darin
> >>
> >>
> >>
> >> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> >> yufeld...@yahoo.com.invalid> wrote:
> >>
> >> > Great list.
> >> > I would include MYRIAD-160 to the list - I am working on that one. Of
> >> > course as a workaround we could not use Mesos ports and let NM ports
> >> > randomization kick in.I also almost done with MYRIAD-148 - it was
> really
> >> > tricky. Should submit PR tonight.
> >> > Thanks,Yuliya
> >> >   From: Darin Johnson 
> >> >  To: Dev 
> >> >  Sent: Thursday, October 22, 2015 8:29 PM
> >> >  Subject: Re: JIRA work for 0.1.0
> >> >
> >> > I think this sounds good about right.  A few Jim marked were new
> >> features,
> >> > gotta leave something for the 0.2.0 release :).
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <
> smare...@maprtech.com
> >> >
> >> > wrote:
> >> >
> >> > > I looked at the JIRAs currently marked with fix version
> >> "myriad-0.1.0".
> >> > > There were 19 of them. I moved a few out. We are currently at 14.
> >> > >
> >> > > However, IMO the show stoppers are really the following:
> >> > >
> >> > > MYRIAD-43 Replace com.ebay namespace with org.apache
> >> > > MYRIAD-44 Prepare for 0.1.0 release
> >> > > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> >> > > MYRIAD-114 Automatic dashboard building
> >> > > MYRIAD-145 Document Myriad Release Process
> >> > > MYRIAD-150 Update NOTICE file
> >> > > MYRIAD-159 Change default mesos version to 0.24
> >> > >
> >> > > Unless anyone thinks there are other JIRAs that are show stoppers, I
> >> > think
> >> > > we should stick to the above list
> >> > > and cut a RC as soon as we address the above.
> >> > >
> >> > > **I'm positive the above can be fixed by early next week (10/27) and
> >> we
> >> > can
> >> > > have a RC out for voting mid next week (10/28).**
> >> > >
> >> > > If we can't fix the above JIRAs in time or if new ones come up as
> >> "show
> >> > > stoppers", we will have a revised date.
> >> > > And, of course, more fixes are welcome, as long as they can be
> merged
> >> > > before 10/27.
> >> > >
> >> > > Just to let everyone know about the Apache release process (@Adam,
> >> feel
> >> > > free to chime in):
> >> > >  - Apache requires that a RC be put out for voting on
> >> > dev@myriad.incubator
> >> > > for 72 hrs or until 3 binding +1s and no binding -1s,
> >> > >  - followed by a similar voting round on general@incubator.
> >> > >
> >> > > Now to run the last mile..!
> 

[jira] [Resolved] (MYRIAD-159) Change default mesos version to 0.24

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella resolved MYRIAD-159.

Resolution: Fixed

> Change default mesos version to 0.24
> 
>
> Key: MYRIAD-159
> URL: https://issues.apache.org/jira/browse/MYRIAD-159
> Project: Myriad
>  Issue Type: Improvement
>  Components: Scheduler
>Reporter: Yuliya Feldman
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> Since Mesos moved to 0.25 version by now it would be beneficial to change 
> Mesos version with which myriad is built to 0.24 at least



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-159) Change default mesos version to 0.24

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella updated MYRIAD-159:
---
Assignee: DarinJ  (was: Yuliya Feldman)

> Change default mesos version to 0.24
> 
>
> Key: MYRIAD-159
> URL: https://issues.apache.org/jira/browse/MYRIAD-159
> Project: Myriad
>  Issue Type: Improvement
>  Components: Scheduler
>Reporter: Yuliya Feldman
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> Since Mesos moved to 0.25 version by now it would be beneficial to change 
> Mesos version with which myriad is built to 0.24 at least



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Please Read] Git branch cleanup

[Santosh Marella]

Just want to do a quick check on the following:

a. Is anyone using a branch other than 'master' from apache/incubator-myriad
 repo?
b. Is anyone (still) using the old git repo mesos/myriad
 ?

Unless there are any objections, we can delete the unused/old git branches
from apache/incubator-myriad and perhaps the whole mesos/myriad repository.

Thanks,
Santosh

[jira] [Commented] (MYRIAD-159) Change default mesos version to 0.24

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980685#comment-14980685
 ] 

Santosh Marella commented on MYRIAD-159:


Moved the Mesos version to 0.24.1.
Fixed with https://github.com/apache/incubator-myriad/pull/23

> Change default mesos version to 0.24
> 
>
> Key: MYRIAD-159
> URL: https://issues.apache.org/jira/browse/MYRIAD-159
> Project: Myriad
>  Issue Type: Improvement
>  Components: Scheduler
>Reporter: Yuliya Feldman
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> Since Mesos moved to 0.25 version by now it would be beneficial to change 
> Mesos version with which myriad is built to 0.24 at least



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA work for 0.1.0

[Santosh Marella]

I agree. I'll send out a separate email on that.

Let's come back to the status of the release again.

* MYRIAD-114 <= AFAIK this is the only blocker for the release at the
moment.
* MYRIAD-43 and MYRIAD-98 are fixed (albeit a couple of minor PRs Jim
submitted this morning)
* MYRIAD-150: Jim, do you think anything is remaining? Can we close this?
* MYRIAD-159: Fixed
* MYRIAD-160: Almost Fixed (Yuliya is addressing a review comment)
* MYRIAD-145,MYRIAD-44: On me, will be closed just before the release.

Thanks,
Santosh

On Thu, Oct 29, 2015 at 5:09 AM, Jim Klucar  wrote:

> Santosh,
>
> How about deleting some old/dead branches from git before the release?
>
> Jim
>
> On Tue, Oct 27, 2015 at 5:57 PM, Santosh Marella 
> wrote:
>
> > Hello everyone,
> >
> >   Just a quick update on the release status. We still need the following
> > JIRAs to be resolved.
> >
> >   * MYRIAD-43 Replace com.ebay namespace with org.apache
> >   * MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> >  - Jim is working on both of them
> >
> >   * MYRIAD-114 Automatic dashboard building
> >  - Not sure if anyone is working on this one.
> >The last comment on JIRA indicates Jim and Ken are collaborating
> to
> > resolve this.
> >
> >   * MYRIAD-150 Update NOTICE file
> >  - I think this is done. The JIRA has to be just marked as Resolved.
> >
> >   * MYRIAD-159 Change default mesos version to 0.24
> >  - A minor one line change, but someone still needs to do it.
> >
> >   * MYRIAD-145 Document Myriad Release Process
> >  - Created a draft on the wiki. Will update it as we tag the RC and
> > publish it for voting.
> >   * MYRIAD-44 Prepare for 0.1.0 release
> >  - Just an Umbrella JIRA keeping track of the individual items.
> >
> >   In addition, Yuliya submitted a PR for MYRIAD-160.
> >
> > Thanks,
> > Santosh
> >
> > On Fri, Oct 23, 2015 at 11:32 AM, Santosh Marella  >
> > wrote:
> >
> > > > let Myriad pick a random port outside of Mesos
> > >
> > > Agree with Darin. Picking a port outside of Mesos is prohibitive.
> > >
> > > I felt the simplest solution would be to pick a random port from the
> list
> > > of ports offered to Myriad.
> > >
> > > The right solution, as Darin mentioned in the JIRA, would be to remove
> NM
> > > from RM's memory.
> > > I haven't looked at YARN's code deep enough to see if that's possible.
> > But
> > > that's something worth exploring.
> > >
> > > Thanks,
> > > Santosh
> > >
> > > On Fri, Oct 23, 2015 at 9:57 AM, Darin Johnson <
> dbjohnson1...@gmail.com>
> > > wrote:
> > >
> > >> On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
> > >> outside of Mesos as other frameworks might select that port the
> > >> probability
> > >> of that happening is 1-(number of ports requested by other
> > >> frameworks)/(number of ports in use).  This could get near 50% and
> cause
> > >> frameworks that are being good citizens to crash.
> > >>
> > >> I'm also not convinced randomizing the port is in fact the correct fix
> > for
> > >> this in the long term, as there is still a non-zero chance you'll get
> > that
> > >> port again.
> > >>
> > >> Darin
> > >>
> > >>
> > >>
> > >> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> > >> yufeld...@yahoo.com.invalid> wrote:
> > >>
> > >> > Great list.
> > >> > I would include MYRIAD-160 to the list - I am working on that one.
> Of
> > >> > course as a workaround we could not use Mesos ports and let NM ports
> > >> > randomization kick in.I also almost done with MYRIAD-148 - it was
> > really
> > >> > tricky. Should submit PR tonight.
> > >> > Thanks,Yuliya
> > >> >   From: Darin Johnson 
> > >> >  To: Dev 
> > >> >  Sent: Thursday, October 22, 2015 8:29 PM
> > >> >  Subject: Re: JIRA work for 0.1.0
> > >> >
> > >> > I think this sounds good about right.  A few Jim marked were new
> > >> features,
> > >> > gotta leave something for the 0.2.0 release :).
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <
> > smare...@maprtech.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > I looked at the JIRAs currently marked with fix version
> > >> "myriad-0.1.0".
> > >> > > There were 19 of them. I moved a few out. We are currently at 14.
> > >> > >
> > >> > > However, IMO the show stoppers are really the following:
> > >> > >
> > >> > > MYRIAD-43 Replace com.ebay namespace with org.apache
> > >> > > MYRIAD-44 Prepare for 0.1.0 release
> > >> > > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > >> > > MYRIAD-114 Automatic dashboard building
> > >> > > MYRIAD-145 Document Myriad Release Process
> > >> > > MYRIAD-150 Update NOTICE file
> > >> > > MYRIAD-159 Change default mesos version to 0.24
> > >> > >
> > >> > > Unless anyone thinks there are other JIRAs that are show
> stoppers, I
> > >> > think
> > >> > > we should stick to the above list
> > 

[jira] [Updated] (MYRIAD-160) Randomize ports on which NM starts

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella updated MYRIAD-160:
---
Fix Version/s: Myriad 0.1.0

> Randomize ports on which NM starts
> --
>
> Key: MYRIAD-160
> URL: https://issues.apache.org/jira/browse/MYRIAD-160
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Yuliya Feldman
> Fix For: Myriad 0.1.0
>
>
> When a CGS node is present and gets replaced by a FGS node after a flex-down 
> and flex-up, the memory calculation is going wrong. I see the applications 
> hang and the AM container fails to get scheduled.
> By default when a NM is started, it is started on a random port. Only the web 
> server port is fixed to 8042. 
> The behavior should be similar in Myriad. We need to randomize the port of 
> the NM from the offer obtained from mesos. So when we flex down and flex-up 
> it starts on a different port. Otherwise Myriad considers this as the same NM 
> with different resources. Hence the memory calculation issue. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980728#comment-14980728
 ] 

DarinJ commented on MYRIAD-162:
---

I've got a pretty good handle on how to fix should be ready for review by 
Monday/Tuesday.  

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus(*): 1.2; mem(*): 
> 1305.6; ports(*): [31005-31005,31006-31006,...] greater than offered 
> cpu(*):1, mem(*): 1400, ports(*): [ ... ], cpu(roleA): 3, mem(roleA): 1, 
> ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Please Read] Git branch cleanup

[Jim Klucar]

I'm good to go. You might want to keep the old mesos/myriad repo if people
have it forked or bookmarked. You can just delete everything and put a
README commit pointing to the apache stuff.

On Thu, Oct 29, 2015 at 12:13 PM, Santosh Marella 
wrote:

> Just want to do a quick check on the following:
>
> a. Is anyone using a branch other than 'master' from
> apache/incubator-myriad
>  repo?
> b. Is anyone (still) using the old git repo mesos/myriad
>  ?
>
> Unless there are any objections, we can delete the unused/old git branches
> from apache/incubator-myriad and perhaps the whole mesos/myriad repository.
>
> Thanks,
> Santosh
>

[jira] [Commented] (MYRIAD-150) Update NOTICE file

[Jim Klucar (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980898#comment-14980898
 ] 

Jim Klucar commented on MYRIAD-150:
---

I just re-read the apache site regarding this stuff and what we have looks ok 
to me. I'm sure incubator mentors will review and let us know if we went wrong.

> Update NOTICE file
> --
>
> Key: MYRIAD-150
> URL: https://issues.apache.org/jira/browse/MYRIAD-150
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Jim Klucar
>Assignee: Jim Klucar
> Fix For: Myriad 0.1.0
>
>
> The NOTICE file needs to be updated. I know we no long use dropwizard, and 
> perhaps some others. The WebUI uses a TON of packages, not sure if we need to 
> list them all or in a separate file or what. I used NPM to handle the 
> dependencies and there's a script to go grab all the licenses if we need 
> them. At one point I did a scrub to make sure they're all Apache2 compatable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DarinJ updated MYRIAD-162:
--
Description: When using Offers that have Resources from multiple roles, one 
needs to use the setRole(String role) method to specify which role the resource 
belongs to.  Myriad currently doesn't do this which causes TASK_LOST, with an 
error in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( 
* ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
1, ports(roleA): [...].  (was: When using Offers that have Resources from 
multiple roles, one needs to use the setRole(String role) method to specify 
which role the resource belongs to.  Myriad currently doesn't do this which 
causes TASK_LOST, with an error in the mesos-master log stating in "attempted 
to use cpus(*): 1.2; mem(*): 1305.6; ports(*): [31005-31005,31006-31006,...] 
greater than offered cpu(*):1, mem(*): 1400, ports(*): [ ... ], cpu(roleA): 3, 
mem(roleA): 1, ports(roleA): [...].)

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (MYRIAD-150) Update NOTICE file

[Jim Klucar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Klucar closed MYRIAD-150.
-
Resolution: Fixed

> Update NOTICE file
> --
>
> Key: MYRIAD-150
> URL: https://issues.apache.org/jira/browse/MYRIAD-150
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Jim Klucar
>Assignee: Jim Klucar
> Fix For: Myriad 0.1.0
>
>
> The NOTICE file needs to be updated. I know we no long use dropwizard, and 
> perhaps some others. The WebUI uses a TON of packages, not sure if we need to 
> list them all or in a separate file or what. I used NPM to handle the 
> dependencies and there's a script to go grab all the licenses if we need 
> them. At one point I did a scrub to make sure they're all Apache2 compatable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980815#comment-14980815
 ] 

DarinJ commented on MYRIAD-162:
---

Close but not quite.  We want Myriad to be able to use Resources from both the 
frameworkUserRole AND the defaultRole (usually *).  So we need to be aware of 
both, otherwise you're essentially statically partitioning your data center 
instead of reserving resources. The idea isn't hard and shouldn't require a 
major refactor.  I'm prototyping now, will know more soon.

If you wish to recreate simply add a role with some cpu resources to you're 
cluster and set frameworkRole to that role.  You may need to ensure you're 
tasks require enough cpus/mem that they have to use the new role.

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA work for 0.1.0

[Jim Klucar]

I closed MYRIAD-150.

I'll play with getting MYRIAD-114 to auto-build this afternoon unless Ken
chimes in. It just needs a hook into the main gradle script to kick it off.

Only thing missing from the list is MYRIAD-144 - the podling site. I think
we should have this worked out before we release.

On Thu, Oct 29, 2015 at 12:08 PM, Santosh Marella 
wrote:

> I agree. I'll send out a separate email on that.
>
> Let's come back to the status of the release again.
>
> * MYRIAD-114 <= AFAIK this is the only blocker for the release at the
> moment.
> * MYRIAD-43 and MYRIAD-98 are fixed (albeit a couple of minor PRs Jim
> submitted this morning)
> * MYRIAD-150: Jim, do you think anything is remaining? Can we close this?
> * MYRIAD-159: Fixed
> * MYRIAD-160: Almost Fixed (Yuliya is addressing a review comment)
> * MYRIAD-145,MYRIAD-44: On me, will be closed just before the release.
>
> Thanks,
> Santosh
>
> On Thu, Oct 29, 2015 at 5:09 AM, Jim Klucar  wrote:
>
> > Santosh,
> >
> > How about deleting some old/dead branches from git before the release?
> >
> > Jim
> >
> > On Tue, Oct 27, 2015 at 5:57 PM, Santosh Marella 
> > wrote:
> >
> > > Hello everyone,
> > >
> > >   Just a quick update on the release status. We still need the
> following
> > > JIRAs to be resolved.
> > >
> > >   * MYRIAD-43 Replace com.ebay namespace with org.apache
> > >   * MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > >  - Jim is working on both of them
> > >
> > >   * MYRIAD-114 Automatic dashboard building
> > >  - Not sure if anyone is working on this one.
> > >The last comment on JIRA indicates Jim and Ken are collaborating
> > to
> > > resolve this.
> > >
> > >   * MYRIAD-150 Update NOTICE file
> > >  - I think this is done. The JIRA has to be just marked as
> Resolved.
> > >
> > >   * MYRIAD-159 Change default mesos version to 0.24
> > >  - A minor one line change, but someone still needs to do it.
> > >
> > >   * MYRIAD-145 Document Myriad Release Process
> > >  - Created a draft on the wiki. Will update it as we tag the RC and
> > > publish it for voting.
> > >   * MYRIAD-44 Prepare for 0.1.0 release
> > >  - Just an Umbrella JIRA keeping track of the individual items.
> > >
> > >   In addition, Yuliya submitted a PR for MYRIAD-160.
> > >
> > > Thanks,
> > > Santosh
> > >
> > > On Fri, Oct 23, 2015 at 11:32 AM, Santosh Marella <
> smare...@maprtech.com
> > >
> > > wrote:
> > >
> > > > > let Myriad pick a random port outside of Mesos
> > > >
> > > > Agree with Darin. Picking a port outside of Mesos is prohibitive.
> > > >
> > > > I felt the simplest solution would be to pick a random port from the
> > list
> > > > of ports offered to Myriad.
> > > >
> > > > The right solution, as Darin mentioned in the JIRA, would be to
> remove
> > NM
> > > > from RM's memory.
> > > > I haven't looked at YARN's code deep enough to see if that's
> possible.
> > > But
> > > > that's something worth exploring.
> > > >
> > > > Thanks,
> > > > Santosh
> > > >
> > > > On Fri, Oct 23, 2015 at 9:57 AM, Darin Johnson <
> > dbjohnson1...@gmail.com>
> > > > wrote:
> > > >
> > > >> On MYRIAD-160, it's a really bad idea to let Myriad pick a random
> port
> > > >> outside of Mesos as other frameworks might select that port the
> > > >> probability
> > > >> of that happening is 1-(number of ports requested by other
> > > >> frameworks)/(number of ports in use).  This could get near 50% and
> > cause
> > > >> frameworks that are being good citizens to crash.
> > > >>
> > > >> I'm also not convinced randomizing the port is in fact the correct
> fix
> > > for
> > > >> this in the long term, as there is still a non-zero chance you'll
> get
> > > that
> > > >> port again.
> > > >>
> > > >> Darin
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> > > >> yufeld...@yahoo.com.invalid> wrote:
> > > >>
> > > >> > Great list.
> > > >> > I would include MYRIAD-160 to the list - I am working on that one.
> > Of
> > > >> > course as a workaround we could not use Mesos ports and let NM
> ports
> > > >> > randomization kick in.I also almost done with MYRIAD-148 - it was
> > > really
> > > >> > tricky. Should submit PR tonight.
> > > >> > Thanks,Yuliya
> > > >> >   From: Darin Johnson 
> > > >> >  To: Dev 
> > > >> >  Sent: Thursday, October 22, 2015 8:29 PM
> > > >> >  Subject: Re: JIRA work for 0.1.0
> > > >> >
> > > >> > I think this sounds good about right.  A few Jim marked were new
> > > >> features,
> > > >> > gotta leave something for the 0.2.0 release :).
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <
> > > smare...@maprtech.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > I looked at the JIRAs currently marked with fix version
> > > >> "myriad-0.1.0".
> > > >> > > There were 19 of them. I moved a 

Re: JIRA work for 0.1.0

[Santosh Marella]

Cool. The Travis build succeeded:
https://travis-ci.org/apache/incubator-myriad/builds/88205610.

I'll try out the UI on my dev setup and merge the PR if the UI comes up
well.

Thanks Jim!

Santosh

On Thu, Oct 29, 2015 at 1:33 PM, Jim Klucar  wrote:

> Just submitted a PR for MYRIAD-144.
>
> https://github.com/apache/incubator-myriad/pull/28
>
> I'll be interested to see if it builds in Travis as gradle will now
> download node, npm, gulp and all that to build the UI.
>
> On Thu, Oct 29, 2015 at 1:51 PM, Jim Klucar  wrote:
>
> > I closed MYRIAD-150.
> >
> > I'll play with getting MYRIAD-114 to auto-build this afternoon unless Ken
> > chimes in. It just needs a hook into the main gradle script to kick it
> off.
> >
> > Only thing missing from the list is MYRIAD-144 - the podling site. I
> think
> > we should have this worked out before we release.
> >
> > On Thu, Oct 29, 2015 at 12:08 PM, Santosh Marella  >
> > wrote:
> >
> >> I agree. I'll send out a separate email on that.
> >>
> >> Let's come back to the status of the release again.
> >>
> >> * MYRIAD-114 <= AFAIK this is the only blocker for the release at the
> >> moment.
> >> * MYRIAD-43 and MYRIAD-98 are fixed (albeit a couple of minor PRs Jim
> >> submitted this morning)
> >> * MYRIAD-150: Jim, do you think anything is remaining? Can we close
> this?
> >> * MYRIAD-159: Fixed
> >> * MYRIAD-160: Almost Fixed (Yuliya is addressing a review comment)
> >> * MYRIAD-145,MYRIAD-44: On me, will be closed just before the release.
> >>
> >> Thanks,
> >> Santosh
> >>
> >> On Thu, Oct 29, 2015 at 5:09 AM, Jim Klucar  wrote:
> >>
> >> > Santosh,
> >> >
> >> > How about deleting some old/dead branches from git before the release?
> >> >
> >> > Jim
> >> >
> >> > On Tue, Oct 27, 2015 at 5:57 PM, Santosh Marella <
> smare...@maprtech.com
> >> >
> >> > wrote:
> >> >
> >> > > Hello everyone,
> >> > >
> >> > >   Just a quick update on the release status. We still need the
> >> following
> >> > > JIRAs to be resolved.
> >> > >
> >> > >   * MYRIAD-43 Replace com.ebay namespace with org.apache
> >> > >   * MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> >> > >  - Jim is working on both of them
> >> > >
> >> > >   * MYRIAD-114 Automatic dashboard building
> >> > >  - Not sure if anyone is working on this one.
> >> > >The last comment on JIRA indicates Jim and Ken are
> >> collaborating
> >> > to
> >> > > resolve this.
> >> > >
> >> > >   * MYRIAD-150 Update NOTICE file
> >> > >  - I think this is done. The JIRA has to be just marked as
> >> Resolved.
> >> > >
> >> > >   * MYRIAD-159 Change default mesos version to 0.24
> >> > >  - A minor one line change, but someone still needs to do it.
> >> > >
> >> > >   * MYRIAD-145 Document Myriad Release Process
> >> > >  - Created a draft on the wiki. Will update it as we tag the RC
> >> and
> >> > > publish it for voting.
> >> > >   * MYRIAD-44 Prepare for 0.1.0 release
> >> > >  - Just an Umbrella JIRA keeping track of the individual items.
> >> > >
> >> > >   In addition, Yuliya submitted a PR for MYRIAD-160.
> >> > >
> >> > > Thanks,
> >> > > Santosh
> >> > >
> >> > > On Fri, Oct 23, 2015 at 11:32 AM, Santosh Marella <
> >> smare...@maprtech.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > > let Myriad pick a random port outside of Mesos
> >> > > >
> >> > > > Agree with Darin. Picking a port outside of Mesos is prohibitive.
> >> > > >
> >> > > > I felt the simplest solution would be to pick a random port from
> the
> >> > list
> >> > > > of ports offered to Myriad.
> >> > > >
> >> > > > The right solution, as Darin mentioned in the JIRA, would be to
> >> remove
> >> > NM
> >> > > > from RM's memory.
> >> > > > I haven't looked at YARN's code deep enough to see if that's
> >> possible.
> >> > > But
> >> > > > that's something worth exploring.
> >> > > >
> >> > > > Thanks,
> >> > > > Santosh
> >> > > >
> >> > > > On Fri, Oct 23, 2015 at 9:57 AM, Darin Johnson <
> >> > dbjohnson1...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > >> On MYRIAD-160, it's a really bad idea to let Myriad pick a random
> >> port
> >> > > >> outside of Mesos as other frameworks might select that port the
> >> > > >> probability
> >> > > >> of that happening is 1-(number of ports requested by other
> >> > > >> frameworks)/(number of ports in use).  This could get near 50%
> and
> >> > cause
> >> > > >> frameworks that are being good citizens to crash.
> >> > > >>
> >> > > >> I'm also not convinced randomizing the port is in fact the
> correct
> >> fix
> >> > > for
> >> > > >> this in the long term, as there is still a non-zero chance you'll
> >> get
> >> > > that
> >> > > >> port again.
> >> > > >>
> >> > > >> Darin
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> >> > > >> yufeld...@yahoo.com.invalid> wrote:
> >> > > >>
> >> > > >> > Great list.
> >> > > >> > I would include 

[jira] [Commented] (MYRIAD-144) Podling Incubator Site

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981298#comment-14981298
 ] 

Santosh Marella commented on MYRIAD-144:


According to https://issues.apache.org/jira/browse/INFRA-8988, kylin guys 
publish their generated artifacts to 
https://svn.apache.org/repos/asf/incubator/kylin and Apache's svnpubsub 
publishes the website to http://kylin.incubator.apache.org. I think Mesos seems 
to follow similar model, except that Mesos has a "site/publish" folder that 
gets published.

Also, Kylin's website code lives in their git at 
https://github.com/apache/incubator-kylin/tree/master/website. The README 
(https://github.com/apache/incubator-kylin/blob/master/website/README.md) 
suggests they generate the artifacts (and test the site locally with 'jekyll 
s') and push the generated artifacts to SVN. 

Shall we follow the same model?

Either ways, it looks like we need a INFRA ticket to have a "myriad" folder 
created under https://svn.apache.org/repos/asf/incubator/. I'll request one of 
our Mentors create a INFRA ticket to have 
https://svn.apache.org/repos/asf/incubator/myriad created for us.


> Podling Incubator Site
> --
>
> Key: MYRIAD-144
> URL: https://issues.apache.org/jira/browse/MYRIAD-144
> Project: Myriad
>  Issue Type: Improvement
>Reporter: Jim Klucar
>Assignee: Jim Klucar
>Priority: Minor
> Fix For: Myriad 0.1.0
>
>
> This is just to track the Incubator site work.
> The site has already been designed and tested using Jekyll (http://jekyll.rb) 
> Preview is available at http://klucar.github.io/myriad-site



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Help Needed from Mentors] Creation of a SVN folder for Myriad

[Santosh Marella]

Dear Mentors,

  Request your help on a couple of INFRA items to help launch Apache
website for Myriad.

1. Myriad needs a "myriad" folder under
https://svn.apache.org/repos/asf/incubator/ to place the artifacts required
for the website.

2. The contents of
https://svn.apache.org/repos/asf/incubator/myriad/site need to be published
to http://myriad.incubator.apache.org.

Thanks,
Santosh

[jira] [Commented] (MYRIAD-144) Podling Incubator Site

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981326#comment-14981326
 ] 

Santosh Marella commented on MYRIAD-144:


Posting [~jklucar]'s reply from dev@:

I was just looking at this! I think your analysis is correct. I can put the
site in a top level directory site/ and then jekyll will output everything
in site/_site/ We can then push _site to the correct place in SVN. (not
sure how to do that yet)  Let me know if you think of a better name for the
top level directory.

> Podling Incubator Site
> --
>
> Key: MYRIAD-144
> URL: https://issues.apache.org/jira/browse/MYRIAD-144
> Project: Myriad
>  Issue Type: Improvement
>Reporter: Jim Klucar
>Assignee: Jim Klucar
>Priority: Minor
> Fix For: Myriad 0.1.0
>
>
> This is just to track the Incubator site work.
> The site has already been designed and tested using Jekyll (http://jekyll.rb) 
> Preview is available at http://klucar.github.io/myriad-site



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-144) Podling Incubator Site

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981327#comment-14981327
 ] 

Santosh Marella commented on MYRIAD-144:


How about "website". "site" and "_site" are too close to cause confusion.

> Podling Incubator Site
> --
>
> Key: MYRIAD-144
> URL: https://issues.apache.org/jira/browse/MYRIAD-144
> Project: Myriad
>  Issue Type: Improvement
>Reporter: Jim Klucar
>Assignee: Jim Klucar
>Priority: Minor
> Fix For: Myriad 0.1.0
>
>
> This is just to track the Incubator site work.
> The site has already been designed and tested using Jekyll (http://jekyll.rb) 
> Preview is available at http://klucar.github.io/myriad-site



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA work for 0.1.0

[Jim Klucar]

Just submitted a PR for MYRIAD-144.

https://github.com/apache/incubator-myriad/pull/28

I'll be interested to see if it builds in Travis as gradle will now
download node, npm, gulp and all that to build the UI.

On Thu, Oct 29, 2015 at 1:51 PM, Jim Klucar  wrote:

> I closed MYRIAD-150.
>
> I'll play with getting MYRIAD-114 to auto-build this afternoon unless Ken
> chimes in. It just needs a hook into the main gradle script to kick it off.
>
> Only thing missing from the list is MYRIAD-144 - the podling site. I think
> we should have this worked out before we release.
>
> On Thu, Oct 29, 2015 at 12:08 PM, Santosh Marella 
> wrote:
>
>> I agree. I'll send out a separate email on that.
>>
>> Let's come back to the status of the release again.
>>
>> * MYRIAD-114 <= AFAIK this is the only blocker for the release at the
>> moment.
>> * MYRIAD-43 and MYRIAD-98 are fixed (albeit a couple of minor PRs Jim
>> submitted this morning)
>> * MYRIAD-150: Jim, do you think anything is remaining? Can we close this?
>> * MYRIAD-159: Fixed
>> * MYRIAD-160: Almost Fixed (Yuliya is addressing a review comment)
>> * MYRIAD-145,MYRIAD-44: On me, will be closed just before the release.
>>
>> Thanks,
>> Santosh
>>
>> On Thu, Oct 29, 2015 at 5:09 AM, Jim Klucar  wrote:
>>
>> > Santosh,
>> >
>> > How about deleting some old/dead branches from git before the release?
>> >
>> > Jim
>> >
>> > On Tue, Oct 27, 2015 at 5:57 PM, Santosh Marella > >
>> > wrote:
>> >
>> > > Hello everyone,
>> > >
>> > >   Just a quick update on the release status. We still need the
>> following
>> > > JIRAs to be resolved.
>> > >
>> > >   * MYRIAD-43 Replace com.ebay namespace with org.apache
>> > >   * MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
>> > >  - Jim is working on both of them
>> > >
>> > >   * MYRIAD-114 Automatic dashboard building
>> > >  - Not sure if anyone is working on this one.
>> > >The last comment on JIRA indicates Jim and Ken are
>> collaborating
>> > to
>> > > resolve this.
>> > >
>> > >   * MYRIAD-150 Update NOTICE file
>> > >  - I think this is done. The JIRA has to be just marked as
>> Resolved.
>> > >
>> > >   * MYRIAD-159 Change default mesos version to 0.24
>> > >  - A minor one line change, but someone still needs to do it.
>> > >
>> > >   * MYRIAD-145 Document Myriad Release Process
>> > >  - Created a draft on the wiki. Will update it as we tag the RC
>> and
>> > > publish it for voting.
>> > >   * MYRIAD-44 Prepare for 0.1.0 release
>> > >  - Just an Umbrella JIRA keeping track of the individual items.
>> > >
>> > >   In addition, Yuliya submitted a PR for MYRIAD-160.
>> > >
>> > > Thanks,
>> > > Santosh
>> > >
>> > > On Fri, Oct 23, 2015 at 11:32 AM, Santosh Marella <
>> smare...@maprtech.com
>> > >
>> > > wrote:
>> > >
>> > > > > let Myriad pick a random port outside of Mesos
>> > > >
>> > > > Agree with Darin. Picking a port outside of Mesos is prohibitive.
>> > > >
>> > > > I felt the simplest solution would be to pick a random port from the
>> > list
>> > > > of ports offered to Myriad.
>> > > >
>> > > > The right solution, as Darin mentioned in the JIRA, would be to
>> remove
>> > NM
>> > > > from RM's memory.
>> > > > I haven't looked at YARN's code deep enough to see if that's
>> possible.
>> > > But
>> > > > that's something worth exploring.
>> > > >
>> > > > Thanks,
>> > > > Santosh
>> > > >
>> > > > On Fri, Oct 23, 2015 at 9:57 AM, Darin Johnson <
>> > dbjohnson1...@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> On MYRIAD-160, it's a really bad idea to let Myriad pick a random
>> port
>> > > >> outside of Mesos as other frameworks might select that port the
>> > > >> probability
>> > > >> of that happening is 1-(number of ports requested by other
>> > > >> frameworks)/(number of ports in use).  This could get near 50% and
>> > cause
>> > > >> frameworks that are being good citizens to crash.
>> > > >>
>> > > >> I'm also not convinced randomizing the port is in fact the correct
>> fix
>> > > for
>> > > >> this in the long term, as there is still a non-zero chance you'll
>> get
>> > > that
>> > > >> port again.
>> > > >>
>> > > >> Darin
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
>> > > >> yufeld...@yahoo.com.invalid> wrote:
>> > > >>
>> > > >> > Great list.
>> > > >> > I would include MYRIAD-160 to the list - I am working on that
>> one.
>> > Of
>> > > >> > course as a workaround we could not use Mesos ports and let NM
>> ports
>> > > >> > randomization kick in.I also almost done with MYRIAD-148 - it was
>> > > really
>> > > >> > tricky. Should submit PR tonight.
>> > > >> > Thanks,Yuliya
>> > > >> >   From: Darin Johnson 
>> > > >> >  To: Dev 
>> > > >> >  Sent: Thursday, October 22, 2015 8:29 PM
>> > > >> >  Subject: Re: JIRA work for 0.1.0
>> > > >> >
>> > > >> > I think this 

[jira] [Commented] (MYRIAD-114) Automatic dashboard building

[Jim Klucar (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981250#comment-14981250
 ] 

Jim Klucar commented on MYRIAD-114:
---

Submitted PR.

https://github.com/apache/incubator-myriad/pull/28

gradle now calls 'gulp build' as part of myriad-scheduler build

> Automatic dashboard building
> 
>
> Key: MYRIAD-114
> URL: https://issues.apache.org/jira/browse/MYRIAD-114
> Project: Myriad
>  Issue Type: Bug
>Reporter: Jim Klucar
>Assignee: Jim Klucar
> Fix For: Myriad 0.1.0
>
>
> Build the dashboard as part of the Myriad build process.  This would require:
> - [ ] npm install of dashboard dependencies
> - [ ] removing files under webapp/public (could be part of gulp)
> - [ ] running gulp script
> Benefits: 
> Can remove myriad-scheduler/src/main/resources/webapp/public/* from git. This 
> is auto-generated each time gulp is run, so any diffs in this directory don't 
> mean anything and are just noise.
> Drawbacks:
> Users must have npm/node installed to build Myriad. (maybe we only build if 
> they have node installed?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (MYRIAD-144) Podling Incubator Site

[Jim Klucar]

I was just looking at this! I think your analysis is correct. I can put the
site in a top level directory site/ and then jekyll will output everything
in site/_site/ We can then push _site to the correct place in SVN. (not
sure how to do that yet)  Let me know if you think of a better name for the
top level directory.

On Thu, Oct 29, 2015 at 5:18 PM, Santosh Marella (JIRA) 
wrote:

>
> [
> https://issues.apache.org/jira/browse/MYRIAD-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981298#comment-14981298
> ]
>
> Santosh Marella commented on MYRIAD-144:
> 
>
> According to https://issues.apache.org/jira/browse/INFRA-8988, kylin guys
> publish their generated artifacts to
> https://svn.apache.org/repos/asf/incubator/kylin and Apache's svnpubsub
> publishes the website to http://kylin.incubator.apache.org. I think Mesos
> seems to follow similar model, except that Mesos has a "site/publish"
> folder that gets published.
>
> Also, Kylin's website code lives in their git at
> https://github.com/apache/incubator-kylin/tree/master/website. The README
> (https://github.com/apache/incubator-kylin/blob/master/website/README.md)
> suggests they generate the artifacts (and test the site locally with
> 'jekyll s') and push the generated artifacts to SVN.
>
> Shall we follow the same model?
>
> Either ways, it looks like we need a INFRA ticket to have a "myriad"
> folder created under https://svn.apache.org/repos/asf/incubator/. I'll
> request one of our Mentors create a INFRA ticket to have
> https://svn.apache.org/repos/asf/incubator/myriad created for us.
>
>
> > Podling Incubator Site
> > --
> >
> > Key: MYRIAD-144
> > URL: https://issues.apache.org/jira/browse/MYRIAD-144
> > Project: Myriad
> >  Issue Type: Improvement
> >Reporter: Jim Klucar
> >Assignee: Jim Klucar
> >Priority: Minor
> > Fix For: Myriad 0.1.0
> >
> >
> > This is just to track the Incubator site work.
> > The site has already been designed and tested using Jekyll (
> http://jekyll.rb) Preview is available at
> http://klucar.github.io/myriad-site
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

[jira] [Resolved] (MYRIAD-160) Randomize ports on which NM starts

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella resolved MYRIAD-160.

Resolution: Fixed

> Randomize ports on which NM starts
> --
>
> Key: MYRIAD-160
> URL: https://issues.apache.org/jira/browse/MYRIAD-160
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Yuliya Feldman
> Fix For: Myriad 0.1.0
>
>
> When a CGS node is present and gets replaced by a FGS node after a flex-down 
> and flex-up, the memory calculation is going wrong. I see the applications 
> hang and the AM container fails to get scheduled.
> By default when a NM is started, it is started on a random port. Only the web 
> server port is fixed to 8042. 
> The behavior should be similar in Myriad. We need to randomize the port of 
> the NM from the offer obtained from mesos. So when we flex down and flex-up 
> it starts on a different port. Otherwise Myriad considers this as the same NM 
> with different resources. Hence the memory calculation issue. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-160) Randomize ports on which NM starts

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981354#comment-14981354
 ] 

Santosh Marella commented on MYRIAD-160:


And PR: https://github.com/apache/incubator-myriad/pull/27

> Randomize ports on which NM starts
> --
>
> Key: MYRIAD-160
> URL: https://issues.apache.org/jira/browse/MYRIAD-160
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Yuliya Feldman
> Fix For: Myriad 0.1.0
>
>
> When a CGS node is present and gets replaced by a FGS node after a flex-down 
> and flex-up, the memory calculation is going wrong. I see the applications 
> hang and the AM container fails to get scheduled.
> By default when a NM is started, it is started on a random port. Only the web 
> server port is fixed to 8042. 
> The behavior should be similar in Myriad. We need to randomize the port of 
> the NM from the offer obtained from mesos. So when we flex down and flex-up 
> it starts on a different port. Otherwise Myriad considers this as the same NM 
> with different resources. Hence the memory calculation issue. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

DarinJ created MYRIAD-162:
-

 Summary: Myriad Not Correctly Dealing with Resources from Multiple 
Roles
 Key: MYRIAD-162
 URL: https://issues.apache.org/jira/browse/MYRIAD-162
 Project: Myriad
  Issue Type: Bug
  Components: Scheduler
Affects Versions: Myriad 0.1.0
 Environment: Any where frameworkRole is not *
Reporter: DarinJ
 Fix For: Myriad 0.1.0


When using Offers that have Resources from multiple roles, one needs to use the 
setRole(String role) method to specify which role the resource belongs to.  
Myriad currently doesn't do this which causes TASK_LOST, with an error in the 
mesos-master log stating in "attempted to use cpus(*): 1.2; mem(*): 1305.6; 
ports(*): [31005-31005,31006-31006,...] greater than offered cpu(*):1, mem(*): 
1400, ports(*): [ ... ], cpu(roleA): 3, mem(roleA): 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980766#comment-14980766
 ] 

Santosh Marella commented on MYRIAD-162:


Looks like an important issue. Is the fix to add a "role" in the 
myriad-config-default.yml and have Myriad use that role when picking offers 
from Mesos?

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981545#comment-14981545
 ] 

Santosh Marella commented on MYRIAD-162:


Sounds good.

I'm trying to get everyone close-in on the fixes/changes required for the 0.1.0 
release. 
Without a fix for this, Myriad would still work when "roles" are not used in 
Mesos. 

Do you think the current behavior is good enough for a 0.1.0 release?

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981698#comment-14981698
 ] 

DarinJ edited comment on MYRIAD-162 at 10/30/15 5:01 AM:
-

Without a fix Myriad will work if frameworkRole is not present or set to *, but 
fail badly if a frameworkRole is present.  For a 0.1.0 release, the easiest 
nice things to do are 1. add a check if frameworkRole is present and log a 
warning and exit or 2. log and then set role to * anyway.  After those options 
it's simply to fix it.  I'm close to a fix (mem/cpu done working ports now) and 
would like to have it for 0.1.0, but it can wait for 0.1.1.

Current copy of changes here: https://github.com/darinj/incubator-myriad

Todo ports (still).  This could be delayed with the assumption ports aren't 
generally reserved for random ports, then we just force Myriad to get ports 
from the default role (done in TaskFactoryImpl, would need to do for 
ServiceTaskFactoryImpl).


was (Author: darinj):
Without a fix Myriad will work if frameworkRole is not present or set to *, but 
fail badly if a frameworkRole is present.  For a 0.1.0 release, the easiest 
nice things to do are 1. add a check if frameworkRole is present and log a 
warning and exit or 2. log and then set role to * anyway.  After those options 
it's simply to fix it.  I'm close to a fix (mem/cpu done working ports now) and 
would like to have it for 0.1.0, but it can wait for 0.1.1.

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-162) Myriad Not Correctly Dealing with Resources from Multiple Roles

[DarinJ (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981698#comment-14981698
 ] 

DarinJ commented on MYRIAD-162:
---

Without a fix Myriad will work if frameworkRole is not present or set to *, but 
fail badly if a frameworkRole is present.  For a 0.1.0 release, the easiest 
nice things to do are 1. add a check if frameworkRole is present and log a 
warning and exit or 2. log and then set role to * anyway.  After those options 
it's simply to fix it.  I'm close to a fix (mem/cpu done working ports now) and 
would like to have it for 0.1.0, but it can wait for 0.1.1.

> Myriad Not Correctly Dealing with Resources from Multiple Roles
> ---
>
> Key: MYRIAD-162
> URL: https://issues.apache.org/jira/browse/MYRIAD-162
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Affects Versions: Myriad 0.1.0
> Environment: Any where frameworkRole is not *
>Reporter: DarinJ
>Assignee: DarinJ
> Fix For: Myriad 0.1.0
>
>
> When using Offers that have Resources from multiple roles, one needs to use 
> the setRole(String role) method to specify which role the resource belongs 
> to.  Myriad currently doesn't do this which causes TASK_LOST, with an error 
> in the mesos-master log stating in "attempted to use cpus( * ): 1.2; mem( * 
> ): 1305.6; ports( * ): [31005-31005,31006-31006,...] greater than offered 
> cpu( * ):1, mem( * ): 1400, ports( * ): [ ... ], cpu(roleA): 3, mem(roleA): 
> 1, ports(roleA): [...].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)