Re: Myriad 0.1 release scope
Ok, I was going off https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md I will try it. John On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: You actually do not need to rebuild even today - just keep this file in hadoop config directory that is on the classpath: like .../etc/hadoop From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org Sent: Tuesday, August 18, 2015 1:35 PM Subject: Re: Myriad 0.1 release scope On the release scope, will having the myriad configuration file exist outside the jar (i.e. you can change configuration without rebuilding) be part of the .1 release scope? On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com wrote: Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
Ok, so I tried the remote distribution of the Myriad per the docs, I guess,it could probably use some information related to how to run resource manager if it's in the tar.gz. Perhaps an example marathon json. I am playing with it now to figure it out. On Tue, Aug 18, 2015 at 3:48 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: mesos/myriad is the right one so far From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com Sent: Tuesday, August 18, 2015 1:44 PM Subject: Re: Myriad 0.1 release scope (So if I clone that repo, am I cloning the right one?) On Tue, Aug 18, 2015 at 3:43 PM, John Omernik j...@omernik.com wrote: Ok, I was going off https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md I will try it. John On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: You actually do not need to rebuild even today - just keep this file in hadoop config directory that is on the classpath: like .../etc/hadoop From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org Sent: Tuesday, August 18, 2015 1:35 PM Subject: Re: Myriad 0.1 release scope On the release scope, will having the myriad configuration file exist outside the jar (i.e. you can change configuration without rebuilding) be part of the .1 release scope? On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com wrote: Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
On the release scope, will having the myriad configuration file exist outside the jar (i.e. you can change configuration without rebuilding) be part of the .1 release scope? On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com wrote: Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
mesos/myriad is the right one so far From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com Sent: Tuesday, August 18, 2015 1:44 PM Subject: Re: Myriad 0.1 release scope (So if I clone that repo, am I cloning the right one?) On Tue, Aug 18, 2015 at 3:43 PM, John Omernik j...@omernik.com wrote: Ok, I was going off https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md I will try it. John On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: You actually do not need to rebuild even today - just keep this file in hadoop config directory that is on the classpath: like .../etc/hadoop From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org Sent: Tuesday, August 18, 2015 1:35 PM Subject: Re: Myriad 0.1 release scope On the release scope, will having the myriad configuration file exist outside the jar (i.e. you can change configuration without rebuilding) be part of the .1 release scope? On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com wrote: Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
John, The remote distribution doesn't require the nm to be run from marathon though it's possible. Essentially, it's the same configuration for the rm you'd do for the non remote version + adding a uri for the tarball. I've got jsons for running the rm in marathon, I'll try to get them and some documentation up soon. Currently at a conference though which means probably next week. Darin Darin On Aug 18, 2015 2:49 PM, John Omernik j...@omernik.com wrote: Ok, so I tried the remote distribution of the Myriad per the docs, I guess,it could probably use some information related to how to run resource manager if it's in the tar.gz. Perhaps an example marathon json. I am playing with it now to figure it out. On Tue, Aug 18, 2015 at 3:48 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: mesos/myriad is the right one so far From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org; yuliya Feldman yufeld...@yahoo.com Sent: Tuesday, August 18, 2015 1:44 PM Subject: Re: Myriad 0.1 release scope (So if I clone that repo, am I cloning the right one?) On Tue, Aug 18, 2015 at 3:43 PM, John Omernik j...@omernik.com wrote: Ok, I was going off https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md I will try it. John On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman yufeld...@yahoo.com.invalid wrote: You actually do not need to rebuild even today - just keep this file in hadoop config directory that is on the classpath: like .../etc/hadoop From: John Omernik j...@omernik.com To: dev@myriad.incubator.apache.org Sent: Tuesday, August 18, 2015 1:35 PM Subject: Re: Myriad 0.1 release scope On the release scope, will having the myriad configuration file exist outside the jar (i.e. you can change configuration without rebuilding) be part of the .1 release scope? On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella smare...@maprtech.com wrote: Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
Hello All, I've merged the FGS changes into phase1. Built and tested both coarse grained scaling and fine grained scaling, UI on a 4 node cluster. If anyone finds things are not working as expected, please let me know. Thanks, Santosh On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
Not sure it's that simple. I think this requires reflection based API invocation based on hadoop version. Also the hadoop version itself needs to be passed from build file into Java code. Hadoop eco system components like Hive adopt a technique called shim loaders to solve this type of problems. -- Sent from mobile On Aug 7, 2015 3:54 PM, Darin Johnson dbjohnson1...@gmail.com wrote: So I compiled the 2.5 fgs against 2.6 when I was testing. If we abstract this right it may just be an if statement or two. On Aug 7, 2015 6:47 PM, Santosh Marella smare...@maprtech.com wrote: Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. Verified the above. As long as FGS (zero profile NM) is not used, Myriad compiled against hadoop 2.7 will work on hadoop 2.5. Thanks, Santosh On Fri, Aug 7, 2015 at 2:20 PM, Santosh Marella smare...@maprtech.com wrote: It will make working on HA easier Oh Yes! how do we facilitate that? Profiles? Profiles might be one way. Currently, FGS is supported for zero profile only. And we have seen there was an API incompatibility from 2.5 to 2.6+ in FGS code. So, ideally (since I haven't tried it myself), when FGS is merged into phase1, the Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. (I'll try this out and post back what I find) However, in the long term we need a mechanism to abstract out the APIs that are incompatible across versions. Thanks, Santosh On Fri, Aug 7, 2015 at 12:12 PM, Darin Johnson dbjohnson1...@gmail.com wrote: It will make working on HA easier. However, one concern that's been addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to support 2.5.X (anything lower?) also as Santosh has a way to do that, if so how do we facilitate that? Profiles? Darin On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
So I compiled the 2.5 fgs against 2.6 when I was testing. If we abstract this right it may just be an if statement or two. On Aug 7, 2015 6:47 PM, Santosh Marella smare...@maprtech.com wrote: Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. Verified the above. As long as FGS (zero profile NM) is not used, Myriad compiled against hadoop 2.7 will work on hadoop 2.5. Thanks, Santosh On Fri, Aug 7, 2015 at 2:20 PM, Santosh Marella smare...@maprtech.com wrote: It will make working on HA easier Oh Yes! how do we facilitate that? Profiles? Profiles might be one way. Currently, FGS is supported for zero profile only. And we have seen there was an API incompatibility from 2.5 to 2.6+ in FGS code. So, ideally (since I haven't tried it myself), when FGS is merged into phase1, the Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. (I'll try this out and post back what I find) However, in the long term we need a mechanism to abstract out the APIs that are incompatible across versions. Thanks, Santosh On Fri, Aug 7, 2015 at 12:12 PM, Darin Johnson dbjohnson1...@gmail.com wrote: It will make working on HA easier. However, one concern that's been addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to support 2.5.X (anything lower?) also as Santosh has a way to do that, if so how do we facilitate that? Profiles? Darin On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. Verified the above. As long as FGS (zero profile NM) is not used, Myriad compiled against hadoop 2.7 will work on hadoop 2.5. Thanks, Santosh On Fri, Aug 7, 2015 at 2:20 PM, Santosh Marella smare...@maprtech.com wrote: It will make working on HA easier Oh Yes! how do we facilitate that? Profiles? Profiles might be one way. Currently, FGS is supported for zero profile only. And we have seen there was an API incompatibility from 2.5 to 2.6+ in FGS code. So, ideally (since I haven't tried it myself), when FGS is merged into phase1, the Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. (I'll try this out and post back what I find) However, in the long term we need a mechanism to abstract out the APIs that are incompatible across versions. Thanks, Santosh On Fri, Aug 7, 2015 at 12:12 PM, Darin Johnson dbjohnson1...@gmail.com wrote: It will make working on HA easier. However, one concern that's been addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to support 2.5.X (anything lower?) also as Santosh has a way to do that, if so how do we facilitate that? Profiles? Darin On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
It will make working on HA easier. However, one concern that's been addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to support 2.5.X (anything lower?) also as Santosh has a way to do that, if so how do we facilitate that? Profiles? Darin On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
It will make working on HA easier Oh Yes! how do we facilitate that? Profiles? Profiles might be one way. Currently, FGS is supported for zero profile only. And we have seen there was an API incompatibility from 2.5 to 2.6+ in FGS code. So, ideally (since I haven't tried it myself), when FGS is merged into phase1, the Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5 cluster as long as FGS (i.e. zero profile NM) is not used. (I'll try this out and post back what I find) However, in the long term we need a mechanism to abstract out the APIs that are incompatible across versions. Thanks, Santosh On Fri, Aug 7, 2015 at 12:12 PM, Darin Johnson dbjohnson1...@gmail.com wrote: It will make working on HA easier. However, one concern that's been addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to support 2.5.X (anything lower?) also as Santosh has a way to do that, if so how do we facilitate that? Profiles? Darin On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com wrote: Hello guys, I propose merging FGS into phase1. As I said before, I think it's at a point where the functionality works reasonably well. Any future improvements/fixes/UI changes can be done via separate JIRAs. Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD Monday* (PDT). Thanks, Santosh On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
any UI or Rest tweaks desired for this? On Thu, Aug 6, 2015 at 3:01 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Just learned from Santosh that my email last night did not make it to the mailing list. resending... Regards Swapnil -- Forwarded message -- From: Swapnil Daingade swapnil.daing...@gmail.com Date: Thu, Aug 6, 2015 at 12:00 AM Subject: Re: Myriad 0.1 release scope To: dev@myriad.incubator.apache.org I think Myriad HA is also in good shape. We discussed work preserving NM restart internally and it looks like it is disabled by default in Apache Hadoop. https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html All 3 major distro's seem to have it disabled by default. I think we can revisit it in the next Myriad release. For the time being we can continue with the NM + Executor merge. It too has been tried out and reviewed by Darin and Santosh. The current PR we have for Myriad state store implementation should work for phase1. There is some additional work required for supporting FGS that I am working on. Should be able to complete by the time FGS is merged into phase1. Regards Swapnil Santosh Marella wrote: I feel FGS is very close to making it into 0.1. PR 116 addresses moving to hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x clusters and it seemed to have worked as expected. Unless there are more reviews/feedback, it can be merged into issue_14. Once PR 116 is merged into issue_14, issue_14 can be merged into phase1. Thanks, Santosh On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io a...@mesosphere.io wrote: We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
Re: Myriad 0.1 release scope
We do have a JIRA 0.1.0 fix version field, but none of our issues use it yet. I think the goal was just to take what we have and make it work under Apache infrastructure, then vote on that for 0.1.0. Although other features like HA or FGS would be great, let's try to get our first Apache release out ASAP. We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with other issues/features. Roadmap would be great. (I'm just summarizing what we discussed a month or two ago. Feel free to correct me or disagree with this approach.) On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade swapnil.daing...@gmail.com wrote: Hi all, Was wondering what would be the scope for the Myriad 0.1 release. It would be nice to have a roadmap page somewhere and target features to releases (JIRA 'fix version' field perhaps) Regards Swapnil
