Re: Apache Slider and Apache Myriad
Hi Darin, You are right Slider is a kind of framework for long running applications into YARN, and it's a way of running applications on YARN without changing the application code, there is no need to develop a custom Application Master or other YARN code. Apparently you can run unmodified applications on YARN via Slider. This shift towards Hadoop code (instead of separate project, incubator slider) doesn't affect Myriad at all. However I was thinking any kind of collaboration with Slider project, and this collaboration is unclear right now, because of fuzzy continuity of Slider. -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson wrote: > Javi can you explain how this effects myriad? My understanding is that > slider is similar to marathon but for yarn. > > On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: > >> FYI >> >> Regarding the recent conversation about YARN applications development >> via Apache Slider, as a different approach for creating data services >> applications on Apache Mesos (via Myriad,) "article 4" here [1], will >> probably have to wait until the resolution of this discussion at >> Apache Slider list [2]. Basically the Slider community is discussing >> about the retirement of the project, because the Slider core is being >> included in Hadoop 3 [3]. >> >> [1] https://goo.gl/RVkFUF >> [2] https://goo.gl/CPR2TF >> [3] https://issues.apache.org/jira/browse/YARN-5079 >> >> -- >> Javi Roman >> >> Twitter: @javiromanrh >> GitHub: github.com/javiroman >> Linkedin: es.linkedin.com/in/javiroman >> Big Data Blog: dataintensive.info >>
Re: Apache Slider and Apache Myriad
Javi can you explain how this effects myriad? My understanding is that slider is similar to marathon but for yarn. On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: > FYI > > Regarding the recent conversation about YARN applications development > via Apache Slider, as a different approach for creating data services > applications on Apache Mesos (via Myriad,) "article 4" here [1], will > probably have to wait until the resolution of this discussion at > Apache Slider list [2]. Basically the Slider community is discussing > about the retirement of the project, because the Slider core is being > included in Hadoop 3 [3]. > > [1] https://goo.gl/RVkFUF > [2] https://goo.gl/CPR2TF > [3] https://issues.apache.org/jira/browse/YARN-5079 > > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info >
[jira] [Closed] (MYRIAD-99) Task names for node managers
[ https://issues.apache.org/jira/browse/MYRIAD-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javi Roman closed MYRIAD-99. Resolution: Fixed Fixed here https://issues.apache.org/jira/browse/MYRIAD-141 > Task names for node managers > > > Key: MYRIAD-99 > URL: https://issues.apache.org/jira/browse/MYRIAD-99 > Project: Myriad > Issue Type: Bug >Reporter: WalterDalton > > In anticipation of being able to run multiple yarn clusters on a mesos > cluster with yarn, should we include the framework name that launched he node > manager in the node manager task name? Right now I believe it's just > size-unique identifier, and it could be framework-size-nm-identifier. > Like if you had one instance of myriad as yarn-prod and another as yarn-dev > eh node manager task names could be yarn-prod-nm-large-%uid% and > yarn-dev-nm-small-%uid% > This could help with administration and management as well as general > organization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (MYRIAD-141) Include myriad framework name into task name
[ https://issues.apache.org/jira/browse/MYRIAD-141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javi Roman closed MYRIAD-141. - > Include myriad framework name into task name > > > Key: MYRIAD-141 > URL: https://issues.apache.org/jira/browse/MYRIAD-141 > Project: Myriad > Issue Type: Improvement > Components: Scheduler >Affects Versions: Myriad 0.1.0 >Reporter: Yuliya Feldman >Assignee: Brandon Gulla >Priority: Major > Fix For: Myriad 0.2.0, Myriad 0.1.1 > > > when people plan to run multiple myriad clusters under same Mesos it is very > hard to figure out which task belongs to which framework when getting list of > active tasks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-99) Task names for node managers
[ https://issues.apache.org/jira/browse/MYRIAD-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447366#comment-16447366 ] Javi Roman commented on MYRIAD-99: -- Yes [~alvarof] this feature was already added. The framework named is by configuration. > Task names for node managers > > > Key: MYRIAD-99 > URL: https://issues.apache.org/jira/browse/MYRIAD-99 > Project: Myriad > Issue Type: Bug >Reporter: WalterDalton > > In anticipation of being able to run multiple yarn clusters on a mesos > cluster with yarn, should we include the framework name that launched he node > manager in the node manager task name? Right now I believe it's just > size-unique identifier, and it could be framework-size-nm-identifier. > Like if you had one instance of myriad as yarn-prod and another as yarn-dev > eh node manager task names could be yarn-prod-nm-large-%uid% and > yarn-dev-nm-small-%uid% > This could help with administration and management as well as general > organization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (MYRIAD-54) Allow admin to modify Myriad Configuration
[ https://issues.apache.org/jira/browse/MYRIAD-54?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javi Roman closed MYRIAD-54. Resolution: Fixed > Allow admin to modify Myriad Configuration > -- > > Key: MYRIAD-54 > URL: https://issues.apache.org/jira/browse/MYRIAD-54 > Project: Myriad > Issue Type: Bug >Reporter: Santosh Marella > > Currently myriad-config-default.yml is embedded into Framework jar > (myriad-0.0.1.jar) forcing users to specify all the config options (including > mesos master address/zk address) during build time. We perhaps need something > like following: > - A default config file that is embedded into the jar. > - A admin modifiable config file that admin can place on Myriad's classpath. > - Ability in the code to honor admin overrides of the defaults. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (MYRIAD-87) Distribute config yml outside of framework/hadoop binary
[ https://issues.apache.org/jira/browse/MYRIAD-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javi Roman closed MYRIAD-87. Resolution: Fixed > Distribute config yml outside of framework/hadoop binary > > > Key: MYRIAD-87 > URL: https://issues.apache.org/jira/browse/MYRIAD-87 > Project: Myriad > Issue Type: Bug >Reporter: Adam B > > From user JohnO, "The myriad-config-default.yml needs to be moved outside the > bundled jar so we can update our clusters without rebuilding." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-87) Distribute config yml outside of framework/hadoop binary
[ https://issues.apache.org/jira/browse/MYRIAD-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447357#comment-16447357 ] Javi Roman commented on MYRIAD-87: -- Totally agree [~alvarof] *myriad-default-config.yml* already can be shipped out of JAR files. This issue can be closed. > Distribute config yml outside of framework/hadoop binary > > > Key: MYRIAD-87 > URL: https://issues.apache.org/jira/browse/MYRIAD-87 > Project: Myriad > Issue Type: Bug >Reporter: Adam B > > From user JohnO, "The myriad-config-default.yml needs to be moved outside the > bundled jar so we can update our clusters without rebuilding." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-99) Task names for node managers
[ https://issues.apache.org/jira/browse/MYRIAD-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447330#comment-16447330 ] Alvaro Fernandez commented on MYRIAD-99: Hi! This seems fixed to me in [MYRIAD-141|https://issues.apache.org/jira/browse/MYRIAD-141], maybe we can close this ticket? Regards. > Task names for node managers > > > Key: MYRIAD-99 > URL: https://issues.apache.org/jira/browse/MYRIAD-99 > Project: Myriad > Issue Type: Bug >Reporter: WalterDalton > > In anticipation of being able to run multiple yarn clusters on a mesos > cluster with yarn, should we include the framework name that launched he node > manager in the node manager task name? Right now I believe it's just > size-unique identifier, and it could be framework-size-nm-identifier. > Like if you had one instance of myriad as yarn-prod and another as yarn-dev > eh node manager task names could be yarn-prod-nm-large-%uid% and > yarn-dev-nm-small-%uid% > This could help with administration and management as well as general > organization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-86) Consider an API for desired # instances, rather than flex up/down
[ https://issues.apache.org/jira/browse/MYRIAD-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447326#comment-16447326 ] Alvaro Fernandez commented on MYRIAD-86: Hi! I may be missing something, but it looks to me that the cluster API fex up/down has a parameter to pass the number of instances. Perhaps this was added to the cluster API and the Jira wasn't closed, or is it referring to something else? Regards > Consider an API for desired # instances, rather than flex up/down > - > > Key: MYRIAD-86 > URL: https://issues.apache.org/jira/browse/MYRIAD-86 > Project: Myriad > Issue Type: Improvement >Reporter: Adam B > > There is value in having both APIs, so I'm not suggesting we remove flex > up/down, just add a new call. > This could also be valuable for AutoScaling (MYRIAD-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MYRIAD-87) Distribute config yml outside of framework/hadoop binary
[ https://issues.apache.org/jira/browse/MYRIAD-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447322#comment-16447322 ] Alvaro Fernandez commented on MYRIAD-87: Hi! Doing some housekeeping in JIRA, and as the previous comment points out, just wondering if this issue can be closed as a duplicate of MYRIAD-54? (and it is already possible to modify *$YARN_HOME/etc/hadoop/myriad-default-config.yml* and restart with a different config) Regards. > Distribute config yml outside of framework/hadoop binary > > > Key: MYRIAD-87 > URL: https://issues.apache.org/jira/browse/MYRIAD-87 > Project: Myriad > Issue Type: Bug >Reporter: Adam B > > From user JohnO, "The myriad-config-default.yml needs to be moved outside the > bundled jar so we can update our clusters without rebuilding." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
