[jira] [Commented] (MYRIAD-249) Should set NodeManager vcores more flexibly
[
https://issues.apache.org/jira/browse/MYRIAD-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704397#comment-15704397
]
Tao Jie commented on MYRIAD-249:
https://github.com/apache/incubator-myriad/pull/97
> Should set NodeManager vcores more flexibly
> ---
>
> Key: MYRIAD-249
> URL: https://issues.apache.org/jira/browse/MYRIAD-249
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> Today we set resource of NodeManager by configuration like:
> {code}
> profiles:
> zero: # NMs launched with this profile dynamically obtain cpu/mem from
> Mesos
> cpu: 0
> mem: 0
> small:
> cpu: 2
> mem: 1024
> medium:
> cpu: 4
> mem: 4096
> large:
> cpu: 10
> mem: 12288
> {code}
> cpu/mem here is request for Mesos. We launch NodeManager and set
> {{nodemanager.resource.cpu-vcores}} and {{nodemanager.resource.memory-mb}} as
> cpu/mem once resource is allocated. However the meaning of vcores in YARN is
> not extremely the same with cpu in Mesos. In Yarn, we may set vcores to 12
> when physical cpu is 6, and it would be converted to real cpu when request
> for vcores. Also in yarn, requested vcores must be integer, as a result each
> task would take at least one vcore(actually not necessary one real cpu).
> We could have one more field configuration that multiply real cpu in mesos to
> vcores in yarn. Perhaps set vcores directly in configuration?
> I am freshman of Myriad and Mesos, please correct me if I am wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MYRIAD-248) Fail to launch Nodemanager when frameworkRole is default value "*"
[
https://issues.apache.org/jira/browse/MYRIAD-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704085#comment-15704085
]
DarinJ commented on MYRIAD-248:
---
Quickly hacked this together after recreating the problem:
https://github.com/darinj/incubator-myriad. I'd like to put some unit tests in
place that demonstrate the problem before a pr though.
> Fail to launch Nodemanager when frameworkRole is default value "*"
> --
>
> Key: MYRIAD-248
> URL: https://issues.apache.org/jira/browse/MYRIAD-248
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> I tried to start hadoop cluster with myriad-0.2.0, but got error message in
> rm log:
> {code}
> 2016-11-25 10:32:50,750 ERROR
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
> Exception thrown while trying to create a task for nm
> java.lang.IllegalArgumentException: n must be positive
> at java.util.Random.nextInt(Random.java:300)
> at
> org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
> at
> org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
> at
> org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
> at
> org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:146)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:51)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I seems that the failure is due to the default value("*") of frameworkRole in
> myriad-config-default.yml.
> I set value of frameworkRole to someone, then it worked well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MYRIAD-249) Should set NodeManager vcores more flexibly
Tao Jie created MYRIAD-249:
--
Summary: Should set NodeManager vcores more flexibly
Key: MYRIAD-249
URL: https://issues.apache.org/jira/browse/MYRIAD-249
Project: Myriad
Issue Type: Bug
Affects Versions: Myriad 0.2.0
Reporter: Tao Jie
Today we set resource of NodeManager by configuration like:
{code}
profiles:
zero: # NMs launched with this profile dynamically obtain cpu/mem from Mesos
cpu: 0
mem: 0
small:
cpu: 2
mem: 1024
medium:
cpu: 4
mem: 4096
large:
cpu: 10
mem: 12288
{code}
cpu/mem here is request for Mesos. We launch NodeManager and set
{{nodemanager.resource.cpu-vcores}} and {{nodemanager.resource.memory-mb}} as
cpu/mem once resource is allocated. However the meaning of vcores in YARN is
not extremely the same with cpu in Mesos. In Yarn, we may set vcores to 12 when
physical cpu is 6, and it would be converted to real cpu when request for
vcores. Also in yarn, requested vcores must be integer, as a result each task
would take at least one vcore(actually not necessary one real cpu).
We could have one more field configuration that multiply real cpu in mesos to
vcores in yarn. Perhaps set vcores directly in configuration?
I am freshman of Myriad and Mesos, please correct me if I am wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MYRIAD-248) Fail to launch Nodemanager when frameworkRole is default value "*"
[
https://issues.apache.org/jira/browse/MYRIAD-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702397#comment-15702397
]
DarinJ commented on MYRIAD-248:
---
Will take a look, I think this can be solved pretty easily if I can get a
little time.
> Fail to launch Nodemanager when frameworkRole is default value "*"
> --
>
> Key: MYRIAD-248
> URL: https://issues.apache.org/jira/browse/MYRIAD-248
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> I tried to start hadoop cluster with myriad-0.2.0, but got error message in
> rm log:
> {code}
> 2016-11-25 10:32:50,750 ERROR
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
> Exception thrown while trying to create a task for nm
> java.lang.IllegalArgumentException: n must be positive
> at java.util.Random.nextInt(Random.java:300)
> at
> org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
> at
> org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
> at
> org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
> at
> org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:146)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:51)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I seems that the failure is due to the default value("*") of frameworkRole in
> myriad-config-default.yml.
> I set value of frameworkRole to someone, then it worked well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MYRIAD-248) Fail to launch Nodemanager when frameworkRole is default value "*"
[
https://issues.apache.org/jira/browse/MYRIAD-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702381#comment-15702381
]
Tao Jie commented on MYRIAD-248:
[~yufeldman], I am using code on master branch.
> Fail to launch Nodemanager when frameworkRole is default value "*"
> --
>
> Key: MYRIAD-248
> URL: https://issues.apache.org/jira/browse/MYRIAD-248
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> I tried to start hadoop cluster with myriad-0.2.0, but got error message in
> rm log:
> {code}
> 2016-11-25 10:32:50,750 ERROR
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
> Exception thrown while trying to create a task for nm
> java.lang.IllegalArgumentException: n must be positive
> at java.util.Random.nextInt(Random.java:300)
> at
> org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
> at
> org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
> at
> org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
> at
> org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:146)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:51)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I seems that the failure is due to the default value("*") of frameworkRole in
> myriad-config-default.yml.
> I set value of frameworkRole to someone, then it worked well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MYRIAD-247) Fail to fetch yarnConfiguration from Resourcemanager
[
https://issues.apache.org/jira/browse/MYRIAD-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702233#comment-15702233
]
Yuliya Feldman commented on MYRIAD-247:
---
[~Tao Jie] - we use hadoop version 2.7.0 and above, but I don't think it is
relevant.
> Fail to fetch yarnConfiguration from Resourcemanager
>
>
> Key: MYRIAD-247
> URL: https://issues.apache.org/jira/browse/MYRIAD-247
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> I setuped cluster with Mesos-1.0 and Myriad-0.20. When I tried to start
> nodemanager, the mesos task tried download hadoop configuration file by
> fetching {{http://rm-addr:8088/yarnConfiguration}}, but it failed.
> It seems that yarn-configuration file is available in
> {{http://rm-addr:8088/conf}} rather than
> {{http://rm-addr:8088/yarnConfiguration}}.
> I tried modify ExecutorCommandLineGenerator.java and set uri to
> {{http://rm-addr:8088/yarnConfiguration}}. Then the nodemanager started
> successfully.
> I am not sure if it is a problem. Please correct me if I am wrong.
> My Hadoop version is 2.6.0 and I also tried Hadoop-3.0.0-alpha, and found no
> difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MYRIAD-248) Fail to launch Nodemanager when frameworkRole is default value "*"
[
https://issues.apache.org/jira/browse/MYRIAD-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702226#comment-15702226
]
Yuliya Feldman commented on MYRIAD-248:
---
Feels like an issue handling roles in the resources - I think there was a
workaround for that.
[~Tao Jie] - are you sure it is on 0.2 and not on master branch?
[~darinj] - I think it is a good time to revisit roles handling in Myriad
> Fail to launch Nodemanager when frameworkRole is default value "*"
> --
>
> Key: MYRIAD-248
> URL: https://issues.apache.org/jira/browse/MYRIAD-248
> Project: Myriad
> Issue Type: Bug
>Affects Versions: Myriad 0.2.0
>Reporter: Tao Jie
>
> I tried to start hadoop cluster with myriad-0.2.0, but got error message in
> rm log:
> {code}
> 2016-11-25 10:32:50,750 ERROR
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
> Exception thrown while trying to create a task for nm
> java.lang.IllegalArgumentException: n must be positive
> at java.util.Random.nextInt(Random.java:300)
> at
> org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
> at
> org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
> at
> org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
> at
> org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:146)
> at
> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:51)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I seems that the failure is due to the default value("*") of frameworkRole in
> myriad-config-default.yml.
> I set value of frameworkRole to someone, then it worked well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Podling Report Reminder - December 2016
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 21 December 2016, 10:30 am PDT. The report for your podling will form a part of the Incubator PMC report. The Incubator PMC requires your report to be submitted 2 weeks before the board meeting, to allow sufficient time for review and submission (Wed, December 07). Please submit your report with sufficient time to allow the Incubator PMC, and subsequently board members to review and digest. Again, the very latest you should submit your report is 2 weeks prior to the board meeting. Thanks, The Apache Incubator PMC Submitting your Report -- Your report should contain the following: * Your project name * A brief description of your project, which assumes no knowledge of the project or necessarily of its field * A list of the three most important issues to address in the move towards graduation. * Any issues that the Incubator PMC or ASF Board might wish/need to be aware of * How has the community developed since the last report * How has the project developed since the last report. This should be appended to the Incubator Wiki page at: https://wiki.apache.org/incubator/December2016 Note: This is manually populated. You may need to wait a little before this page is created from a template. Mentors --- Mentors should review reports for their project(s) and sign them off on the Incubator wiki page. Signing off reports shows that you are following the project - projects that are not signed may raise alarms for the Incubator PMC. Incubator PMC
[jira] [Created] (MYRIAD-248) Fail to launch Nodemanager when frameworkRole is default value "*"
Tao Jie created MYRIAD-248:
--
Summary: Fail to launch Nodemanager when frameworkRole is default
value "*"
Key: MYRIAD-248
URL: https://issues.apache.org/jira/browse/MYRIAD-248
Project: Myriad
Issue Type: Bug
Affects Versions: Myriad 0.2.0
Reporter: Tao Jie
I tried to start hadoop cluster with myriad-0.2.0, but got error message in rm
log:
{code}
2016-11-25 10:32:50,750 ERROR
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
Exception thrown while trying to create a task for nm
java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:300)
at
org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
at
org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
at
org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
at
org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
at
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:146)
at
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:51)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
I seems that the failure is due to the default value("*") of frameworkRole in
myriad-config-default.yml.
I set value of frameworkRole to someone, then it worked well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MYRIAD-247) Fail to fetch yarnConfiguration from Resourcemanager
Tao Jie created MYRIAD-247:
--
Summary: Fail to fetch yarnConfiguration from Resourcemanager
Key: MYRIAD-247
URL: https://issues.apache.org/jira/browse/MYRIAD-247
Project: Myriad
Issue Type: Bug
Affects Versions: Myriad 0.2.0
Reporter: Tao Jie
I setuped cluster with Mesos-1.0 and Myriad-0.20. When I tried to start
nodemanager, the mesos task tried download hadoop configuration file by
fetching {{http://rm-addr:8088/yarnConfiguration}}, but it failed.
It seems that yarn-configuration file is available in
{{http://rm-addr:8088/conf}} rather than
{{http://rm-addr:8088/yarnConfiguration}}.
I tried modify ExecutorCommandLineGenerator.java and set uri to
{{http://rm-addr:8088/yarnConfiguration}}. Then the nodemanager started
successfully.
I am not sure if it is a problem. Please correct me if I am wrong.
My Hadoop version is 2.6.0 and I also tried Hadoop-3.0.0-alpha, and found no
difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
