[jira] [Commented] (MYRIAD-251) ZERO size NodeManager fail to obtain resource from Mesos Offer

2016-12-14 Thread DarinJ (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749451#comment-15749451
 ] 

DarinJ commented on MYRIAD-251:
---

[~hokiegeek2] looks like this occurs in PR-91, on line 145 of 
NMHeartBeatManager, instead on not using any of the resource in they are above 
the theshold, you should take the min of resources and the theshold.

> ZERO size NodeManager fail to obtain resource from Mesos Offer
> --
>
> Key: MYRIAD-251
> URL: https://issues.apache.org/jira/browse/MYRIAD-251
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Tao Jie
>
> I tried Fine-grained Scaling and flexed up zero size NodeManager, then I run 
> a MR job which request for resource.
> However zero size NM did not obtain resource from mesos offer. RM logs like:
> {code}
> 2016-12-14 16:58:23,929 INFO 
> org.apache.myriad.scheduler.fgs.NMHeartBeatHandler: Did not update 
> bdi13.cmss.com with 10 cores and 5888 memory, over max cpu cores and/or max 
> memory
> 2016-12-14 16:58:23,931 WARN 
> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Asked to set Node 
> bdi13.cmss.com:31905 to a value less than zero!  Had , 
> setting to .
> 2016-12-14 16:58:23,931 WARN 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>  Update resource on node: bdi13.cmss.com with the same resource:  vCores:0>
> {code}
> It seems that mesos offer with memory larger than 2252.8mb would be denied, 
> and 2252.8mb is fixed value in code :
> {code}
> private Double generateNodeManagerMemory() {
> return (NodeManagerConfiguration.DEFAULT_JVM_MAX_MEMORY_MB) * (1 + 
> NodeManagerConfiguration.JVM_OVERHEAD);
>   }
> {code}
> where DEFAULT_JVM_MAX_MEMORY_MB=2048 and 
> NodeManagerConfiguration.JVM_OVERHEAD=0.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Unable to join dev sync

2016-12-14 Thread Adam Bordelon 
Sorry, I just joined, and you should be able to now as well.

On Wed, Dec 14, 2016 at 9:08 AM, Darin Johnson 
wrote:

> Eom
>

Unable to join dev sync

2016-12-14 Thread Darin Johnson 
Eom

[jira] [Comment Edited] (MYRIAD-251) ZERO size NodeManager fail to obtain resource from Mesos Offer

2016-12-14 Thread Yuliya Feldman (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748793#comment-15748793
 ] 

Yuliya Feldman edited comment on MYRIAD-251 at 12/14/16 4:40 PM:
-

[~Tao Jie]  I suggest you debug the issue since you have it at hand.

generateNodeManagerMemory() is a memory to run NM itself, not the tasks on NM - 
at least what it should be.


was (Author: yufeldman):
[~cassanada]  I suggest you debug the issue since you have it at hand.

generateNodeManagerMemory() is a memory to run NM itself, not the tasks on NM - 
at least what it should be.

> ZERO size NodeManager fail to obtain resource from Mesos Offer
> --
>
> Key: MYRIAD-251
> URL: https://issues.apache.org/jira/browse/MYRIAD-251
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Tao Jie
>
> I tried Fine-grained Scaling and flexed up zero size NodeManager, then I run 
> a MR job which request for resource.
> However zero size NM did not obtain resource from mesos offer. RM logs like:
> {code}
> 2016-12-14 16:58:23,929 INFO 
> org.apache.myriad.scheduler.fgs.NMHeartBeatHandler: Did not update 
> bdi13.cmss.com with 10 cores and 5888 memory, over max cpu cores and/or max 
> memory
> 2016-12-14 16:58:23,931 WARN 
> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Asked to set Node 
> bdi13.cmss.com:31905 to a value less than zero!  Had , 
> setting to .
> 2016-12-14 16:58:23,931 WARN 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>  Update resource on node: bdi13.cmss.com with the same resource:  vCores:0>
> {code}
> It seems that mesos offer with memory larger than 2252.8mb would be denied, 
> and 2252.8mb is fixed value in code :
> {code}
> private Double generateNodeManagerMemory() {
> return (NodeManagerConfiguration.DEFAULT_JVM_MAX_MEMORY_MB) * (1 + 
> NodeManagerConfiguration.JVM_OVERHEAD);
>   }
> {code}
> where DEFAULT_JVM_MAX_MEMORY_MB=2048 and 
> NodeManagerConfiguration.JVM_OVERHEAD=0.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MYRIAD-251) ZERO size NodeManager fail to obtain resource from Mesos Offer

2016-12-14 Thread Yuliya Feldman (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748793#comment-15748793
 ] 

Yuliya Feldman commented on MYRIAD-251:
---

[~taojie]  I suggest you debug the issue since you have it at hand.

generateNodeManagerMemory() is a memory to run NM itself, not the tasks on NM - 
at least what it should be.

> ZERO size NodeManager fail to obtain resource from Mesos Offer
> --
>
> Key: MYRIAD-251
> URL: https://issues.apache.org/jira/browse/MYRIAD-251
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Tao Jie
>
> I tried Fine-grained Scaling and flexed up zero size NodeManager, then I run 
> a MR job which request for resource.
> However zero size NM did not obtain resource from mesos offer. RM logs like:
> {code}
> 2016-12-14 16:58:23,929 INFO 
> org.apache.myriad.scheduler.fgs.NMHeartBeatHandler: Did not update 
> bdi13.cmss.com with 10 cores and 5888 memory, over max cpu cores and/or max 
> memory
> 2016-12-14 16:58:23,931 WARN 
> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Asked to set Node 
> bdi13.cmss.com:31905 to a value less than zero!  Had , 
> setting to .
> 2016-12-14 16:58:23,931 WARN 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>  Update resource on node: bdi13.cmss.com with the same resource:  vCores:0>
> {code}
> It seems that mesos offer with memory larger than 2252.8mb would be denied, 
> and 2252.8mb is fixed value in code :
> {code}
> private Double generateNodeManagerMemory() {
> return (NodeManagerConfiguration.DEFAULT_JVM_MAX_MEMORY_MB) * (1 + 
> NodeManagerConfiguration.JVM_OVERHEAD);
>   }
> {code}
> where DEFAULT_JVM_MAX_MEMORY_MB=2048 and 
> NodeManagerConfiguration.JVM_OVERHEAD=0.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MYRIAD-251) ZERO size NodeManager fail to obtain resource from Mesos Offer

2016-12-14 Thread Tao Jie (JIRA) 
Tao Jie created MYRIAD-251:
--

 Summary: ZERO size NodeManager fail to obtain resource from Mesos 
Offer
 Key: MYRIAD-251
 URL: https://issues.apache.org/jira/browse/MYRIAD-251
 Project: Myriad
  Issue Type: Bug
  Components: Scheduler
Reporter: Tao Jie


I tried Fine-grained Scaling and flexed up zero size NodeManager, then I run a 
MR job which request for resource.
However zero size NM did not obtain resource from mesos offer. RM logs like:
{code}
2016-12-14 16:58:23,929 INFO 
org.apache.myriad.scheduler.fgs.NMHeartBeatHandler: Did not update 
bdi13.cmss.com with 10 cores and 5888 memory, over max cpu cores and/or max 
memory
2016-12-14 16:58:23,931 WARN 
org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Asked to set Node 
bdi13.cmss.com:31905 to a value less than zero!  Had , 
setting to .
2016-12-14 16:58:23,931 WARN 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: 
Update resource on node: bdi13.cmss.com with the same resource: 
{code}
It seems that mesos offer with memory larger than 2252.8mb would be denied, and 
2252.8mb is fixed value in code :
{code}
private Double generateNodeManagerMemory() {
return (NodeManagerConfiguration.DEFAULT_JVM_MAX_MEMORY_MB) * (1 + 
NodeManagerConfiguration.JVM_OVERHEAD);
  }
{code}
where DEFAULT_JVM_MAX_MEMORY_MB=2048 and 
NodeManagerConfiguration.JVM_OVERHEAD=0.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)