[jira] [Commented] (YARN-6196) [YARN-3368] Invalid information in Node pages and improve Resource Donut chart with better label

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887494#comment-15887494
 ] 

Sunil G commented on YARN-6196:
---

When there are 0 nodes in cluster ,we get below response
{code}
nodes: { }
{code}

This has to be handled in serializer to avoid page hang.

> [YARN-3368] Invalid information in Node pages and improve Resource Donut 
> chart with better label
> 
>
> Key: YARN-6196
> URL: https://issues.apache.org/jira/browse/YARN-6196
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Akhil PB
>Assignee: Akhil PB
>
> In nodes page:
> # Change 'Nodes Table' label to 'Information'
> # Change 'VCores Avail' label to 'VCores Available'
> # Show Health Report as N/A if not available
> # Change 'Mem Avail' to 'Mem Available'
> In node page:
> # Node Health Report missing
> # NodeManager Start Time shows Invalid Date
> # Reverse colors in the 'Resouce - Memory' and 'Resource - VCores' donut 
> charts
> # Convert Resource Memory into GB/TB
> # Diagnostics is empty in Container Information



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5335) Use em-table in app/nodes pages for new YARN UI

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5335:
--
Attachment: YARN-5335.0003.patch

Sorry. There was a minor link issue (extra backslash). Removing that.

> Use em-table in app/nodes pages for new YARN UI
> ---
>
> Key: YARN-5335
> URL: https://issues.apache.org/jira/browse/YARN-5335
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5335.0001.patch, YARN-5335.0002.patch, 
> YARN-5335.0003.patch
>
>
> Convert to em-table for better flexibility in nodes and app pages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6182) [YARN-3368] Fix alignment issues and missing information in Queue pages

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6182:
---
Description: 
In Queues page:
# Queue Capacities : Absolute Max Capacity should be aligned better.
# Queue Information: State is coming empty

  was:
1. Queue -> Queue Capacities : Absolute Max Capacity should be aligned better.
2. Queue -> Queue Information: State is coming empty


> [YARN-3368] Fix alignment issues and missing information in Queue pages
> ---
>
> Key: YARN-6182
> URL: https://issues.apache.org/jira/browse/YARN-6182
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6182.001.patch, YARN-6182.002.patch
>
>
> In Queues page:
> # Queue Capacities : Absolute Max Capacity should be aligned better.
> # Queue Information: State is coming empty



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6196) [YARN-3368] Invalid information in Node pages and improve Resource Donut chart with better label

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6196:
---
Description: 
In nodes page:
# Change 'Nodes Table' label to 'Information'
# Change 'VCores Avail' label to 'VCores Available'
# Show Health Report as N/A if not available
# Change 'Mem Avail' to 'Mem Available'

In node page:
# Node Health Report missing
# NodeManager Start Time shows Invalid Date
# Reverse colors in the 'Resouce - Memory' and 'Resource - VCores' donut charts
# Convert Resource Memory into GB/TB
# Diagnostics is empty in Container Information

  was:
In nodes page:
# Change 'Nodes Table' label to 'Information'
# Change 'VCores Avail' label to 'VCores Available'
# Show Health Report as N/A if not available
# Change 'Mem Used' to 'Memory Used'
# Change 'Mem Avail' to 'Memory Available'

In node page:
# Node Health Report missing
# NodeManager Start Time shows Invalid Date
# Reverse colors in the 'Resouce - Memory' and 'Resource - VCores' donut charts
# Convert Resource Memory into GB/TB
# In List of Apps or List of Containers page, Node link is broken
# Diagnostics is empty in Container Information


> [YARN-3368] Invalid information in Node pages and improve Resource Donut 
> chart with better label
> 
>
> Key: YARN-6196
> URL: https://issues.apache.org/jira/browse/YARN-6196
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Akhil PB
>Assignee: Akhil PB
>
> In nodes page:
> # Change 'Nodes Table' label to 'Information'
> # Change 'VCores Avail' label to 'VCores Available'
> # Show Health Report as N/A if not available
> # Change 'Mem Avail' to 'Mem Available'
> In node page:
> # Node Health Report missing
> # NodeManager Start Time shows Invalid Date
> # Reverse colors in the 'Resouce - Memory' and 'Resource - VCores' donut 
> charts
> # Convert Resource Memory into GB/TB
> # Diagnostics is empty in Container Information



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6153) keepContainer does not work when AM retry window is set

[kyungwan nam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kyungwan nam updated YARN-6153:
---
Attachment: YARN-6153.006.patch

Thanks for your review.
Ok. I'm uploading a new patch 006, which it has been fixed according to your 
comment.

> keepContainer does not work when AM retry window is set
> ---
>
> Key: YARN-6153
> URL: https://issues.apache.org/jira/browse/YARN-6153
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: kyungwan nam
> Attachments: YARN-6153.001.patch, YARN-6153.002.patch, 
> YARN-6153.003.patch, YARN-6153.004.patch, YARN-6153.005.patch, 
> YARN-6153.006.patch
>
>
> yarn.resourcemanager.am.max-attempts has been configured to 2 in my cluster.
> I submitted a YARN application (slider app) that keepContainers=true, 
> attemptFailuresValidityInterval=30.
> it did work properly when AM was failed firstly.
> all containers launched by previous AM were resynced with new AM (attempt2) 
> without killing containers.
> after 10 minutes, I thought AM failure count was reset by 
> attemptFailuresValidityInterval (5 minutes).
> but, all containers were killed when AM was failed secondly. (new AM attempt3 
> was launched properly)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887442#comment-15887442
 ] 

Varun Saxena edited comment on YARN-6027 at 2/28/17 7:12 AM:
-

bq. Any converter class that does both can implement both separate interfaces.
To be honest, this is what I meant when I first made the comment about 
separating this out into an interface. But after seeing Rohith's patch thought 
that as all row key converters will be requiring encoding/decoding string row 
key variant so let us go with what's in the patch. However, your point about 
this being orthogonal to current key version. And this being more about 
conversion to/from String makes sense to me. So let us go with 2 separate 
interfaces. 

bq. In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
Actually, the current patch does not use Separator enum for from id conversion. 
It uses 2 constants. My comment was more about using these constants in tests.
However, I did think of reusing Separator enum. But how do we fit in the escape 
char? Move ESCAPE char also to Separator enum? Probably can be done but then it 
will not be used at all in methods defined in Separator enum. And picking one 
from Separator enum and other from a constant I thought would be weird.
Also, I thought we will be mixing it up because for FROM_ID we would want URL 
safe characters as separator and escape characters because this will be carried 
in URL in eventually. Whereas Separator class is primarily used for conversion 
fo HBase bytes. So if somebody wishes to change separator in HBase row key, 
would he have to take care of this being URL safe character as well? We can 
probably leave a comment but then it would be somewhat restrictive. 
I had even thought of moving split and join methods used for String escaping to 
Separator enum but Separator enum is defined at the storage layer and these 
methods are also used for UID conversion.
This was the thought process behind not suggesting reuse of 
Separator#QUALIFIERS. Your thoughts on the same? 
cc [~sjlee0]



was (Author: varun_saxena):
bq. Any converter class that does both can implement both separate interfaces.
To be honest, this is what I meant when I first made the comment about 
separating this out into an interface. But after seeing Rohith's patch thought 
that as all row key converters will be requiring encoding/decoding string row 
key variant so let us go with what's in the patch. However, your point about 
this being orthogonal to current key version. And this being more about 
conversion to/from String makes sense to me. So let us go with 2 separate 
interfaces. 

bq. In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
Actually, the current patch does not use Separator enum for from id conversion. 
It uses 2 constants. My comment was more about using these constants in tests.
However, I did think of reusing Separator enum. But how do we fit in the escape 
char? Move ESCAPE char also to Separator enum? Probably can be done but then it 
will not be used at all in Separator enum. Picking one from Separator enum and 
other from a constant I thought would be weird.
Also, I thought we will be mixing it up because for FROM_ID we would want URL 
safe characters as separator and escape characters because this will be carried 
in URL in eventually. Whereas Separator class is primarily used for conversion 
fo HBase bytes. So if somebody wishes to change separator in HBase row key, 
would he have to take care of this being URL safe character as well? We can 
probably leave a comment but then it would be somewhat restrictive. 
I had even thought of moving split and join methods used for String escaping to 
Separator enum but Separator enum is defined at the storage layer and these 
methods are also used for UID conversion.
This was the thought process behind not suggesting reuse of 
Separator#QUALIFIERS. Your thoughts on the same? 

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>

[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887442#comment-15887442
 ] 

Varun Saxena edited comment on YARN-6027 at 2/28/17 7:09 AM:
-

bq. Any converter class that does both can implement both separate interfaces.
To be honest, this is what I meant when I first made the comment about 
separating this out into an interface. But after seeing Rohith's patch thought 
that as all row key converters will be requiring encoding/decoding string row 
key variant so let us go with what's in the patch. However, your point about 
this being orthogonal to current key version. And this being more about 
conversion to/from String makes sense to me. So let us go with 2 separate 
interfaces. 

bq. In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
Actually, the current patch does not use Separator enum for from id conversion. 
It uses 2 constants. My comment was more about using these constants in tests.
However, I did think of reusing Separator enum. But how do we fit in the escape 
char? Move ESCAPE char also to Separator enum? Probably can be done but then it 
will not be used at all in Separator enum. Picking one from Separator enum and 
other from a constant I thought would be weird.
Also, I thought we will be mixing it up because for FROM_ID we would want URL 
safe characters as separator and escape characters because this will be carried 
in URL in eventually. Whereas Separator class is primarily used for conversion 
fo HBase bytes. So if somebody wishes to change separator in HBase row key, 
would he have to take care of this being URL safe character as well? We can 
probably leave a comment but then it would be somewhat restrictive. 
I had even thought of moving split and join methods used for String escaping to 
Separator enum but Separator enum is defined at the storage layer and these 
methods are also used for UID conversion.
This was the thought process behind not suggesting reuse of 
Separator#QUALIFIERS. Your thoughts on the same? 


was (Author: varun_saxena):
bq. Any converter class that does both can implement both separate interfaces.
To be honest, this is what I meant when I first made the comment about 
separating this out into an interface. But after seeing Rohith's patch thought 
that as all row key converters will be requiring encoding/decoding string row 
key variant so let us go with what's in the patch. However, your point about 
this being orthogonal to current key version. And this being more about 
conversion to/from String makes sense to me. So let us go with 2 separate 
interfaces. 

bq. In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
Actually, the current patch does not use Separator enum for from id conversion. 
It uses 2 constants. My comment was more about using these constants in tests.
However, I did think of reusing Separator enum. But how do we fit in the escape 
char? Move ESCAPE char also to Separator enum? Probably can be done but then it 
will not be used at all in Separator enum. Picking one from Separator enum and 
other from a constant I thought would be weird.
Also, I thought we will be mixing it up because for FROM_ID we would want URL 
safe characters as separator and escape characters because this will be carried 
in URL in eventually. Whereas Separator class is primarily used for conversion 
fo HBase bytes. So if somebody wishes to change separator in HBase row key, 
would he have to take care of this being URL safe character as well? We can 
probably leave a comment but then it would be somewhat restrictive.
This was the thought process behind not suggesting reuse of 
Separator#QUALIFIERS. Your thoughts on the same? 

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we 

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887442#comment-15887442
 ] 

Varun Saxena commented on YARN-6027:


bq. Any converter class that does both can implement both separate interfaces.
To be honest, this is what I meant when I first made the comment about 
separating this out into an interface. But after seeing Rohith's patch thought 
that as all row key converters will be requiring encoding/decoding string row 
key variant so let us go with what's in the patch. However, your point about 
this being orthogonal to current key version. And this being more about 
conversion to/from String makes sense to me. So let us go with 2 separate 
interfaces. 

bq. In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
Actually, the current patch does not use Separator enum for from id conversion. 
It uses 2 constants. My comment was more about using these constants in tests.
However, I did think of reusing Separator enum. But how do we fit in the escape 
char? Move ESCAPE char also to Separator enum? Probably can be done but then it 
will not be used at all in Separator enum. Picking one from Separator enum and 
other from a constant I thought would be weird.
Also, I thought we will be mixing it up because for FROM_ID we would want URL 
safe characters as separator and escape characters because this will be carried 
in URL in eventually. Whereas Separator class is primarily used for conversion 
fo HBase bytes. So if somebody wishes to change separator in HBase row key, 
would he have to take care of this being URL safe character as well? We can 
probably leave a comment but then it would be somewhat restrictive.
This was the thought process behind not suggesting reuse of 
Separator#QUALIFIERS. Your thoughts on the same? 

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6231) FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887429#comment-15887429
 ] 

Hudson commented on YARN-6231:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11316 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11316/])
YARN-6231. FairSchedulerTestBase helper methods should call (kasha: rev 
f187d63816584b783fbfe238475c8f37decdb6dc)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java


> FairSchedulerTestBase helper methods should call scheduler.update to avoid 
> flakiness
> 
>
> Key: YARN-6231
> URL: https://issues.apache.org/jira/browse/YARN-6231
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Arun Suresh
>Assignee: Karthik Kambatla
> Fix For: 2.9.0
>
> Attachments: YARN-6231.001.patch, YARN-6231.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5335) Use em-table in app/nodes pages for new YARN UI

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887416#comment-15887416
 ] 

Sunil G commented on YARN-5335:
---

Thanks [~leftnoteasy]. Committing the same.

> Use em-table in app/nodes pages for new YARN UI
> ---
>
> Key: YARN-5335
> URL: https://issues.apache.org/jira/browse/YARN-5335
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5335.0001.patch, YARN-5335.0002.patch
>
>
> Convert to em-table for better flexibility in nodes and app pages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4652) Skip label configuration and calculation of queue if nodelabel disabled

[Ying Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887366#comment-15887366
 ] 

Ying Zhang commented on YARN-4652:
--

Hi [~bibinchundatt], are you still working on this issue?:-)

> Skip label configuration and calculation of queue if nodelabel disabled
> ---
>
> Key: YARN-4652
> URL: https://issues.apache.org/jira/browse/YARN-4652
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> As per the discussion in YARN-4465 the queue level configuration need to be 
> skipped when node label is disabled in RM
> {quote}
> Queues with label configured will be rejected, including 
> default-node-label-expression, accessible-node-labels (accessible-node-labels 
> is * by default, but we shouldn't allow explicitly set accessible-node-labels 
> to non-empty and non *), and label-related capacities (check 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils#loadCapacitiesByLabelsFromConf)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6231) FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness

[Karthik Kambatla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6231:
---
Summary: FairSchedulerTestBase helper methods should call scheduler.update 
to avoid flakiness  (was: 
TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on 
branch-2 )

> FairSchedulerTestBase helper methods should call scheduler.update to avoid 
> flakiness
> 
>
> Key: YARN-6231
> URL: https://issues.apache.org/jira/browse/YARN-6231
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Arun Suresh
>Assignee: Karthik Kambatla
> Attachments: YARN-6231.001.patch, YARN-6231.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient

[Bibin A Chundatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887359#comment-15887359
 ] 

Bibin A Chundatt commented on YARN-6164:


{quote}
I think lets not deleting any existing configuration or fields from QueueInfo. 
We can add this new map of QueueConfigurations
{quote}
Yes. We should move this direction so that in  future can be accommodate other 
parameters also easily.

> Expose maximum-am-resource-percent in YarnClient
> 
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch, 
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the 
> [Cluster Scheduler 
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
>  but not through 
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by 
> default), it would be nice to expose `maximum-am-resource-percent` in 
> YarnClient as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6231) TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on branch-2

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-6231:
--

+1.. The latest patch looks good.. Thanks karthik




> TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints failing on 
> branch-2 
> ---
>
> Key: YARN-6231
> URL: https://issues.apache.org/jira/browse/YARN-6231
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Arun Suresh
>Assignee: Karthik Kambatla
> Attachments: YARN-6231.001.patch, YARN-6231.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6239) Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887224#comment-15887224
 ] 

Yiqun Lin commented on YARN-6239:
-

Thanks [~liuml07] for the review! Will commit at the end of day.

> Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs
> --
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887179#comment-15887179
 ] 

Rohith Sharma K S commented on YARN-6027:
-

bq. I am also not 100% sold on RowKey. Again, it is not really about them being 
a row key type. Should we not introduce this interface (and even the converter 
interface for that matter) for now until it becomes clear we need polymorphism 
on this?
Agree, will remove this interface. One of the intention behind this to add is 
method getRowKey() is present in all the key classes. Thought it will be good 
to move as separate interface. 

As a summary on interface, below are the consensus. Please add if I have 
missed/misunderstood the consensus. 
# Lets create a separate interface KeyConverterToString. 
# Convertor classes can implement both KeyConvertor as well as 
KeyConverterToString. {code}final private static class 
FlowActivityRowKeyConverter implements
  KeyConverter, 
KeyConverterAsString {{code}
# Rowkey Interface need not required. 
# Will create methods with explicit names by its purpose. 
# Regarding using Separator.QUALIFIERS, let me check code for feasibility. 

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887160#comment-15887160
 ] 

Hadoop QA commented on YARN-1728:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-1728 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12855045/test-case-for-trunk.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f71806ba0f31 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 55c07bb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15101/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/15101/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: test-case-for-trunk.patch, YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch, YARN-1728-branch-2.005.patch
>
>
> For example, going to the job 

[jira] [Commented] (YARN-6248) Killing an app with pending container requests leaves the user in UsersManager

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887148#comment-15887148
 ] 

Sunil G commented on YARN-6248:
---

Thanks [~eepayne]. Looks like an issue. {{LeafQueue.finishApplication}} invokes 
{{UM#deactivateApplication}} ideally. I will also do test and help to debug 
same.

> Killing an app with pending container requests leaves the user in UsersManager
> --
>
> Key: YARN-6248
> URL: https://issues.apache.org/jira/browse/YARN-6248
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: User Left Over.jpg
>
>
> If an app is still asking for resources when it is killed, the user is left 
> in the UsersManager structure and shows up on the GUI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6243) CapacityScheduler: canAssignToThisQueue should be called only when necessary

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887143#comment-15887143
 ] 

Sunil G commented on YARN-6243:
---

During async sync thread allocation, is it possible that queue used may be 
changed?
If queue used is not changed, may {{canAssignToThisQueue}} need not have to be 
invoked provided there are no resource changes in cluster (add/remove node). Is 
this what you are also thinking?

> CapacityScheduler: canAssignToThisQueue should be called only when necessary
> 
>
> Key: YARN-6243
> URL: https://issues.apache.org/jira/browse/YARN-6243
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>
> Now canAssignToThisQueue is called when iterating apps in a LeafQueue, we 
> should optimize this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths

[Yuanbo Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887139#comment-15887139
 ] 

Yuanbo Liu commented on YARN-1728:
--

[~jira.shegalov] Thanks for your comments.
Upload v5 patch and test case for trunk to address your comments.

> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: test-case-for-trunk.patch, YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch, YARN-1728-branch-2.005.patch
>
>
> For example, going to the job history server page 
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
>  results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId: 
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent 
> encoding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-1728) History server doesn't understand percent encoded paths

[Yuanbo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuanbo Liu updated YARN-1728:
-
Attachment: test-case-for-trunk.patch

> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: test-case-for-trunk.patch, YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch, YARN-1728-branch-2.005.patch
>
>
> For example, going to the job history server page 
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
>  results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId: 
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent 
> encoding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-1728) History server doesn't understand percent encoded paths

[Yuanbo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuanbo Liu updated YARN-1728:
-
Attachment: YARN-1728-branch-2.005.patch

> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch, YARN-1728-branch-2.005.patch
>
>
> For example, going to the job history server page 
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
>  results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId: 
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent 
> encoding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887105#comment-15887105
 ] 

Hadoop QA commented on YARN-6247:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
41s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
32s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
44s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
22s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
17s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6247 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12855033/YARN-6247-YARN-2915.v2.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 4ee1637e357d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-2915 / 611a7fe |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15099/testReport/ |
| modules | C: 

[jira] [Commented] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887080#comment-15887080
 ] 

Hadoop QA commented on YARN-6190:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
12s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
13s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6190 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12854286/YARN-6190-YARN-2915.v2.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 4a48886201f4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-2915 / 611a7fe |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15100/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/15100/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Bug in LocalityMulticastAMRMProxyPolicy argument validation
> ---
>
> Key: YARN-6190
> URL: https://issues.apache.org/jira/browse/YARN-6190
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: federation
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6190-YARN-2915.v1.patch, 
> 

[jira] [Assigned] (YARN-6249) TestFairSchedulerPreemption is inconsistently failing on trunk

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu reassigned YARN-6249:
--

Assignee: Yufei Gu

> TestFairSchedulerPreemption is inconsistently failing on trunk
> --
>
> Key: YARN-6249
> URL: https://issues.apache.org/jira/browse/YARN-6249
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler, resourcemanager
>Affects Versions: 2.9.0
>Reporter: Sean Po
>Assignee: Yufei Gu
>
> Tests in TestFairSchedulerPreemption.java will inconsistently fail on trunk. 
> An example stack trace: 
> Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.879 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
> testPreemptionSelectNonAMContainer[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption)
>   Time elapsed: 10.475 sec  <<< FAILURE!
> java.lang.AssertionError: Incorrect number of containers on the greedy app 
> expected:<4> but was:<8>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:288)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionSelectNonAMContainer(TestFairSchedulerPreemption.java:363)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6190) Bug in LocalityMulticastAMRMProxyPolicy argument validation

[Carlo Curino (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887051#comment-15887051
 ] 

Carlo Curino commented on YARN-6190:


Thanks [~botong] for the patch. What you say is consistent with our offline 
discussion and I am ok with it. Patch LGTM, I kicked yetus again, let's make 
sure we get a +1 from there, and then we can commit to branch 2915.


> Bug in LocalityMulticastAMRMProxyPolicy argument validation
> ---
>
> Key: YARN-6190
> URL: https://issues.apache.org/jira/browse/YARN-6190
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: federation
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6190-YARN-2915.v1.patch, 
> YARN-6190-YARN-2915.v2.patch
>
>
> A bug fix in LocalityMulticastAMRMProxyPolicy on policy array condition 
> check, along with misc cleanups. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem

[Sean Po (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887042#comment-15887042
 ] 

Sean Po commented on YARN-5385:
---

Two of the test failures are accounted for with a JIRA.

YARN-6231 - 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints
YARN-5548 - 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart::testFinishedAppRemovalAfterRMRestart

The remaining failure 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption::testPreemptionSelectNonAMContainer
 appears to be a flaky test. A local run of the test was very inconsistent, 
with some fails and some successes across multiple tests. YARN-6249 was created 
to track this problem. 

> Add a PriorityAgent in ReservationSystem 
> -
>
> Key: YARN-5385
> URL: https://issues.apache.org/jira/browse/YARN-5385
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
>  Labels: oct16-hard
> Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, 
> YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, 
> YARN-5385.v007.patch, YARN-5385.v1.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations 
> in the YARN ReservationSystem. This JIRA is a sub-task to track the addition 
> of a priority agent to accomplish it. Please refer to the design doc in the 
> parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6249) TestFairSchedulerPreemption is inconsistently failing on trunk

[Sean Po (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-6249:
--
Affects Version/s: 2.9.0

> TestFairSchedulerPreemption is inconsistently failing on trunk
> --
>
> Key: YARN-6249
> URL: https://issues.apache.org/jira/browse/YARN-6249
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler, resourcemanager
>Affects Versions: 2.9.0
>Reporter: Sean Po
>
> Tests in TestFairSchedulerPreemption.java will inconsistently fail on trunk. 
> An example stack trace: 
> Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.879 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
> testPreemptionSelectNonAMContainer[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption)
>   Time elapsed: 10.475 sec  <<< FAILURE!
> java.lang.AssertionError: Incorrect number of containers on the greedy app 
> expected:<4> but was:<8>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:288)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionSelectNonAMContainer(TestFairSchedulerPreemption.java:363)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6249) TestFairSchedulerPreemption is inconsistently failing on trunk

[Sean Po (JIRA)]

Sean Po created YARN-6249:
-

 Summary: TestFairSchedulerPreemption is inconsistently failing on 
trunk
 Key: YARN-6249
 URL: https://issues.apache.org/jira/browse/YARN-6249
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler, resourcemanager
Reporter: Sean Po


Tests in TestFairSchedulerPreemption.java will inconsistently fail on trunk. An 
example stack trace: 

Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.879 sec <<< 
FAILURE! - in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
testPreemptionSelectNonAMContainer[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption)
  Time elapsed: 10.475 sec  <<< FAILURE!
java.lang.AssertionError: Incorrect number of containers on the greedy app 
expected:<4> but was:<8>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:288)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionSelectNonAMContainer(TestFairSchedulerPreemption.java:363)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-6247:
---
Attachment: YARN-6247-YARN-2915.v2.patch

> Add SubClusterResolver into FederationStateStoreFacade
> --
>
> Key: YARN-6247
> URL: https://issues.apache.org/jira/browse/YARN-6247
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6247-YARN-2915.v1.patch, 
> YARN-6247-YARN-2915.v2.patch
>
>
> Add SubClusterResolver into FederationStateStoreFacade. Since the resolver 
> might involve some overhead (read file in the background, potentially 
> periodically), it is good to put it inside FederationStateStoreFacade 
> singleton, so that only one instance will be created. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886917#comment-15886917
 ] 

Hadoop QA commented on YARN-6247:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
27s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
41s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
39s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
22s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common 
generated 1 new + 162 unchanged - 0 fixed = 163 total (was 162) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 38s{color} 
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
20s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6247 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12855006/YARN-6247-YARN-2915.v1.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d48e8432c68b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-2915 / 611a7fe |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| javadoc | 
https://builds.apache.org/job/PreCommit-YARN-Build/15098/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
 |
| 

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886893#comment-15886893
 ] 

Sangjin Lee commented on YARN-6027:
---

To me {{RowKeyConverter}} is really an orthogonal conversion from the existing 
key conversion and operates on the string representation. In that sense, I 
would slightly prefer if it is not an extension of {{KeyConverter}}. Any 
converter class that does both can implement both separate interfaces.

Also, perhaps we could find a better name for {{RowKeyConverter}}. The key 
thing is not so much that it is dealing with a row key as opposed to other 
{{KeyConverter}} types aren't. Most of {{KeyConverter}} implementations do deal 
with row keys after all. I think what separates this interface is the fact that 
it deals with the string representation (to be suitable for embedding it in 
URLs). Perhaps something like {{KeyConverterToString}} or some other name 
that's explicit to its purpose would be good.

I am also not 100% sold on {{RowKey}}. Again, it is not really about them being 
a row key type. Should we not introduce this interface (and even the converter 
interface for that matter) for now until it becomes clear we need polymorphism 
on this?

In terms of method names, how about making them very explicit about dealing 
with strings? I would be more comfortable with names such as 
{{encodeAsString()}}, {{decodeFromString()}}, {{getRowKeyAsString()}}, 
{{parseRowKeyFromString()}}, and so on.

In terms of separator characters, I am of a little different opinion from 
Varun's. IMO it would be better if we could reuse the same separator character 
(Separator.QUALIFIERS) as a constant (not a new constant with the same value). 
It is not so easy to keep track of many encoding schemes, and it would be hard 
to keep track of all encoding characters. If we can reuse the same whenever 
possible, it would help understand and maintain this code. I can be persuaded 
otherwise, but I wanted to state my current take for the record.

bq. utils classes are not expected to be subclassed. Right? They just 
encapsulate a bunch of static methods and constants.

Agree. It can be both public and final.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6153) keepContainer does not work when AM retry window is set

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886768#comment-15886768
 ] 

Jian He commented on YARN-6153:
---

Hi [~kyungwan nam],  any update ?

> keepContainer does not work when AM retry window is set
> ---
>
> Key: YARN-6153
> URL: https://issues.apache.org/jira/browse/YARN-6153
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: kyungwan nam
> Attachments: YARN-6153.001.patch, YARN-6153.002.patch, 
> YARN-6153.003.patch, YARN-6153.004.patch, YARN-6153.005.patch
>
>
> yarn.resourcemanager.am.max-attempts has been configured to 2 in my cluster.
> I submitted a YARN application (slider app) that keepContainers=true, 
> attemptFailuresValidityInterval=30.
> it did work properly when AM was failed firstly.
> all containers launched by previous AM were resynced with new AM (attempt2) 
> without killing containers.
> after 10 minutes, I thought AM failure count was reset by 
> attemptFailuresValidityInterval (5 minutes).
> but, all containers were killed when AM was failed secondly. (new AM attempt3 
> was launched properly)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886730#comment-15886730
 ] 

Yufei Gu commented on YARN-4090:


[~zsl2007], I believe Karthik has answered your question in YARN-4752.

> Make Collections.sort() more efficient in FSParentQueue.java
> 
>
> Key: YARN-4090
> URL: https://issues.apache.org/jira/browse/YARN-4090
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Reporter: Xianyin Xin
>Assignee: zhangshilong
> Attachments: sampling1.jpg, sampling2.jpg, YARN-4090.001.patch, 
> YARN-4090.002.patch, YARN-4090.003.patch, YARN-4090.004.patch, 
> YARN-4090.005.patch, YARN-4090.006.patch, YARN-4090-preview.patch, 
> YARN-4090-TestResult.pdf
>
>
> Collections.sort() consumes too much time in a scheduling round.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths

[Gera Shegalov (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886729#comment-15886729
 ] 

Gera Shegalov commented on YARN-1728:
-

Minor thing, since we have this catch clause, can we add the pathInfo value and 
stack trace to the log message:
{code}
}  catch (URISyntaxException ex) {
  LOG.error(pathInfo + ": Failed to decode path.", ex);
}
{code}

> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch
>
>
> For example, going to the job history server page 
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
>  results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId: 
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent 
> encoding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-6247:
---
Attachment: YARN-6247-YARN-2915.v1.patch

> Add SubClusterResolver into FederationStateStoreFacade
> --
>
> Key: YARN-6247
> URL: https://issues.apache.org/jira/browse/YARN-6247
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6247-YARN-2915.v1.patch
>
>
> Add SubClusterResolver into FederationStateStoreFacade. Since the resolver 
> might involve some overhead (read file in the background, potentially 
> periodically), it is good to put it inside FederationStateStoreFacade 
> singleton, so that only one instance will be created. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6248) Killing an app with pending container requests leaves the user in UsersManager

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated YARN-6248:
-
Attachment: User Left Over.jpg

> Killing an app with pending container requests leaves the user in UsersManager
> --
>
> Key: YARN-6248
> URL: https://issues.apache.org/jira/browse/YARN-6248
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: User Left Over.jpg
>
>
> If an app is still asking for resources when it is killed, the user is left 
> in the UsersManager structure and shows up on the GUI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6248) Killing an app with pending container requests leaves the user in UsersManager

[Eric Payne (JIRA)]

Eric Payne created YARN-6248:


 Summary: Killing an app with pending container requests leaves the 
user in UsersManager
 Key: YARN-6248
 URL: https://issues.apache.org/jira/browse/YARN-6248
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.0.0-alpha3
Reporter: Eric Payne
Assignee: Eric Payne


If an app is still asking for resources when it is killed, the user is left in 
the UsersManager structure and shows up on the GUI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-6247:
---
Attachment: (was: YARN-6247-YARN-2915.v1.patch)

> Add SubClusterResolver into FederationStateStoreFacade
> --
>
> Key: YARN-6247
> URL: https://issues.apache.org/jira/browse/YARN-6247
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
>
> Add SubClusterResolver into FederationStateStoreFacade. Since the resolver 
> might involve some overhead (read file in the background, potentially 
> periodically), it is good to put it inside FederationStateStoreFacade 
> singleton, so that only one instance will be created. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-6247:
---
Attachment: YARN-6247-YARN-2915.v1.patch

> Add SubClusterResolver into FederationStateStoreFacade
> --
>
> Key: YARN-6247
> URL: https://issues.apache.org/jira/browse/YARN-6247
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-6247-YARN-2915.v1.patch
>
>
> Add SubClusterResolver into FederationStateStoreFacade. Since the resolver 
> might involve some overhead (read file in the background, potentially 
> periodically), it is good to put it inside FederationStateStoreFacade 
> singleton, so that only one instance will be created. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6247) Add SubClusterResolver into FederationStateStoreFacade

[Botong Huang (JIRA)]

Botong Huang created YARN-6247:
--

 Summary: Add SubClusterResolver into FederationStateStoreFacade
 Key: YARN-6247
 URL: https://issues.apache.org/jira/browse/YARN-6247
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Botong Huang
Assignee: Botong Huang
Priority: Minor


Add SubClusterResolver into FederationStateStoreFacade. Since the resolver 
might involve some overhead (read file in the background, potentially 
periodically), it is good to put it inside FederationStateStoreFacade 
singleton, so that only one instance will be created. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6245) Add FinalResource object to reduce overhead of Resource class instancing

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886660#comment-15886660
 ] 

Wangda Tan commented on YARN-6245:
--

Thanks [~kasha],

bq. What do you think of an approach where ImmutableResource extends Resource 
but throws an UnsupportedException on modification? 
Sounds good to me.

bq. Resource could have a getImmutableCopy() method that returns a copy.
So does it mean: {{immutable_add(a, b) = new 
ImmutableResource(add(a.getImmutable(), b.getImmutable()))}}? If so, this 
approach might be inefficient since it will generate lots of immutable objects. 
And under the context of YARN-3926, I'm not sure what is the most efficient way 
to handle ImmutableResource object with extensible types.

> Add FinalResource object to reduce overhead of Resource class instancing
> 
>
> Key: YARN-6245
> URL: https://issues.apache.org/jira/browse/YARN-6245
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
> Attachments: YARN-6245.preliminary-staled.1.patch
>
>
> There're lots of Resource object creation in YARN Scheduler, since Resource 
> object is backed by protobuf, creation of such objects is expensive and 
> becomes bottleneck.
> To address the problem, we can introduce a FinalResource (Is it better to 
> call it ImmutableResource?) object, which is not backed by PBImpl. We can use 
> this object in frequent invoke paths in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6246) Move app starvation identification out of the update thread

[Karthik Kambatla (JIRA)]

Karthik Kambatla created YARN-6246:
--

 Summary: Move app starvation identification out of the update 
thread
 Key: YARN-6246
 URL: https://issues.apache.org/jira/browse/YARN-6246
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: fairscheduler
Affects Versions: 2.9.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla


Given the update thread holds the scheduler write-lock, we are probably better 
of computing starvation and identification of starved apps in a different 
thread. 

I am averse to adding a thread that runs on a *configurable timeout*, but maybe 
we could trigger this thread after every update run, or do this in the update 
thread but outside of the write-lock. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6239) Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886643#comment-15886643
 ] 

Mingliang Liu commented on YARN-6239:
-

+1

failing UT is not related.

> Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs
> --
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6245) Add FinalResource object to reduce overhead of Resource class instancing

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886640#comment-15886640
 ] 

Karthik Kambatla commented on YARN-6245:


Barely skimmed through the patch. 

What do you think of an approach where ImmutableResource extends Resource but 
throws an UnsupportedException on modification? I have not measured it, but the 
instantiation might not be as expensive given there is no protobuf. Resource 
could have a getImmutableCopy() method that returns a copy. We could play 
around with creating this on every get call or only on an update. 

> Add FinalResource object to reduce overhead of Resource class instancing
> 
>
> Key: YARN-6245
> URL: https://issues.apache.org/jira/browse/YARN-6245
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
> Attachments: YARN-6245.preliminary-staled.1.patch
>
>
> There're lots of Resource object creation in YARN Scheduler, since Resource 
> object is backed by protobuf, creation of such objects is expensive and 
> becomes bottleneck.
> To address the problem, we can introduce a FinalResource (Is it better to 
> call it ImmutableResource?) object, which is not backed by PBImpl. We can use 
> this object in frequent invoke paths in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6245) Add FinalResource object to reduce overhead of Resource class instancing

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886632#comment-15886632
 ] 

Karthik Kambatla commented on YARN-6245:


I am very much in favor of doing this; was even considering bringing it up. Due 
to the lack of an ImmutableResource, I see that we do a combination of (1) 
clone the Resource, (2) lock at every usage, or (3) to avoid the performance 
penalties, just leave the race around hoping no one would use that Resource 
inappropriately.  

> Add FinalResource object to reduce overhead of Resource class instancing
> 
>
> Key: YARN-6245
> URL: https://issues.apache.org/jira/browse/YARN-6245
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
> Attachments: YARN-6245.preliminary-staled.1.patch
>
>
> There're lots of Resource object creation in YARN Scheduler, since Resource 
> object is backed by protobuf, creation of such objects is expensive and 
> becomes bottleneck.
> To address the problem, we can introduce a FinalResource (Is it better to 
> call it ImmutableResource?) object, which is not backed by PBImpl. We can use 
> this object in frequent invoke paths in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6242) [Umbrella] Miscellaneous Scheduler Performance Improvements

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886622#comment-15886622
 ] 

Wangda Tan commented on YARN-6242:
--

[~kasha], [~templedf] I remember there're some perf issues in FS as well, 
better to move them under the same JIRA so we can handle common issues together?

> [Umbrella] Miscellaneous Scheduler Performance Improvements
> ---
>
> Key: YARN-6242
> URL: https://issues.apache.org/jira/browse/YARN-6242
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Wangda Tan
>
> There're some performance issues of scheduler. YARN-3091 is majorly targeted 
> to solve locking issues of scheduler, Let's use this JIRA to track 
> non-locking issues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6245) Add FinalResource object to reduce overhead of Resource class instancing

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6245:
-
Attachment: YARN-6245.preliminary-staled.1.patch

Attached (potentially staled) patch while working on YARN-5139 tests.

> Add FinalResource object to reduce overhead of Resource class instancing
> 
>
> Key: YARN-6245
> URL: https://issues.apache.org/jira/browse/YARN-6245
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
> Attachments: YARN-6245.preliminary-staled.1.patch
>
>
> There're lots of Resource object creation in YARN Scheduler, since Resource 
> object is backed by protobuf, creation of such objects is expensive and 
> becomes bottleneck.
> To address the problem, we can introduce a FinalResource (Is it better to 
> call it ImmutableResource?) object, which is not backed by PBImpl. We can use 
> this object in frequent invoke paths in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6245) Add FinalResource object to reduce overhead of Resource class instancing

[Wangda Tan (JIRA)]

Wangda Tan created YARN-6245:


 Summary: Add FinalResource object to reduce overhead of Resource 
class instancing
 Key: YARN-6245
 URL: https://issues.apache.org/jira/browse/YARN-6245
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


There're lots of Resource object creation in YARN Scheduler, since Resource 
object is backed by protobuf, creation of such objects is expensive and becomes 
bottleneck.

To address the problem, we can introduce a FinalResource (Is it better to call 
it ImmutableResource?) object, which is not backed by PBImpl. We can use this 
object in frequent invoke paths in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6244) Introduce AtomicResource in ResourceUsage to avoid read-write lock.

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6244:
-
Attachment: YARN-6244.preliminary.0.patch

Attached preliminary patch.

> Introduce AtomicResource in ResourceUsage to avoid read-write lock.
> ---
>
> Key: YARN-6244
> URL: https://issues.apache.org/jira/browse/YARN-6244
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler, fairscheduler, resourcemanager, 
> scheduler
>Reporter: Wangda Tan
> Attachments: YARN-6244.preliminary.0.patch
>
>
> While doing SLS tests of YARN-5139, I found when multiple threads are doing 
> scheduling at the same time, read/write lock of ResourceUsage becomes 
> bottleneck. This could be improved.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6244) Introduce AtomicResource in ResourceUsage to avoid read-write lock.

[Wangda Tan (JIRA)]

Wangda Tan created YARN-6244:


 Summary: Introduce AtomicResource in ResourceUsage to avoid 
read-write lock.
 Key: YARN-6244
 URL: https://issues.apache.org/jira/browse/YARN-6244
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


While doing SLS tests of YARN-5139, I found when multiple threads are doing 
scheduling at the same time, read/write lock of ResourceUsage becomes 
bottleneck. This could be improved.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6243) CapacityScheduler: canAssignToThisQueue should be called only when necessary

[Wangda Tan (JIRA)]

Wangda Tan created YARN-6243:


 Summary: CapacityScheduler: canAssignToThisQueue should be called 
only when necessary
 Key: YARN-6243
 URL: https://issues.apache.org/jira/browse/YARN-6243
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


Now canAssignToThisQueue is called when iterating apps in a LeafQueue, we 
should optimize this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6242) [Umbrella] Miscellaneous Scheduler Performance Improvements

[Wangda Tan (JIRA)]

Wangda Tan created YARN-6242:


 Summary: [Umbrella] Miscellaneous Scheduler Performance 
Improvements
 Key: YARN-6242
 URL: https://issues.apache.org/jira/browse/YARN-6242
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan


There're some performance issues of scheduler. YARN-3091 is majorly targeted to 
solve locking issues of scheduler, Let's use this JIRA to track non-locking 
issues.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886451#comment-15886451
 ] 

Hadoop QA commented on YARN-6232:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 11 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
45s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
34s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
6s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
52s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
6s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
24s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 8 new + 919 unchanged - 20 fixed = 927 total (was 939) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 27s{color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 58s{color} 
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m 
59s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 49s{color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m  8s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
|   | hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
| Timed out junit tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | 

[jira] [Commented] (YARN-6109) Add an ability to convert ChildQueue to ParentQueue

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886372#comment-15886372
 ] 

Wangda Tan commented on YARN-6109:
--

Thanks [~subru], make sense to me, [~xgong], could you look at this to see how 
much efforts required to make reservation system works fine with delete/convert 
queue. (including unit tests).

> Add an ability to convert ChildQueue to ParentQueue
> ---
>
> Key: YARN-6109
> URL: https://issues.apache.org/jira/browse/YARN-6109
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-6109.1.patch, YARN-6109.2.patch, 
> YARN-6109.rebase.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5151) [YARN-3368] Support kill application from new YARN UI

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886368#comment-15886368
 ] 

Wangda Tan commented on YARN-5151:
--

Thanks [~GergelyNovak], could you include screenshot of this change for easier 
review?

> [YARN-3368] Support kill application from new YARN UI
> -
>
> Key: YARN-5151
> URL: https://issues.apache.org/jira/browse/YARN-5151
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Gergely Novák
> Attachments: YARN-5151.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6216) Unify Container Resizing code paths with Container Updates making it scheduler agnostic

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886367#comment-15886367
 ] 

Wangda Tan commented on YARN-6216:
--

Thanks [~asuresh], LGTM, +1 to latest patch, will commit *tomorrow* if no more 
opposite opinions.

> Unify Container Resizing code paths with Container Updates making it 
> scheduler agnostic
> ---
>
> Key: YARN-6216
> URL: https://issues.apache.org/jira/browse/YARN-6216
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6216.001.patch, YARN-6216.002.patch, 
> YARN-6216.003.patch
>
>
> YARN-5959 introduced an {{ContainerUpdateContext}} which can be used to 
> update the ExecutionType of a container in a scheduler agnostic manner. As 
> mentioned in that JIRA, extending that to encompass Container resizing is 
> trivial.
> This JIRA proposes to remove all the CapacityScheduler specific code paths. 
> (CapacityScheduler, CSQueue, FicaSchedulerApp etc.) and modify the code to 
> use the framework introduced in YARN-5959 without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886351#comment-15886351
 ] 

Wangda Tan commented on YARN-5892:
--

Thanks [~eepayne] for the elaborations,

I have a discussion with [~jlowe]/[~curino] during community sync up and have a 
separate offline discussion with [~sunilg].

I was thinking solutions like dynamic creation of user-queue and assign quotas 
to these queues, etc. But it seems doesn't fit well for FIFO + user-quota. Also 
mentioned by [~jlowe], dynamic queue exposes implementation details to end user.

[~curino] proposed to use reservation system to solve the problem, but I'm not 
sure how, Carlo could you please add your thoughts? 

In my mind, the MULP (minimum-user-limit-percentage) is a quota applied to 
active users, which gives enough resource for app (or user which the app 
belongs to) to run. If there're more active users than (100 / MULP), scheduler 
tries to give resource to first (100 / MULP) users. If #active_users < (100 / 
MULP), available resource will be equally divided to these users.

So I preferred to keep the semantic more similar to existing one, I propose to 
introduce a weight of users instead of overriding the MULP: scheduler will 
continue assign MULP% shares to each "unit users", but different user can have 
different weight to adjust quota based on share of "unit users". Also, the 
weights of users can be used independent from MULP: because in the future we 
may want to replace concept of user limit by different ones. (Like setting 
quota for each user, give weighted fair share to users, etc.)

Also, regarding to config, there's a collision for 
{{..MULP}} when queue's name equals to {{}}. 
How about adding one more layer to specify settings of users, like:
{code}
... .user-settings..weight = x
... .user-settings..weight = y
...
{code}?

> Capacity Scheduler: Support user-specific minimum user limit percent
> 
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-5892.001.patch, YARN-5892.002.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} 
> property is per queue. A cluster admin should be able to set the minimum user 
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled 
> (YARN-4945 / YARN-2113), some users can be deemed as more important than 
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user 
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed 
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like 
> this:
> {code}
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>   
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6241) Remove -jt flag

[Allen Wittenauer (JIRA)]

Allen Wittenauer created YARN-6241:
--

 Summary: Remove -jt flag
 Key: YARN-6241
 URL: https://issues.apache.org/jira/browse/YARN-6241
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Allen Wittenauer


The -jt flag is used to send a job to a remote resourcemanager.  Given the 
flag, this is clearly left over from pre-YARN days.  With the addition of the 
time line server and other YARN services, the flag doesn't really work that 
well anymore.  It's probably better to deprecate it in 2.x and remove from 3.x 
than attempt to fix it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6216) Unify Container Resizing code paths with Container Updates making it scheduler agnostic

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886339#comment-15886339
 ] 

Arun Suresh commented on YARN-6216:
---

The remaining test failures are un-related.
[~leftnoteasy], do let me know if you think the latest patch is fine..

> Unify Container Resizing code paths with Container Updates making it 
> scheduler agnostic
> ---
>
> Key: YARN-6216
> URL: https://issues.apache.org/jira/browse/YARN-6216
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6216.001.patch, YARN-6216.002.patch, 
> YARN-6216.003.patch
>
>
> YARN-5959 introduced an {{ContainerUpdateContext}} which can be used to 
> update the ExecutionType of a container in a scheduler agnostic manner. As 
> mentioned in that JIRA, extending that to encompass Container resizing is 
> trivial.
> This JIRA proposes to remove all the CapacityScheduler specific code paths. 
> (CapacityScheduler, CSQueue, FicaSchedulerApp etc.) and modify the code to 
> use the framework introduced in YARN-5959 without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6235) YARN UI - expand the tool bars by default

[Andres Perez (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886334#comment-15886334
 ] 

Andres Perez commented on YARN-6235:


Not only that but add a visual indication that are expandable like an arrow ( 
▶︎and ▼), or a + and -

> YARN UI - expand the tool bars by default 
> --
>
> Key: YARN-6235
> URL: https://issues.apache.org/jira/browse/YARN-6235
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jian He
>
> On RM UI, We have tool bars on the left hand side which is collapsed by 
> default. Similarly, on the NodeManager UI,  only the current list is expanded 
> by default.
> Just because they are collapsed, some people not familiar  with the UI don't 
> know they are expandable.
> IMO, there are anyways much space left on the UI, why not expand all of them 
> by default to be more clear for first-sight and also more convenient  to 
> navigate. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886307#comment-15886307
 ] 

Hadoop QA commented on YARN-2962:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 14s{color} 
| {color:red} hadoop-yarn-project_hadoop-yarn generated 15 new + 39 unchanged - 
0 fixed = 54 total (was 39) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
40s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 46s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 18s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
|   | 
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-2962 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12854929/YARN-2962.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux f1075a57b897 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Updated] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types

[Varun Vasudev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-6232:

Attachment: YARN-6232-YARN-3926.002.patch

Uploaded a new version to fix the failing unit tests.

> Update resource usage and preempted resource calculations to take into 
> account all resource types
> -
>
> Key: YARN-6232
> URL: https://issues.apache.org/jira/browse/YARN-6232
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6232-YARN-3926.001.patch, 
> YARN-6232-YARN-3926.002.patch
>
>
> The chargeback calculations that take place on the RM should be updated to 
> take all resource types into account.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886195#comment-15886195
 ] 

Varun Saxena edited comment on YARN-6027 at 2/27/17 5:51 PM:
-

[~rohithsharma],
bq. both are same right? any issues will happen because of enum? I debug the 
code, the value is ! itself in cluster. May be need to add more combination of 
tests.
They are both same right now. But what if somebody changes them in future? The 
variables/enums used are different. FlowActivityRowKeyConverter#decode(byte[]) 
uses Separator#QUALIFIERS whereas FlowActivityRowKeyConverter#decode(String) 
uses the constants defined in TimelineReaderUtils. Same goes for encode methods.
If somebody in future changes either one of them i.e. changes QUALIFIERS but 
not  SEPARATOR_CHAR to something other than "!", the intention of test case 
won't be served. So it would be better to use the respective constant/enum as 
per the converter method being invoked. This was the intention behind comment.
bq. IIUC, public static final constants need not to be annotated 
Sorry, my bad. We do use FROM_ID outside the scope of tests and 
TimelineReaderUtils.
bq. RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different.
What I meant was to define RowKeyConverter interface as under. We do not expect 
RowKeyConverter to be extended for anything other than type parameter being 
subtype of RowKey. Right? As your javadoc also says this interface has to be 
implemented *for encoding and decoding row keys only*. Defining it like below 
would sort of enforce this condition if all row keys implement RowKey interface.
{code}
public interface RowKeyConverter extends KeyConverter
{code}
bq. Even I got doubt while removing final, why it should be final? utils class 
should be public
utils classes are not expected to be subclassed. Right? They just encapsulate a 
bunch of static methods and constants.


was (Author: varun_saxena):
[~rohithsharma],
bq. both are same right? any issues will happen because of enum? I debug the 
code, the value is ! itself in cluster. May be need to add more combination of 
tests.
They are both same right now. But what if somebody changes them in future? The 
variables/enums used are different. FlowActivityRowKeyConverter#decode(byte[]) 
uses Separator#QUALIFIERS whereas FlowActivityRowKeyConverter#decode(String) 
uses the constants defined in TimelineReaderUtils. Same goes for encode methods.
If somebody in future changes either one of them i.e. changes QUALIFIERS but 
not  SEPARATOR_CHAR to something other than "!", the intention of test case 
won't be served. So it would be better to use the respective constant/enum as 
per the converter method being invoked. This was the intention behind comment.
bq. IIUC, public static final constants need not to be annotated 
Sorry, my bad. We do use FROM_ID outside the scope of tests and 
TimelineReaderUtils.
bq. RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different.
What I meant was to define RowKeyConverter interface as under. We do not expect 
RowKeyConverter to be extended for anything other than type parameter being 
subtype of RowKey. Right? 
{code}
public interface RowKeyConverter extends KeyConverter
{code}
bq. Even I got doubt while removing final, why it should be final? utils class 
should be public
utils classes are not expected to be subclassed. Right? They just encapsulate a 
bunch of static methods and constants.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, 

[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886195#comment-15886195
 ] 

Varun Saxena edited comment on YARN-6027 at 2/27/17 5:42 PM:
-

[~rohithsharma],
bq. both are same right? any issues will happen because of enum? I debug the 
code, the value is ! itself in cluster. May be need to add more combination of 
tests.
They are both same right now. But what if somebody changes them in future? The 
variables/enums used are different. FlowActivityRowKeyConverter#decode(byte[]) 
uses Separator#QUALIFIERS whereas FlowActivityRowKeyConverter#decode(String) 
uses the constants defined in TimelineReaderUtils. Same goes for encode methods.
If somebody in future changes either one of them i.e. changes QUALIFIERS but 
not  SEPARATOR_CHAR to something other than "!", the intention of test case 
won't be served. So it would be better to use the respective constant/enum as 
per the converter method being invoked. This was the intention behind comment.
bq. IIUC, public static final constants need not to be annotated 
Sorry, my bad. We do use FROM_ID outside the scope of tests and 
TimelineReaderUtils.
bq. RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different.
What I meant was to define RowKeyConverter interface as under. We do not expect 
RowKeyConverter to be extended for anything other than type parameter being 
subtype of RowKey. Right? 
{code}
public interface RowKeyConverter extends KeyConverter
{code}
bq. Even I got doubt while removing final, why it should be final? utils class 
should be public
utils classes are not expected to be subclassed. Right? They just encapsulate a 
bunch of static methods and constants.


was (Author: varun_saxena):
[~rohithsharma],
bq. both are same right? any issues will happen because of enum? I debug the 
code, the value is ! itself in cluster. May be need to add more combination of 
tests.
They are both same right now. But what if somebody changes them in future? The 
variables/enums used are different. FlowActivityRowKeyConverter#decode(byte[]) 
uses Separator#QUALIFIERS whereas FlowActivityRowKeyConverter#decode(String) 
uses the constants defined in TimelineReaderUtils. Same goes for encode methods.
If somebody in future changes either one of them i.e. changes either of 
QUALIFIERS or SEPARATOR_CHAR to something other than "!", the intention of test 
case won't be served. So it would be better to use the respective constant/enum 
as per the converter method being invoked. This was the intention behind 
comment.
bq. IIUC, public static final constants need not to be annotated 
Sorry, my bad. We do use FROM_ID outside the scope of tests and 
TimelineReaderUtils.
bq. RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different.
What I mean was to define RowKeyConverter interface as under. We do not expect 
RowKeyConverter to be extended for anything other than type parameter being 
subtype of RowKey. Right? 
{code}
public interface RowKeyConverter extends KeyConverter
{code}
bq. Even I got doubt while removing final, why it should be final? utils class 
should be public
utils classes are not expected to be subclassed. Right? They just encapsulate a 
bunch of static methods and constants.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886195#comment-15886195
 ] 

Varun Saxena commented on YARN-6027:


[~rohithsharma],
bq. both are same right? any issues will happen because of enum? I debug the 
code, the value is ! itself in cluster. May be need to add more combination of 
tests.
They are both same right now. But what if somebody changes them in future? The 
variables/enums used are different. FlowActivityRowKeyConverter#decode(byte[]) 
uses Separator#QUALIFIERS whereas FlowActivityRowKeyConverter#decode(String) 
uses the constants defined in TimelineReaderUtils. Same goes for encode methods.
If somebody in future changes either one of them i.e. changes either of 
QUALIFIERS or SEPARATOR_CHAR to something other than "!", the intention of test 
case won't be served. So it would be better to use the respective constant/enum 
as per the converter method being invoked. This was the intention behind 
comment.
bq. IIUC, public static final constants need not to be annotated 
Sorry, my bad. We do use FROM_ID outside the scope of tests and 
TimelineReaderUtils.
bq. RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different.
What I mean was to define RowKeyConverter interface as under. We do not expect 
RowKeyConverter to be extended for anything other than type parameter being 
subtype of RowKey. Right? 
{code}
public interface RowKeyConverter extends KeyConverter
{code}
bq. Even I got doubt while removing final, why it should be final? utils class 
should be public
utils classes are not expected to be subclassed. Right? They just encapsulate a 
bunch of static methods and constants.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886158#comment-15886158
 ] 

Sunil G commented on YARN-6164:
---

I think lets not deleting any existing configuration or fields from QueueInfo. 
We can add this new map of {{QueueConfigurations}}, and once this is matured, 
we can deprecate others if needed.

> Expose maximum-am-resource-percent in YarnClient
> 
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch, 
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the 
> [Cluster Scheduler 
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
>  but not through 
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by 
> default), it would be nice to expose `maximum-am-resource-percent` in 
> YarnClient as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient

[Benson Qiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886150#comment-15886150
 ] 

Benson Qiu commented on YARN-6164:
--

Thanks for the comments [~sunilg].

So 
[QueueInfo|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/QueueInfo.html]
 would expose something like `Map 
getQueueConfigurations()`.

`QueueInfo` already has `getCapacity()` and `getMaximumCapacity()`, which 
currently doesn't take into account node-labels. Are there any concerns with 
breaking backwards compatibility if we delete these two methods from the 
interface?

> Expose maximum-am-resource-percent in YarnClient
> 
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch, 
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the 
> [Cluster Scheduler 
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
>  but not through 
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by 
> default), it would be nice to expose `maximum-am-resource-percent` in 
> YarnClient as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886146#comment-15886146
 ] 

Rohith Sharma K S commented on YARN-6027:
-

bq. RowKeyConverter interface's type parameter can extend RowKey. I guess it 
makes sense only in this scenario.
RowKeyConvertor interface extended from KeyConvertor interface. RowKey is 
interface which is implemented by HBase table row key class . Both 
RowKeyConvertor and RowKey interfaces are different. 

bq. In TestRowKeys, we should use new CLUSTER, flow name and user for 
conversion which uses TimelineReaderUtils#DEFAULT_DELIMITER_CHAR instead of 
Separator#QUALIFIER.
both are same right? any issues will happen because of enum? I debug the code, 
the value is ! itself in cluster. May be need to add more combination of tests.

bq. TimelineReaderUtils should be final class.
Even I got doubt while removing final, why it should be final? utils class 
should be public, but not final right?

bq. FROMID_KEY should be annotated with VisibleForTesting
IIUC, public static final constants need not to be annotated as 
VisibleForTesting since these constants can be used by anyone.

bq. I was wondering if we should throw exception mentioning Invalid fromId is 
specified from within decode(String) method?
make sense, lets create a message inside  FlowActivityEntityReader

bq. Why not throw BadRequestException from newly added code in 
FlowActivityEntityReader?
however, IllegaArgumentException is converted to BadRequestException in 
TimelineReaderWebServices. May be BadRequestException can be thrown directly. 


> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-2962:
---
Attachment: YARN-2962.007.patch

Rebasing. Let's see what Jenkins says.

> ZKRMStateStore: Limit the number of znodes under a znode
> 
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, 
> YARN-2962.01.patch, YARN-2962.04.patch, YARN-2962.05.patch, 
> YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager

[Varun Vasudev (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886123#comment-15886123
 ] 

Varun Vasudev commented on YARN-5280:
-

Ah I see. Thanks for the clarification!

> Allow YARN containers to run with Java Security Manager
> ---
>
> Key: YARN-5280
> URL: https://issues.apache.org/jira/browse/YARN-5280
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager, yarn
>Affects Versions: 2.6.4
>Reporter: Greg Phillips
>Assignee: Greg Phillips
>Priority: Minor
>  Labels: oct16-medium
> Attachments: YARN-5280.001.patch, YARN-5280.002.patch, 
> YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.005.patch, 
> YARN-5280.006.patch, YARN-5280.007.patch, YARN-5280.008.patch, 
> YARN-5280.patch, YARNContainerSandbox.pdf
>
>
> YARN applications have the ability to perform privileged actions which have 
> the potential to add instability into the cluster. The Java Security Manager 
> can be used to prevent users from running privileged actions while still 
> allowing their core data processing use cases. 
> Introduce a YARN flag which will allow a Hadoop administrator to enable the 
> Java Security Manager for user code, while still providing complete 
> permissions to core Hadoop libraries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5335) Use em-table in app/nodes pages for new YARN UI

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886118#comment-15886118
 ] 

Wangda Tan commented on YARN-5335:
--

Thanks [~sunilg]! +1 from my side.

> Use em-table in app/nodes pages for new YARN UI
> ---
>
> Key: YARN-5335
> URL: https://issues.apache.org/jira/browse/YARN-5335
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5335.0001.patch, YARN-5335.0002.patch
>
>
> Convert to em-table for better flexibility in nodes and app pages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886075#comment-15886075
 ] 

Varun Saxena edited comment on YARN-6027 at 2/27/17 4:35 PM:
-

Thanks [~rohithsharma] for the patch.
The approach looks fine to me. As we will use fromId for almost all of the row 
keys, getRowKeyAsString will be required for almost all the row key 
implementations. Hence, although when I gave the comment, I was only talking 
about an interface for row key converter, newly added RowKey interface is also 
fine with me.

Few comments.
# RowKeyConverter interface's type parameter can extend RowKey. I guess it 
makes sense only in this scenario. Right? Change javadoc accordingly if you 
agree to change this.
# Javadoc for fromId mentions flow runs. It should be flows. Infact it can be 
changed to...
# In TestRowKeys, we should use new CLUSTER, flow name and user for conversion 
which uses TimelineReaderUtils#DEFAULT_DELIMITER_CHAR instead of 
Separator#QUALIFIER. Also mix it with DEFAULT_SEPARATOR_CHAR
# Nit: FlowActivityEntityReader l.116, && should be combined with either the 
line above or below.
{code}
If specified, retrieve the next set of flows from the given fromId. The set of 
flows retrieved is inclusive of specified fromId. fromId should be taken from 
the value associated with FROM_ID info key in flow entity response which was 
sent earlier.
{code}
# TimelineReaderUtils should be final class.
# FROMID_KEY should be annotated with VisibleForTesting
# I was wondering if we should throw exception mentioning Invalid fromId is 
specified from within decode(String) method? Because although we use this for 
fromid now, it may not always be the case. Maybe just catch the exception in 
FlowActivityEntityReader and then throw BadRequestException with an an 
appropriate message there.
# Why not throw BadRequestException from newly added code in 
FlowActivityEntityReader?
# Javadoc and checkstyle issues seem fixable. Also some non-javadoc comments 
can be added in FlowActivityRowKeyConverter to explain that default delimiter 
and separator chars are encoded.


was (Author: varun_saxena):
Thanks [~rohithsharma] for the patch.
The approach looks fine to me. As we will use fromId almost all of the row 
keys, getRowKeyAsString will be required for almost all the row key 
implementations. Hence, although when I gave the comment, I was only talking 
about an interface for row key converter, newly added RowKey interface is also 
fine with me.

Few comments.
# RowKeyConverter interface's type parameter can extend RowKey. I guess it 
makes sense only in this scenario. Right? Change javadoc accordingly if you 
agree to change this.
# Javadoc for fromId mentions flow runs. It should be flows. Infact it can be 
changed to...
# In TestRowKeys, we should use new CLUSTER, flow name and user for conversion 
which uses TimelineReaderUtils#DEFAULT_DELIMITER_CHAR instead of 
Separator#QUALIFIER. Also mix it with DEFAULT_SEPARATOR_CHAR
# Nit: FlowActivityEntityReader l.116, && should be combined with either the 
line above or below.
{code}
If specified, retrieve the next set of flows from the given fromId. The set of 
flows retrieved is inclusive of specified fromId. fromId should be taken from 
the value associated with FROM_ID info key in flow entity response which was 
sent earlier.
{code}
# TimelineReaderUtils should be final class.
# FROMID_KEY should be annotated with VisibleForTesting
# I was wondering if we should throw exception mentioning Invalid fromId is 
specified from within decode(String) method? Because although we use this for 
fromid now, it may not always be the case. Maybe just catch the exception in 
FlowActivityEntityReader and then throw BadRequestException with an an 
appropriate message there.
# Why not throw BadRequestException from newly added code in 
FlowActivityEntityReader?
# Javadoc and checkstyle issues seem fixable. Also some non-javadoc comments 
can be added in FlowActivityRowKeyConverter to explain that default delimiter 
and separator chars are encoded.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following 

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886075#comment-15886075
 ] 

Varun Saxena commented on YARN-6027:


Thanks [~rohithsharma] for the patch.
The approach looks fine to me. As we will use fromId almost all of the row 
keys, getRowKeyAsString will be required for almost all the row key 
implementations. Hence, although when I gave the comment, I was only talking 
about an interface for row key converter, newly added RowKey interface is also 
fine with me.

Few comments.
# RowKeyConverter interface's type parameter can extend RowKey. I guess it 
makes sense only in this scenario. Right? Change javadoc accordingly if you 
agree to change this.
# Javadoc for fromId mentions flow runs. It should be flows. Infact it can be 
changed to...
# In TestRowKeys, we should use new CLUSTER, flow name and user for conversion 
which uses TimelineReaderUtils#DEFAULT_DELIMITER_CHAR instead of 
Separator#QUALIFIER. Also mix it with DEFAULT_SEPARATOR_CHAR
# Nit: FlowActivityEntityReader l.116, && should be combined with either the 
line above or below.
{code}
If specified, retrieve the next set of flows from the given fromId. The set of 
flows retrieved is inclusive of specified fromId. fromId should be taken from 
the value associated with FROM_ID info key in flow entity response which was 
sent earlier.
{code}
# TimelineReaderUtils should be final class.
# FROMID_KEY should be annotated with VisibleForTesting
# I was wondering if we should throw exception mentioning Invalid fromId is 
specified from within decode(String) method? Because although we use this for 
fromid now, it may not always be the case. Maybe just catch the exception in 
FlowActivityEntityReader and then throw BadRequestException with an an 
appropriate message there.
# Why not throw BadRequestException from newly added code in 
FlowActivityEntityReader?
# Javadoc and checkstyle issues seem fixable. Also some non-javadoc comments 
can be added in FlowActivityRowKeyConverter to explain that default delimiter 
and separator chars are encoded.

> Support fromid(offset) filter for /flows API
> 
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>  Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch, 
> YARN-6027-YARN-5355.0002.patch, YARN-6027-YARN-5355.0003.patch, 
> YARN-6027-YARN-5355.0004.patch, YARN-6027-YARN-5355.0005.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar 
> filter for flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates 
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5151) [YARN-3368] Support kill application from new YARN UI

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886064#comment-15886064
 ] 

Sunil G commented on YARN-5151:
---

Thanks [~GergelyNovak]

Approach looks fine. I am not very sure whether need to send KILL every 500 ms. 
Do we need to wait and check app status and then issue kill again?
These ToDo's are also good one to be done especially in case of some failures. 
I ll test some error cases and share my thoughts as well.

> [YARN-3368] Support kill application from new YARN UI
> -
>
> Key: YARN-5151
> URL: https://issues.apache.org/jira/browse/YARN-5151
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Gergely Novák
> Attachments: YARN-5151.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5280) Allow YARN containers to run with Java Security Manager

[Greg Phillips (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885969#comment-15885969
 ] 

Greg Phillips edited comment on YARN-5280 at 2/27/17 4:16 PM:
--

[~vvasudev] Thanks for reviewing the patch.  The ContainerRuntimeContext is 
used across all methods in the ContainerRuntime interface:
{code:title=ContainerRuntime.java}
  void prepareContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void launchContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void signalContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void reapContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
{code}
The goal was to conform to the existing ContainerRuntime interface, though it 
definitely could make sense to merge the various Context's in a separate ticket.


was (Author: gphillips):
[~vvasudev] Thanks for reviewing the patch.  The ContainerRuntimeContext is 
used across all methods in the ContainerRuntime interface:
{code:title=ContainerRuntime.java}
  void prepareContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void launchContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void signalContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void reapContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
{code}
The goal was to conform to the existing ContainerRuntime interface, though it 
definitely could make sense to merge the various Context's in a separate patch.

> Allow YARN containers to run with Java Security Manager
> ---
>
> Key: YARN-5280
> URL: https://issues.apache.org/jira/browse/YARN-5280
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager, yarn
>Affects Versions: 2.6.4
>Reporter: Greg Phillips
>Assignee: Greg Phillips
>Priority: Minor
>  Labels: oct16-medium
> Attachments: YARN-5280.001.patch, YARN-5280.002.patch, 
> YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.005.patch, 
> YARN-5280.006.patch, YARN-5280.007.patch, YARN-5280.008.patch, 
> YARN-5280.patch, YARNContainerSandbox.pdf
>
>
> YARN applications have the ability to perform privileged actions which have 
> the potential to add instability into the cluster. The Java Security Manager 
> can be used to prevent users from running privileged actions while still 
> allowing their core data processing use cases. 
> Introduce a YARN flag which will allow a Hadoop administrator to enable the 
> Java Security Manager for user code, while still providing complete 
> permissions to core Hadoop libraries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886003#comment-15886003
 ] 

Hudson commented on YARN-5703:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11313 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11313/])
YARN-5703. ReservationAgents are not correctly configured. Contributed 
(naganarasimha_gr: rev 5f5b031d1f20cb7f621db41979e963eaa42cf52f)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/AlignedPlannerWithGreedy.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/ReservationAgent.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/PlanningAlgorithm.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/TestGreedyReservationAgent.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/GreedyReservationAgent.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/TryManyReservationAgents.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/TestAlignedPlanner.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/AbstractReservationSystem.java


> ReservationAgents are not correctly configured
> --
>
> Key: YARN-5703
> URL: https://issues.apache.org/jira/browse/YARN-5703
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Sean Po
>Assignee: Manikandan R
> Attachments: YARN-5703.001.patch, YARN-5703.002.patch, 
> YARN-5703.003.patch, YARN-5703.004.patch, YARN-5703.005.patch, 
> YARN-5703.006.patch, YARN-5703.007.patch, YARN-5703.008.patch
>
>
> In AbstractReservationSystem, the method that instantiates a ReservationAgent 
> does not properly initialize it with the appropriate configuration because it 
> expects the ReservationAgent to implement Configurable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5496) Make Node Heatmap Chart categories clickable

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885960#comment-15885960
 ] 

Sunil G commented on YARN-5496:
---

Thanks [~GergelyNovak]. I ran some tests in my local machine.

Few Suggestions:
# I guess we need a CLEAR ALL kind of button to reset all selections. Now if we 
select one category, we cant unselect it. Rather we need to select other 
category to unselect previous one.
# I know this may not be that easy. Is it possible to move all selected cells 
to top of the grid?
# Could each cell be given as a link to individual Node page?

> Make Node Heatmap Chart categories clickable
> 
>
> Key: YARN-5496
> URL: https://issues.apache.org/jira/browse/YARN-5496
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Gergely Novák
> Attachments: YARN-5496.001.patch
>
>
> Make Node Heatmap Chart categories clickable. 
> This Heatmap chart has few categories like 10% used, 30% used etc.
> This tags should be clickable. If user clicks on 10% used tag, it should show 
> hosts with 10% usage.  This can be a useful feature for clusters having 1000s 
> of nodes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5280) Allow YARN containers to run with Java Security Manager

[Greg Phillips (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885969#comment-15885969
 ] 

Greg Phillips commented on YARN-5280:
-

[~vvasudev] Thanks for reviewing the patch.  The ContainerRuntimeContext is 
used across all methods in the ContainerRuntime interface:
{code:title=ContainerRuntime.java}
  void prepareContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void launchContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void signalContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
  void reapContainer(ContainerRuntimeContext ctx)
  throws ContainerExecutionException;
{code}
The goal was to conform to the existing ContainerRuntime interface, though it 
definitely could make sense to merge the various Context's in a separate patch.

> Allow YARN containers to run with Java Security Manager
> ---
>
> Key: YARN-5280
> URL: https://issues.apache.org/jira/browse/YARN-5280
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager, yarn
>Affects Versions: 2.6.4
>Reporter: Greg Phillips
>Assignee: Greg Phillips
>Priority: Minor
>  Labels: oct16-medium
> Attachments: YARN-5280.001.patch, YARN-5280.002.patch, 
> YARN-5280.003.patch, YARN-5280.004.patch, YARN-5280.005.patch, 
> YARN-5280.006.patch, YARN-5280.007.patch, YARN-5280.008.patch, 
> YARN-5280.patch, YARNContainerSandbox.pdf
>
>
> YARN applications have the ability to perform privileged actions which have 
> the potential to add instability into the cluster. The Java Security Manager 
> can be used to prevent users from running privileged actions while still 
> allowing their core data processing use cases. 
> Introduce a YARN flag which will allow a Hadoop administrator to enable the 
> Java Security Manager for user code, while still providing complete 
> permissions to core Hadoop libraries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885925#comment-15885925
 ] 

Naganarasimha G R commented on YARN-5703:
-

Thanks for the contributions [~maniraj...@gmail.com] have committed the patch 
to trunk and branch-2.

> ReservationAgents are not correctly configured
> --
>
> Key: YARN-5703
> URL: https://issues.apache.org/jira/browse/YARN-5703
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Sean Po
>Assignee: Manikandan R
> Attachments: YARN-5703.001.patch, YARN-5703.002.patch, 
> YARN-5703.003.patch, YARN-5703.004.patch, YARN-5703.005.patch, 
> YARN-5703.006.patch, YARN-5703.007.patch, YARN-5703.008.patch
>
>
> In AbstractReservationSystem, the method that instantiates a ReservationAgent 
> does not properly initialize it with the appropriate configuration because it 
> expects the ReservationAgent to implement Configurable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5556) CapacityScheduler: Support deleting queues without requiring a RM restart

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885892#comment-15885892
 ] 

Naganarasimha G R commented on YARN-5556:
-

Thanks [~sunilg] for raising the issue will look into it !

> CapacityScheduler: Support deleting queues without requiring a RM restart
> -
>
> Key: YARN-5556
> URL: https://issues.apache.org/jira/browse/YARN-5556
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Xuan Gong
>Assignee: Naganarasimha G R
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5556.v1.001.patch, YARN-5556.v1.002.patch, 
> YARN-5556.v1.003.patch, YARN-5556.v1.004.patch, YARN-5556.v2.005.patch, 
> YARN-5556.v2.006.patch
>
>
> Today, we could add or modify queues without restarting the RM, via a CS 
> refresh. But for deleting queue, we have to restart the ResourceManager. We 
> could support for deleting queues without requiring a RM restart



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-6240:
---

Assignee: Naganarasimha G R

> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue 
> WITHOUT running apps
> Link 
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue 
> tries to delete a queue WITHOUT running apps
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885833#comment-15885833
 ] 

Hadoop QA commented on YARN-5385:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 33s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
43s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5385 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12854881/YARN-5385.v007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux b9f1548b6e1b 3.13.0-108-generic #155-Ubuntu SMP Wed 

[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths

[Gera Shegalov (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885826#comment-15885826
 ] 

Gera Shegalov commented on YARN-1728:
-

[~yuanbo], thanks for the latest patch. I suggested URI.create because we are 
guaranteed to get a valid pathInfo from the servlet container but it's indeed 
good to be defensive since we are already dealing with a servlet bug.

I am generally +1 

In trunk the issue is fixed thanks to guice 4.0/HADOOP-12064 cc: [~ozawa]. And 
as the quote from the spec says we must not decode twice. Therefore I suggest 
we split this patch. The test-only patch should go into both trunk and branch-2 
such that we catch the issue in all releases. The actual fix should go in 
branch-2.


> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
> Attachments: YARN-1728-branch-2.001.patch, 
> YARN-1728-branch-2.002.patch, YARN-1728-branch-2.003.patch, 
> YARN-1728-branch-2.004.patch
>
>
> For example, going to the job history server page 
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
>  results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId: 
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent 
> encoding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed.

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885673#comment-15885673
 ] 

Sunil G commented on YARN-6123:
---

YARN-6240 is raised to track new test case failure, Rest all are tracked.
Committing to branch-2 shortly.

> [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated 
> when childQueues is added or removed.
> --
>
> Key: YARN-6123
> URL: https://issues.apache.org/jira/browse/YARN-6123
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6123.001.patch, YARN-6123.002.patch, 
> YARN-6123-branch-2.0002.patch
>
>
> YARN-5864 added queue ordering policy to ParentQueue, we need to make sure 
> queues of QueueOrderingPolicy will be updated when any changes made for child 
> queues.
> We need to add a test to make sure it works.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly

[Sunil G (JIRA)]

Sunil G created YARN-6240:
-

 Summary: TestCapacityScheduler.testRefreshQueuesWithQueueDelete 
fails randomly
 Key: YARN-6240
 URL: https://issues.apache.org/jira/browse/YARN-6240
 Project: Hadoop YARN
  Issue Type: Bug
  Components: test
Reporter: Sunil G


*Error Message*

Expected to NOT throw exception when refresh queue tries to delete a queue 
WITHOUT running apps

Link 
[here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]

*Stacktrace*

{code}
java.lang.AssertionError: Expected to NOT throw exception when refresh queue 
tries to delete a queue WITHOUT running apps
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5556) CapacityScheduler: Support deleting queues without requiring a RM restart

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885671#comment-15885671
 ] 

Sunil G commented on YARN-5556:
---

I got this test failure in a recent run 
[here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]

I think i ll raise a ticket for same.

> CapacityScheduler: Support deleting queues without requiring a RM restart
> -
>
> Key: YARN-5556
> URL: https://issues.apache.org/jira/browse/YARN-5556
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Xuan Gong
>Assignee: Naganarasimha G R
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5556.v1.001.patch, YARN-5556.v1.002.patch, 
> YARN-5556.v1.003.patch, YARN-5556.v1.004.patch, YARN-5556.v2.005.patch, 
> YARN-5556.v2.006.patch
>
>
> Today, we could add or modify queues without restarting the RM, via a CS 
> refresh. But for deleting queue, we have to restart the ResourceManager. We 
> could support for deleting queues without requiring a RM restart



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem

[Sean Po (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-5385:
--
Attachment: YARN-5385.v007.patch

YARN-5385.v007.patch fixes a configuration bug found while running a single 
node cluster test.

The patch also attempts to fix the find bugs error from the latest QA run. 

> Add a PriorityAgent in ReservationSystem 
> -
>
> Key: YARN-5385
> URL: https://issues.apache.org/jira/browse/YARN-5385
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
>  Labels: oct16-hard
> Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, 
> YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, 
> YARN-5385.v007.patch, YARN-5385.v1.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations 
> in the YARN ReservationSystem. This JIRA is a sub-task to track the addition 
> of a priority agent to accomplish it. Please refer to the design doc in the 
> parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6239) Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885658#comment-15885658
 ] 

Hadoop QA commented on YARN-6239:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
16s{color} | {color:green} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice
 generated 0 new + 1 unchanged - 4 fixed = 1 total (was 5) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 46s{color} 
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 20s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6239 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12854879/YARN-6239.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cf47abe62871 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4d33683 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/15094/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15094/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/15094/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix javadoc warnings in YARN that caused 

[jira] [Updated] (YARN-6239) Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated YARN-6239:

Summary: Fix javadoc warnings in YARN that caused by deprecated FileSystem 
APIs  (was: Fix javadoc warnings in YRAN that caused by deprecated FileSystem 
APIs)

> Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs
> --
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6164) Expose maximum-am-resource-percent in YarnClient

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885641#comment-15885641
 ] 

Sunil G commented on YARN-6164:
---

Thanks [~benson.qiu] for the patch

There are some more high level thoughts after i was checking various CLI and 
other PBImpl code.

# We are not sharing per-label Queue resource usage
# We are not sharing per-label Queue configurations (capacity etc).

More or less I am thinking in line of thoughts shared by [~bibinchundatt] 
earlier. 
bq.Since many informations related to configuration is already exposed by 
QueueInfo, Properties may not a become a better logical grouping unless we move 
all existing config items to new subclass. 
I still kind of feel above statement makes sense. However this is similar to a 
problem addressed in {{QueueCapacities}} etc. And in long run, this may be 
helpful.

My proposal is something like below.

In {{QueueInfo}}, lets have an api called {{setQueueConfigurations}}.  A new 
api record could be added named {{QueueConfigurations}} and it can have below 
sub items
# capacity
# abs-capacity
# max-capacity
# abs-max-capacity
# max-am-perc

If we have a map in {{QueueInfo}} per-label, then {{QueueConfigurations}} could 
be mapped against its accessable-labels.

I would like to have discussion on this approach and we can make it more 
generic. [~benson.qiu], [~bibinchundatt] and [~rohithsharma] , thoughts?

> Expose maximum-am-resource-percent in YarnClient
> 
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch, 
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the 
> [Cluster Scheduler 
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
>  but not through 
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by 
> default), it would be nice to expose `maximum-am-resource-percent` in 
> YarnClient as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6239) Fix javadoc warnings in YRAN that caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated YARN-6239:

Summary: Fix javadoc warnings in YRAN that caused by deprecated FileSystem 
APIs  (was: Fix javadoc warnings in YRAN which caused by deprecated FileSystem 
APIs)

> Fix javadoc warnings in YRAN that caused by deprecated FileSystem APIs
> --
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6239) Fix javadoc warnings in YRAN which caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated YARN-6239:

Summary: Fix javadoc warnings in YRAN which caused by deprecated FileSystem 
APIs  (was: Fix javadoc warnings in YRAN caused by deprecated FileSystem APIs)

> Fix javadoc warnings in YRAN which caused by deprecated FileSystem APIs
> ---
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6239) Fix javadoc warnings in YRAN caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated YARN-6239:

Attachment: YARN-6239.001.patch

> Fix javadoc warnings in YRAN caused by deprecated FileSystem APIs
> -
>
> Key: YARN-6239
> URL: https://issues.apache.org/jira/browse/YARN-6239
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: YARN-6239.001.patch
>
>
> There are some javadoc warnings generated after some FileSystem APIs which 
> promote inefficient call patterns being deprecated in HADOOP-13321. The 
> relevant warnings:
> {code}
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
>  [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING] 
> /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
> {code}
> This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6239) Fix javadoc warnings in YRAN caused by deprecated FileSystem APIs

[Yiqun Lin (JIRA)]

Yiqun Lin created YARN-6239:
---

 Summary: Fix javadoc warnings in YRAN caused by deprecated 
FileSystem APIs
 Key: YARN-6239
 URL: https://issues.apache.org/jira/browse/YARN-6239
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor


There are some javadoc warnings generated after some FileSystem APIs which 
promote inefficient call patterns being deprecated in HADOOP-13321. The 
relevant warnings:
{code}
[WARNING] 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[283,23]
 [deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:[294,28]
 [deprecation] isDirectory(Path) in FileSystem has been deprecated
[WARNING] 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestFileSystemApplicationHistoryStore.java:
{code}
This issue is part of HADOOP-14106.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6238) TestRMRestart testRMRestartRecoveringNodeLabelManager and testRMRestartNodeMapping fails on trunk

[Varun Saxena (JIRA)]

Varun Saxena created YARN-6238:
--

 Summary: TestRMRestart testRMRestartRecoveringNodeLabelManager and 
testRMRestartNodeMapping fails on trunk
 Key: YARN-6238
 URL: https://issues.apache.org/jira/browse/YARN-6238
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Saxena


{noformat}
testRMRestartNodeMapping(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.586 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartNodeMapping(TestRMRestart.java:2413)

testRMRestartRecoveringNodeLabelManager(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.155 sec  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartRecoveringNodeLabelManager(TestRMRestart.java:2276)
{noformat{



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6238) TestRMRestart testRMRestartRecoveringNodeLabelManager and testRMRestartNodeMapping fails on trunk

[Varun Saxena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-6238:
---
Description: 
{noformat}
testRMRestartNodeMapping(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.586 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartNodeMapping(TestRMRestart.java:2413)

testRMRestartRecoveringNodeLabelManager(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.155 sec  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartRecoveringNodeLabelManager(TestRMRestart.java:2276)
{noformat}

  was:
{noformat}
testRMRestartNodeMapping(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.586 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartNodeMapping(TestRMRestart.java:2413)

testRMRestartRecoveringNodeLabelManager(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
  Time elapsed: 0.155 sec  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartRecoveringNodeLabelManager(TestRMRestart.java:2276)
{noformat{


> TestRMRestart testRMRestartRecoveringNodeLabelManager and 
> testRMRestartNodeMapping fails on trunk
> -
>
> Key: YARN-6238
> URL: https://issues.apache.org/jira/browse/YARN-6238
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Varun Saxena
>
> {noformat}
> testRMRestartNodeMapping(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
>   Time elapsed: 0.586 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartNodeMapping(TestRMRestart.java:2413)
> testRMRestartRecoveringNodeLabelManager(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
>   Time elapsed: 0.155 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartRecoveringNodeLabelManager(TestRMRestart.java:2276)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6237) Move UID constant into TimelineReaderUtils

[Rohith Sharma K S (JIRA)]

Rohith Sharma K S created YARN-6237:
---

 Summary: Move UID constant into TimelineReaderUtils
 Key: YARN-6237
 URL: https://issues.apache.org/jira/browse/YARN-6237
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelinereader
Reporter: Rohith Sharma K S


UID constant is kept in TimelineReader Manager. This can be moved to 
TimelineReaderUtils which can keep track of all reader constants. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885509#comment-15885509
 ] 

Rohith Sharma K S commented on YARN-6207:
-

Thanks Bibin for the patch, overall patch looks good to me. Few comments on test
# Can you move these test cases to TestCapacityScheduler? Here, you can mock 
RMApp and directly trigger scheduler event. You can also control order of 
events like AppAdded->moveApplication->AppAttemptAdded and 
AppAdded->AppAttemptAdded->AppAttemptRemoved->moveApplication->AppAttemptAdded. 
Refer {{TestCapacityScheduler}}. Scheduler module remain same and can be test 
which ever the event order to be tested.
# I personally feel NOT to use countdown latch to verify event order. The test 
code is bit confusing also since latch is created multiple times and using old 
references for countDown in some cases. Latch initialization is done at setup 
with 0 and using different references for countdown in test code.
# Below test code hangs randomly. Countdown latch should be created before 
triggering attempt removal event. The test code hang because if event 
app-attempt-added is triggered before latch creating, then *eventAdd.await();* 
will never set to 0 which hangs forever. 
{code}
// remove attempt
app1Attempt.handle(new RMAppAttemptEvent(app1Attempt.getAppAttemptId(),
RMAppAttemptEventType.FAIL, "Launch Fail"));
eventAdd = new CountDownLatch(1);
eventDone = new CountDownLatch(1);
eventAdd.await();
{code}
# Add a test timeout. 


> Move application can  fail when attempt add event is delayed
> 
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6207.001.patch, YARN-6207.002.patch, 
> YARN-6207.003.patch, YARN-6207.004.patch, YARN-6207.005.patch
>
>
> *Steps to reproduce*
> 1.Submit application  and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
>   at 
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
>   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
>   at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
>   at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1429)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1339)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
>   at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
>   ... 16 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885408#comment-15885408
 ] 

Hadoop QA commented on YARN-6027:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
53s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
 {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
26s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
 in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 9 new + 
28 unchanged - 1 fixed = 37 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
12s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase
 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
40s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the 
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
32s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in 
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 18s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-6027