[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407212#comment-15407212
]
Sunil G commented on YARN-5333:
---
Hi [~jianhe] and [~rohithsharma] [~hex108]
bq.I think the ActiveStandbyElector will handle reJoin automatically if
exception thrown from the transtionToActive method.We can use the fail fast
config for this scenario.
I have another view here. If {{refreshAll}} fails due to a corrupted
capacity-scheduler.xml file when a transitionToActive is happening (IOException
will come from *refreshQeues*) as mentioned in YARN-3893, I think we need to
fail-fast. Yes, it will be harsh. But we can avoid a scenario like both RMs in
standby (and both RMs will switchover continuously) if I am not wrong. pls
correct me if I understood this case wrong. This will happen if config is given
not-to fail fast. pls share your thoughts.
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch, YARN-5333.09.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407197#comment-15407197 ] Sunil G commented on YARN-5334: --- Hi [~leftnoteasy], For points 2~4, its same as point 1. For pages like cluster-overview or yarn-apps, we have to unload the model and query/find it again to refresh. The items which I have mentioned in the comments were queried without unloading. So we may not get fresh data in. Tests may seem fine. [~Sreenath], could you also please confirm the same. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5333:
---
Attachment: YARN-5333.09.patch
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch, YARN-5333.09.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407189#comment-15407189
]
Jun Gong commented on YARN-5333:
Attach a new patch 09.patch.
Rename {{refreshXXXWithoutCheck}} to {{refreshXXX}}.
Add {{checkAcls("refreshAll")}} at the beginning of {{refreshAll()}}, then we
could check user's ACL.
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407057#comment-15407057 ] Jian He commented on YARN-5079: --- Hi [~asuresh], Short answer is, agent in Slider will be thrown away once merged in YARN. NodeManager will behave like the agent for upgrade, reconfigure etc. e.g. YARN-4726 will do most of the work. Of course, startContainer call in slider AM will be changed to start the real container, instead of the slider agent. For SLIDER-787, probably the client-facing code will be preserved, the internal logic will be changed to take advantage of YARN-4726. > [Umbrella] Native YARN framework layer for services and beyond > -- > > Key: YARN-5079 > URL: https://issues.apache.org/jira/browse/YARN-5079 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > (See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.1 to track the specific sub-item.) > (This is a companion to YARN-4793 in our effort to simplify the entire story, > but focusing on APIs) > So far, YARN by design has restricted itself to having a very low-level API > that can support any type of application. Frameworks like Apache Hadoop > MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix > and others ended up exposing higher level APIs that end-users can directly > leverage to build their applications on top of YARN. On the services side, > Apache Slider has done something similar. > With our current attention on making services first-class and simplified, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. Beyond the functionality that I outlined in the > previous sections in the doc on how NodeManagers can be enhanced to help > services, the biggest missing piece is the framework itself. There is a lot > of very important functionality that a services' framework can own together > with YARN in executing services end-to-end. > In this JIRA I propose we look at having a native Apache Hadoop framework for > running services natively on YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5449) nodemanager process is hung, and lost from resourcemanager
[ https://issues.apache.org/jira/browse/YARN-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401469#comment-15401469 ] mai shurong edited comment on YARN-5449 at 8/4/16 2:57 AM: --- Sorry, I had added my description of this issue when I created it, but was not submitted to jira by some problems. I would add description as soon as possible. was (Author: shurong.mai): Sorry, I had added my description of this issue when I created it, bu was not submitted to jira by some problems. I would add description as soon as possible. > nodemanager process is hung, and lost from resourcemanager > -- > > Key: YARN-5449 > URL: https://issues.apache.org/jira/browse/YARN-5449 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 > Environment: The os version is 2.6.32-573.8.1.el6.x86_64 GNU/Linux > The java version is jdk1.7.0_45 > The hadoop version is hadoop-2.2.0 >Reporter: mai shurong > > The nodemanager process is hung(is not dead), and lost from resourcemanager. > The nodemanager's log is stopped from printing. > The used cpu of nodemanager process is very low(nearly 0%). > GC of nodemanager jvm process is stopped, and the result of jstat(jstat > -gccause pid 1000 100) is as follows: > S0 S1 E O P YGC YGCTFGCFGCT GCT > LGCC GCC > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > The nodemanager jvm process is also accur this problem using CMS garbage > collector or g1 garbage collector. > The parameters of CMS garbage collector are as following: > -Xmx4096m -Xmn1024m -XX:PermSize=128m -XX:MaxPermSize=128m > -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:ConcGCThreads=4 > -XX:+UseCMSCom pactAtFullCollection -XX:CMSFullGCsBeforeCompaction=8 > -XX:ParallelGCThreads=4 -XX:CMSInitiatingOccupancyFraction=70 > The parameters of g1 garbage collector are as following: > -Xmx8g -Xms8g -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseG1GC > -XX:MaxGCPauseMillis=1000 -XX:G1ReservePercent=30 > -XX:InitiatingHeapOccupancyPercent=45 -XX:ConcGCThreads=4 > -XX:+PrintAdaptiveSizePolicy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407041#comment-15407041
]
Jun Gong commented on YARN-5333:
Thanks [~rohithsharma] for the review.
bq. refreshXXXWithoutCheck does not looks meaning full method name. I think
common general pattern can be followed like below.
refreshXXXWithouCheck means that there is no check for refreshXXX. If
refreshXXX is acceptable, I'd like to change it.
bq. One of my major concern after seeing patch is skipping checkACL which used
to verify user for every transition-to-active. But now it is skipped.
I ignored it... It seems that we need add checkACL. How about adding it in
{{refreshAll}}?
{code}
refreshAll () {
checkACL("XXX");
refreshXXX();
...
}
{code}
bq. Test failure is related to patch change. I think this test can be removed
only since behavior is changed after this patch.
Yes, it is related, I fixed it in the patch 07.patch. The test case seems
useful for testing the case that {{refreshAll}} failed. Maybe we could keep it?
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406995#comment-15406995
]
Hadoop QA commented on YARN-5333:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 28s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821971/YARN-5333.08.patch |
| JIRA Issue | YARN-5333 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bb0a82d2a536 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a1f6564 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12642/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12642/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Some
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406980#comment-15406980
]
Rohith Sharma K S commented on YARN-5333:
-
The approach seems looks good. Few things to consider
# {{refreshXXXWithoutCheck}} does not looks meaning full method name. I think
common general pattern can be followed like below. Thoughts?
{code}
public RefreshXXXResponse refreshXXX(RefreshXXXRequest request){
// ACL and RM check can be combined together
try{
refresh(); // This should includes only loading configuration file and
update required field.
// Success audit log
}catch{
// failure audit log
}
}
private void refresh(){
// load configuration filie
// refresh XXX fields.
}
{code}
# One of my major concern after seeing patch is skipping checkACL which used to
verify user for every transition-to-active. But now it is skipped. Does it
fine? cc:/[~jianhe]
# Test failure is related to patch change. I think this test can be removed
only since behavior is changed after this patch.
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5468) Scheduling of long-running applications
[ https://issues.apache.org/jira/browse/YARN-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated YARN-5468: - Attachment: YARN-5468.prototype.patch > Scheduling of long-running applications > --- > > Key: YARN-5468 > URL: https://issues.apache.org/jira/browse/YARN-5468 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, fairscheduler >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-5468.prototype.patch > > > This JIRA is about the scheduling of applications with long-running tasks. > It will include adding support to the YARN for a richer set of scheduling > constraints (such as affinity, anti-affinity, cardinality and time > constraints), and extending the schedulers to take them into account during > placement of containers to nodes. > We plan to have both an online version that will accommodate such requests as > they arrive, as well as a Long-running Application Planner that will make > more global decisions by considering multiple applications at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5468) Scheduling of long-running applications
[ https://issues.apache.org/jira/browse/YARN-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated YARN-5468: - Attachment: (was: YARN-5468.prototype.patch) > Scheduling of long-running applications > --- > > Key: YARN-5468 > URL: https://issues.apache.org/jira/browse/YARN-5468 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, fairscheduler >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-5468.prototype.patch > > > This JIRA is about the scheduling of applications with long-running tasks. > It will include adding support to the YARN for a richer set of scheduling > constraints (such as affinity, anti-affinity, cardinality and time > constraints), and extending the schedulers to take them into account during > placement of containers to nodes. > We plan to have both an online version that will accommodate such requests as > they arrive, as well as a Long-running Application Planner that will make > more global decisions by considering multiple applications at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5333:
---
Attachment: YARN-5333.08.patch
Fix test case error...
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406938#comment-15406938 ] Li Lu commented on YARN-4061: - Let me revive this thread after the branch merge. From some offline discussion I think our plan is to implement a specialized BufferedMutator so that it can retry when the cluster is down. The benefit of this approach is we do not need to repost the data to buffered mutator, so that saves much memory operations when the cluster is down. We can pretty much reuse some retry logic in our codebase today. The challenge for this design is that we're not persisting anything until the data reaches the HBase cluster. That is to say, with this change we can handle the case when the HBase cluster is down, but cannot handle if collectors themselves are down. If the collector fails when it's retrying, we lose the data. To address this problem, we may use a local journal file to store the state in the buffered mutator. Aggregation status is something we need to recover if collectors fail. However, at the very first phase maybe we can restart everything in the aggregation table on restarts? I know this thread is an old one, but please feel free to chime in since we're targeting to add this feature to the Alpha 2 phase of timeline v2. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Description: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" on Windows platforms. was: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" on Windows platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Affects Version/s: 2.6.0 > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5450) Enhance logging for Cluster.java around InetSocketAddress
[ https://issues.apache.org/jira/browse/YARN-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406891#comment-15406891 ] Li Lu commented on YARN-5450: - Patch LGTM. [~sarun] could you please verify if this would be fine with your suggested use case? Thanks! > Enhance logging for Cluster.java around InetSocketAddress > - > > Key: YARN-5450 > URL: https://issues.apache.org/jira/browse/YARN-5450 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: sarun singla >Assignee: Vrushali C >Priority: Minor > Labels: YARN > Attachments: YARN-5450.01.patch > > > We need to add more logging for cluster.java class around " > initialize(InetSocketAddress jobTrackAddr, Configuration conf) " method to > give better logging like about the source of the property. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Description: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" was: The following check is incorrect: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5472) WIN_MAX_PATH logic is off by one
Brook Zhou created YARN-5472: Summary: WIN_MAX_PATH logic is off by one Key: YARN-5472 URL: https://issues.apache.org/jira/browse/YARN-5472 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Environment: Windows Reporter: Brook Zhou Assignee: Brook Zhou Priority: Minor The following check is incorrect: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: YARN-4091.8.patch rebased to trunk > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.8.patch, YARN-4091.preliminary.1.patch, app_activities v2.json, > app_activities.json, node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406877#comment-15406877
]
Hadoop QA commented on YARN-5384:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 0s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 13s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 6s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 49s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat |
| | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821940/YARN-5384.v1.patch |
| JIRA Issue | YARN-5384 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux bb5449a142d6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a1f6564 |
| Default Java | 1.8.0_101 |
| findbugs |
[jira] [Commented] (YARN-4091) Add REST API to retrieve scheduler activity
[
https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406870#comment-15406870
]
Hadoop QA commented on YARN-4091:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} YARN-4091 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821942/YARN-4091.7.patch |
| JIRA Issue | YARN-4091 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12640/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add REST API to retrieve scheduler activity
> ---
>
> Key: YARN-4091
> URL: https://issues.apache.org/jira/browse/YARN-4091
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Sunil G
>Assignee: Chen Ge
> Attachments: Improvement on debugdiagnostic information - YARN.pdf,
> SchedulerActivityManager-TestReport v2.pdf,
> SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf,
> YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch,
> YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch,
> YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json,
> node_activities v2.json, node_activities.json
>
>
> As schedulers are improved with various new capabilities, more configurations
> which tunes the schedulers starts to take actions such as limit assigning
> containers to an application, or introduce delay to allocate container etc.
> There are no clear information passed down from scheduler to outerworld under
> these various scenarios. This makes debugging very tougher.
> This ticket is an effort to introduce more defined states on various parts in
> scheduler where it skips/rejects container assignment, activate application
> etc. Such information will help user to know whats happening in scheduler.
> Attaching a short proposal for initial discussion. We would like to improve
> on this as we discuss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[
https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406862#comment-15406862
]
Hadoop QA commented on YARN-5307:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821953/YARN-5307-YARN-2915-v5.patch
|
| JIRA Issue | YARN-5307 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux ba80aa7ac143 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 22db8fd |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12639/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12639/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Federation Application State Store internal APIs
>
>
> Key: YARN-5307
> URL: https://issues.apache.org/jira/browse/YARN-5307
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[
https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406853#comment-15406853
]
Hadoop QA commented on YARN-1547:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 19s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged -
0 fixed = 36 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13
new + 208 unchanged - 0 fixed = 221 total (was 208) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s {color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 11s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804094/YARN-1547.prototype.v0.patch
|
| JIRA Issue | YARN-1547 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 02b02dd84b00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a1f6564 |
| Default Java |
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[
https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406843#comment-15406843
]
Hadoop QA commented on YARN-5382:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in branch-2.7 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 31 new + 683 unchanged - 7 fixed = 714 total (was 690)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4582 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 45s
{color} | {color:red} The patch 104 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed
with JDK v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 39s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_101. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 38s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| |
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406827#comment-15406827 ] Subru Krishnan commented on YARN-5307: -- Thanks [~vvasudev] for reviewing the patch. I'll take care of it as soon as [~leftnoteasy] signs off. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch, > YARN-5307-YARN-2915-v4.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[ https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406823#comment-15406823 ] Subru Krishnan commented on YARN-5406: -- +1 on the latest patch. Thanks [~ellenfkh] for addressing our comments, I'll commit it shortly. > In-memory based implementation of the FederationMembershipStateStore > > > Key: YARN-5406 > URL: https://issues.apache.org/jira/browse/YARN-5406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5406-YARN-2915.v0.patch, > YARN-5406-YARN-2915.v1.patch > > > YARN-3662 defines the FederationMembershipStateStore API. This JIRA tracks an > in-memory based implementation which is useful for both single-box testing > and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5390) Federation Subcluster Resolver
[
https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406820#comment-15406820
]
Subru Krishnan edited comment on YARN-5390 at 8/3/16 11:31 PM:
---
Thanks [~leftnoteasy] for reviewing the patch.
Thanks [~ellenfkh] for the patch. It mostly LGTM, just a couple of minor
comments:
* Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are
going to be using it a lot more in subsequent JIRAs.
* Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test
resources.
* I feel the checkstyle warnings are not blockers but will good to fix if
it's easy to do so.
was (Author: subru):
Thanks [~leftnoteasy] for reviewing the patch.
It mostly LGTM, just a couple of minor comments:
* Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are
going to be using it a lot more in subsequent JIRAs.
* Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test
resources.
* I feel the checkstyle warnings are not blockers but will good to fix if
it's easy to do so.
> Federation Subcluster Resolver
> --
>
> Key: YARN-5390
> URL: https://issues.apache.org/jira/browse/YARN-5390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Carlo Curino
>Assignee: Ellen Hui
> Attachments: YARN-5390-YARN-2915.v0.patch,
> YARN-5390-YARN-2915.v1.patch, YARN-5390-YARN-2915.v2.patch
>
>
> This JIRA tracks effort to create a mechanism to resolve nodes/racks resource
> names to sub-cluster identifiers. This is needed by the federation policies
> in YARN-5323, YARN-5324, YARN-5325 to operate correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.2.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5470.1.patch, YARN-5470.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5390) Federation Subcluster Resolver
[
https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406820#comment-15406820
]
Subru Krishnan commented on YARN-5390:
--
Thanks [~leftnoteasy] for reviewing the patch.
It mostly LGTM, just a couple of minor comments:
* Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are
going to be using it a lot more in subsequent JIRAs.
* Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test
resources.
* I feel the checkstyle warnings are not blockers but will good to fix if
it's easy to do so.
> Federation Subcluster Resolver
> --
>
> Key: YARN-5390
> URL: https://issues.apache.org/jira/browse/YARN-5390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Carlo Curino
>Assignee: Ellen Hui
> Attachments: YARN-5390-YARN-2915.v0.patch,
> YARN-5390-YARN-2915.v1.patch, YARN-5390-YARN-2915.v2.patch
>
>
> This JIRA tracks effort to create a mechanism to resolve nodes/racks resource
> names to sub-cluster identifiers. This is needed by the federation policies
> in YARN-5323, YARN-5324, YARN-5325 to operate correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5450) Enhance logging for Cluster.java around InetSocketAddress
[ https://issues.apache.org/jira/browse/YARN-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406783#comment-15406783 ] Vrushali C commented on YARN-5450: -- Hi [~sarun] Did you get a chance to look at the patch? thanks Vrushali > Enhance logging for Cluster.java around InetSocketAddress > - > > Key: YARN-5450 > URL: https://issues.apache.org/jira/browse/YARN-5450 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: sarun singla >Assignee: Vrushali C >Priority: Minor > Labels: YARN > Attachments: YARN-5450.01.patch > > > We need to add more logging for cluster.java class around " > initialize(InetSocketAddress jobTrackAddr, Configuration conf) " method to > give better logging like about the source of the property. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406780#comment-15406780 ] Subru Krishnan commented on YARN-3664: -- Thanks [~leftnoteasy] for your feedback. I agree that using enums wherever possible is better than String. But in the case of types, they are extendible/pluggable and so we see the number growing. For e.g.: We already have 4-5 as part of YARN-5324/YARN-5325 and ideas for a couple more based on how these work out in testing. > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406772#comment-15406772 ] Subru Krishnan commented on YARN-3664: -- Thanks [~vvasudev] for reviewing the patch. Yes, I intentionally renamed _addReservationResourcesToProto_ to _addSubClusterInfosToProto_ as it was originally a copy-paste typo. I'll take care of it as soon as [~leftnoteasy] signs off. > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5471) Adding null pointer checks in ResourceRequest#newInstance
Giovanni Matteo Fumarola created YARN-5471: -- Summary: Adding null pointer checks in ResourceRequest#newInstance Key: YARN-5471 URL: https://issues.apache.org/jira/browse/YARN-5471 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Giovanni Matteo Fumarola Assignee: Giovanni Matteo Fumarola ResourceRequest#newInstance has Priority, Resource and Strings fields. The application master can set these value to null. The proposal is to add null pointer checks in the class. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406742#comment-15406742 ] Wangda Tan commented on YARN-4888: -- [~subru], I would still suggest to merge the two tests even if ParameterizedSchedulerTestBase doesn't support parameterized tests. We're trying to merge common parts of schedulers, it's better to avoid creating more duplications. For example, putting allocation-id related tests to separate test classes (like Test"CS/FS"AllocationId), extends from same test class. Beyond that, patch looks good to me. > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch, > YARN-4888-v5.patch, YARN-4888-v6.patch, YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: YARN-4091.7.patch > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json, > node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406711#comment-15406711 ] Wangda Tan edited comment on YARN-3664 at 8/3/16 10:14 PM: --- [~subru], One minor comment: Can we change getType of SubClusterPolicyConfiguration from String to enum. Using enum should be much better for readability if the type comes from limited choices. And this could also avoid potential bugs. was (Author: leftnoteasy): [~subru], One minor comment: Can we change getType of > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v1.patch YARN-5384.v1.patch adds priority to the ReservationSystem submission API by adding another field to the ReservationDefinition class. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406711#comment-15406711 ] Wangda Tan commented on YARN-3664: -- [~subru], One minor comment: Can we change getType of > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[
https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406706#comment-15406706
]
Hadoop QA commented on YARN-5470:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 10 new + 78 unchanged - 5 fixed = 88 total (was 83) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 21s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 4s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821935/YARN-5470.1.patch |
| JIRA Issue | YARN-5470 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9310e8390fe3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a1f6564 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12634/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12634/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.1.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5470.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
Xuan Gong created YARN-5470: --- Summary: Differentiate exactly match with regex in yarn log CLI Key: YARN-5470 URL: https://issues.apache.org/jira/browse/YARN-5470 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5382: - Attachment: YARN-5382-branch-2.7.12.patch YARN-5382.12.patch Uploading v12 that addresses javadoc warnings. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, YARN-5382.12.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405299#comment-15405299 ] Sangeetha Abdu Jyothi edited comment on YARN-5327 at 8/3/16 9:16 PM: - Please note that the failed test is unrelated to this patch. was (Author: ajsangeetha): Please note that the failed test in unrelated to this patch. > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406600#comment-15406600 ] Vrushali C commented on YARN-5382: -- Will quickly fix the javadoc warnings and one of the findbugs issue. I am not sure I can fix the other findbugs (the one about 9 params instead of 7, there are already 8 params in that function). > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382.06.patch, YARN-5382.07.patch, > YARN-5382.08.patch, YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406555#comment-15406555
]
Hudson commented on YARN-5469:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10207 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10207/])
YARN-5469. Increase timeout of TestAmFilter.testFilter. Contributed by (jlowe:
rev db4a61dc617954cf448a0d44cb4ac79f2abf9db3)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
> Fix For: 2.8.0, 2.7.4
>
> Attachments: YARN-5469.001.patch
>
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at
[jira] [Resolved] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved YARN-5469.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.7.4
2.8.0
Thanks, Eric! I committed this to trunk, branch-2, branch-2.8, and branch-2.7.
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
> Fix For: 2.8.0, 2.7.4
>
> Attachments: YARN-5469.001.patch
>
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at
[jira] [Updated] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5382: - Attachment: YARN-5382.11.patch YARN-5382-branch-2.7.11.patch Thanks [~jianhe], appreciate the feedback! Uploading v11 with suggested changes. Also, the "greater then" wording in the patch on trunk came in because the existing 2.7 codebase has those. But I have fixed that for both trunk and 2.7 in the v11 patch. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382.06.patch, YARN-5382.07.patch, > YARN-5382.08.patch, YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406512#comment-15406512
]
Jason Lowe commented on YARN-5469:
--
+1 lgtm. One second test timeouts are clearly too low. A JVM or I/O hiccup
while loading classes can blow through that as shown by the stack traceback.
Committing this.
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
> Attachments: YARN-5469.001.patch
>
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
[jira] [Commented] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[
https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406502#comment-15406502
]
Hadoop QA commented on YARN-5406:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
51s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821899/YARN-5406-YARN-2915.v1.patch
|
| JIRA Issue | YARN-5406 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7dccb0e8b201 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 22db8fd |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12631/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12631/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> In-memory based implementation of the FederationMembershipStateStore
>
>
> Key: YARN-5406
> URL: https://issues.apache.org/jira/browse/YARN-5406
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5406-YARN-2915.v0.patch,
>
[jira] [Updated] (YARN-4717) TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup
[ https://issues.apache.org/jira/browse/YARN-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-4717: - Fix Version/s: (was: 2.9.0) 2.7.4 2.8.0 Thanks, [~templedf]! I committed this to branch-2.8 and branch-2.7 as well. > TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails > Intermittently due to IllegalArgumentException from cleanup > --- > > Key: YARN-4717 > URL: https://issues.apache.org/jira/browse/YARN-4717 > Project: Hadoop YARN > Issue Type: Test > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 2.8.0, 2.7.4 > > Attachments: YARN-4717.001.patch > > > The same issue that was resolved by [~zxu] in YARN-3602 is back. Looks like > the commons-io package throws an IAE instead of an IOE now if the directory > doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5462) TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406476#comment-15406476
]
Hudson commented on YARN-5462:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10205 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10205/])
YARN-5462. TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown
(jlowe: rev db646540f094077941b56ed681a4f3e5853f5b7f)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
> TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails
> intermittently
> --
>
> Key: YARN-5462
> URL: https://issues.apache.org/jira/browse/YARN-5462
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Fix For: 2.8.0, 2.6.5, 2.7.4
>
> Attachments: YARN-5462.001.patch
>
>
> {noformat}
> java.io.IOException: Failed on local exception: java.io.IOException:
> Connection reset by peer; Host Details : local host is:
> "slave-02.adcd.infra.corp.gq1.yahoo.com/69.147.96.229"; destination host is:
> "127.0.0.1":12345;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1390)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy78.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:101)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.startContainer(TestNodeManagerShutdown.java:248)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown(TestNodeStatusUpdater.java:1492)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:508)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1730)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1078)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406477#comment-15406477 ] Hudson commented on YARN-4280: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10205 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10205/]) YARN-4280. CapacityScheduler reservations may not prevent indefinite (jlowe: rev 4d92aefd35d4517d9435d81bafdec0d77905a7a0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0 > > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406467#comment-15406467 ] Kuhu Shukla commented on YARN-4280: --- Thank you all for the detailed reviews. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0 > > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[ https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5406: Attachment: YARN-5406-YARN-2915.v1.patch Addressed feedback from [~subru] and [~jianhe]. > In-memory based implementation of the FederationMembershipStateStore > > > Key: YARN-5406 > URL: https://issues.apache.org/jira/browse/YARN-5406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5406-YARN-2915.v0.patch, > YARN-5406-YARN-2915.v1.patch > > > YARN-3662 defines the FederationMembershipStateStore API. This JIRA tracks an > in-memory based implementation which is useful for both single-box testing > and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5462) TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406441#comment-15406441
]
Jason Lowe commented on YARN-5462:
--
+1 lgtm. Committing this.
> TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails
> intermittently
> --
>
> Key: YARN-5462
> URL: https://issues.apache.org/jira/browse/YARN-5462
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-5462.001.patch
>
>
> {noformat}
> java.io.IOException: Failed on local exception: java.io.IOException:
> Connection reset by peer; Host Details : local host is:
> "slave-02.adcd.infra.corp.gq1.yahoo.com/69.147.96.229"; destination host is:
> "127.0.0.1":12345;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1390)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy78.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:101)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.startContainer(TestNodeManagerShutdown.java:248)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown(TestNodeStatusUpdater.java:1492)
> Caused by: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:508)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1730)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1078)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4232) TopCLI console support for HA mode
[
https://issues.apache.org/jira/browse/YARN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406440#comment-15406440
]
Hadoop QA commented on YARN-4232:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 5 new + 152 unchanged - 0 fixed = 157 total (was 152) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 26s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821889/YARN-4232.003.patch |
| JIRA Issue | YARN-4232 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b71d2945590e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 580a833 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12630/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12630/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406424#comment-15406424 ] Naganarasimha G R commented on YARN-5287: - I verified the latest patch and seems to rectify all the comments, so hope i did not miss any nits in the patch, [~vvasudev] & [~rohithsharma] anything else needs to be addressed? > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4624) NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI
[
https://issues.apache.org/jira/browse/YARN-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406418#comment-15406418
]
Naganarasimha G R commented on YARN-4624:
-
Thanks for the patch [~sunilg],
Applied patch compiled and tested in an installation, seems like everything is
working on top of trunk code.
If no one has any concerns will commit this patch tomorrow !
> NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI
> ---
>
> Key: YARN-4624
> URL: https://issues.apache.org/jira/browse/YARN-4624
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: SchedulerUIWithOutLabelMapping.png, YARN-2674-002.patch,
> YARN-4624-003.patch, YARN-4624.4.patch, YARN-4624.patch
>
>
> Scenario:
> ===
> Configure nodelables and add to cluster
> Start the cluster
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.PartitionQueueCapacitiesInfo.getMaxAMLimitPercentage(PartitionQueueCapacitiesInfo.java:114)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithPartition(CapacitySchedulerPage.java:105)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.render(CapacitySchedulerPage.java:94)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueueBlock.render(CapacitySchedulerPage.java:293)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:447)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406413#comment-15406413 ] Wangda Tan commented on YARN-5334: -- Thanks so much for working on this! [~Sreenath], this is extremely helpful for end user experiences. I did some tests for this, so far so good. [~sunilg], I'm not quite sure about what 2.-4. means in your comments, I didn't see issues here. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406411#comment-15406411 ] Jason Lowe commented on YARN-4280: -- +1 for the branch-2.8 patch as well. Committing this. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4232) TopCLI console support for HA mode
[
https://issues.apache.org/jira/browse/YARN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4232:
---
Attachment: YARN-4232.003.patch
> TopCLI console support for HA mode
> --
>
> Key: YARN-4232
> URL: https://issues.apache.org/jira/browse/YARN-4232
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4232.patch, 0002-YARN-4232.patch,
> YARN-4232.003.patch
>
>
> *Steps to reproduce*
> Start Top command in YARN in HA mode
> ./yarn top
> {noformat}
> usage: yarn top
> -cols Number of columns on the terminal
> -delay The refresh delay(in seconds), default is 3 seconds
> -help Print usage; for help while the tool is running press 'h'
> + Enter
> -queuesComma separated list of queues to restrict applications
> -rows Number of rows on the terminal
> -types Comma separated list of types to restrict applications,
> case sensitive(though the display is lower case)
> -users Comma separated list of users to restrict applications
> {noformat}
> Execute *for help while the tool is running press 'h' + Enter* while top
> tool is running
> Exception is thrown in console continuously
> {noformat}
> 15/10/07 14:59:28 ERROR cli.TopCLI: Could not fetch RM start time
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
> at
> org.apache.hadoop.yarn.client.cli.TopCLI.getRMStartTime(TopCLI.java:742)
> at org.apache.hadoop.yarn.client.cli.TopCLI.run(TopCLI.java:467)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.yarn.client.cli.TopCLI.main(TopCLI.java:420)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406362#comment-15406362 ] Jason Lowe commented on YARN-5451: -- Note that we can get localizers to stop today (e.g.: when the corresponding container being localized is killed), but only if the localizer is well-behaved. So even with large sized resources as long as the localizer heartbeat thread is still heartbeating to the NM and can respond properly to heartbeat commands things end up working out OK. It becomes a problem when the localizer _doesn't_ behave properly and needs external intervention. I don't think that's common in practice, but it does happen sometimes. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5429) Fix @return related javadoc warnings in yarn-api
[
https://issues.apache.org/jira/browse/YARN-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406273#comment-15406273
]
Hadoop QA commented on YARN-5429:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api:
The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 0 new + 125 unchanged - 31 fixed = 125 total (was 156) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 7s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821866/YARN-5429.04.patch |
| JIRA Issue | YARN-5429 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 03b27480ce69 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 58db263 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12629/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12629/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix @return related javadoc warnings in yarn-api
>
>
> Key: YARN-5429
> URL: https://issues.apache.org/jira/browse/YARN-5429
> Project: Hadoop YARN
> Issue Type: Sub-task
>
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406249#comment-15406249 ] Shane Kumpf commented on YARN-5428: --- They most certainly can. The user would set the docker client config directory at job submission time to point to where their personal config.json file lives. This is simply setting a client config directory. How you and your users decide to use it/enforce it in your large environment is up to you. This does not force you into storing your credentials in the config. The client config is also used for other client config as well, such as HTTP proxy settings, which could be quite useful to define cluster wide. There is no default setting enforced by YARN. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5429) Fix @return related javadoc warnings in yarn-api
[
https://issues.apache.org/jira/browse/YARN-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-5429:
-
Attachment: YARN-5429.04.patch
Thanks [~varun_saxena], uploading v4 that addresses the javadoc issue related
to this patch.
{code}
[machine-channapattan hadoop (trunk)]$ diff ../YARN-5429.04.patch
../YARN-5429.03.patch
22c22
< index 0b65e5c..e24ebdf 100644
---
> index 0b65e5c..462f6a2 100644
29c29
< + * @return the list of {@link ContainerResourceChangeRequest}
---
> + * @return the list of {@link ContainerResourceChangeRequest}>
[machine-13-channapattan hadoop (trunk)]$
{code}
> Fix @return related javadoc warnings in yarn-api
>
>
> Key: YARN-5429
> URL: https://issues.apache.org/jira/browse/YARN-5429
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Vrushali C
> Attachments: YARN-5429.01.patch, YARN-5429.02.patch,
> YARN-5429.03.patch, YARN-5429.04.patch
>
>
> As part of YARN-4977, filing a subtask to fix a subset of the javadoc
> warnings in yarn-api.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406226#comment-15406226 ] Allen Wittenauer commented on YARN-5428: ... and none of that answers the key question: Why is the admin providing the passwords and not the user? Why can't a user provide a config.json as part of their job submission? All users having the exact same access to every docker registry seems like a huge fail in any large (read: enterprise) environment. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5453) FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource
[
https://issues.apache.org/jira/browse/YARN-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406221#comment-15406221
]
Hadoop QA commented on YARN-5453:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 7s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821858/YARN-5453.01.patch |
| JIRA Issue | YARN-5453 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e57a2da6689e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2d82276 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12628/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-5455) LinuxContainerExecutor needs Javadocs
[
https://issues.apache.org/jira/browse/YARN-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406194#comment-15406194
]
Varun Vasudev commented on YARN-5455:
-
Thanks for the patch [~templedf]!
{code}
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_IMAGE} names which image
+ * will be used to launch the Docker container.
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_IMAGE_FILE} is currently ignored.
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE} controls
+ * whether the Docker container's default command is overridden. When set
+ * to {@code true}, the Docker container's command will be
+ * {@code bash }. When unset or set to {@code false}
+ * the Docker container's default command is used.
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK} sets the
+ * network type to be used by the Docker container. It must be a valid
+ * value as determined by the
+ * {@code yarn.nodemanager.runtime.linux.docker.allowed-container-networks}
+ * property.
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER}
+ * controls whether the Docker container is a privileged container. In
order
+ * to use privileged containers, the
+ * {@code
yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed}
+ * property must be set to {@code true}, and the application owner must
+ * appear in the value of the
+ * {@code yarn.nodemanager.runtime.linux.docker.privileged-containers.acl}
+ * property. If this environment variable is set to {@code true}, a
+ * privileged Docker container will be used if allowed. No other value is
+ * allowed, so the environment variable should be left unset rather than
+ * setting it to false.
+ *
+ *
+ * {@code YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS} adds
+ * additional volume mounts to the Docker container. The value of the
+ * environment variable should be a comma-separated list of mounts.
+ * All such mounts must be given as {@code source:dest}, where the
+ * source is an absolute path that is not a symlink and that points to a
+ * localized resource.
+ *
+*
{code}
Maybe we should move this section into the DockerLinuxContainerRuntime.java and
replace the section I highlighted with links to the actual runtime
classes(DefaultLinuxContainerRuntime, DelegatingLinuxContainerRuntime,
DockerLinuxContainerRuntime)?
The rest of it looks good to me.
> LinuxContainerExecutor needs Javadocs
> -
>
> Key: YARN-5455
> URL: https://issues.apache.org/jira/browse/YARN-5455
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5455.001.patch, YARN-5455.002.patch
>
>
> 'Nuff said.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406159#comment-15406159 ] Varun Vasudev commented on YARN-5451: - bq. The localizer should be tracked like containers are tracked so we can control them like we can control containers. +1. This really becomes a problem for containers with large sized resources(like docker). We have to be able to kill a localizer. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406144#comment-15406144
]
Hadoop QA commented on YARN-5333:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 9s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821849/YARN-5333.07.patch |
| JIRA Issue | YARN-5333 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 71a03eb4b485 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2d82276 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12627/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12627/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12627/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12627/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Some recovered apps are put into
[jira] [Comment Edited] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406094#comment-15406094
]
Varun Vasudev edited comment on YARN-5430 at 8/3/16 4:04 PM:
-
Thanks for the patch [~jianhe]!
1)
{code}
+ @Private
+ @Unstable
+ public abstract void setIp(String ip);
+
{code}
We can’t return just one ip. In the non-docker scenario, multiple ips for a
host are common and Docker itself has support for multiple ips for a container.
We don’t currently support that but it’ll probably be required for us to
support soon. Similarly we should change getIpAndHost and split them up into
getIps and getHost calls.
2)
{code}
+ optional string ip = 7;
+ optional string host = 8;
{code}
Instead of named variables - can we make this a bit more generic and add
support for a simple list of string->list? Then the next time we need to send a
new variable, we don’t need to mess around with the protobuf pieces. Something
like
{code}
message StringListMapProto {
required string key = 1;
optional repeated string value = 2;
}
{code}
and in ContainerStatusProto
{code}
optional repeated StringListMapProto attributes_map = 7;
{code}
3)
{code}
+ public String[] getIpAndHost(Container container) {
+return null;
+ }
{code}
The default implementation shouldn’t return null. We should return the host
name and ip.
4)
{code}
+ public void setIpAndHost(String[] ipAndHost) {
+this.ip = ipAndHost[0];
+this.host = ipAndHost[1];
+ }
{code}
This may become a problem if the size of the array is more than 2(multiple ips)
5)
{code}
+ @Override
+ public String[] getIpAndHost(Container container) {
+String[] ipAndHost = new String[2];
+try {
+ InetAddress address = InetAddress.getLocalHost();
+ ipAndHost[0] = address.getHostAddress();
+ ipAndHost[1] = address.getHostName();
+} catch (UnknownHostException e) {
+ LOG.error("Unable to get Local hostname and ip for " + container
+ .getContainerId());
+}
+return ipAndHost;
+ }
}
{code}
Instead of this, we should just use NetUtils.getLocalHostname() and
NetUtils.getIPs. we’ll have to modify getIps() because today it expects a
subnet and we want all ips irrespective of subnet.
6)
{code}
+ LOG.info("Docker inspect output for " + containerId + ": " + output);
{code}
Change log level to debug?
7)
{code}
+// Be sure to not use space in the argument, otherwise the
+// extract_values_delim method in container-executor binary
+// cannot parse the arguments correctly.
+super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}"
++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'");
+super.addCommandArguments(containerName);
+return this;
{code}
Ugh. This is an ugly one. I thought there needed to be a space between range
and NetworkSettings because the code passed is meant to be a go template. Use
this instead - {code}--format
'{{range(.NetworkSettings.Networks)}}{{.IPAddress}},{{end}}{{.Config.Hostname}}'{code}.
The output will look like - {code}172.17.0.2,192.168.127.128,c399439fbda9{code}
8)
{code}
+ if (docker_binary == NULL) {
+docker_binary = "docker";
+ }
{code}
Can we make this a common function - the same code is in
launch_docker_container_as_user?
was (Author: vvasudev):
Thanks for the patch [~jianhe]!
1)
{code}
+ @Private
+ @Unstable
+ public abstract void setIp(String ip);
+
{code}
We can’t return just one ip. In the non-docker scenario, multiple ips for a
host are common and Docker itself has support for multiple ips for a container.
We don’t currently support that but it’ll probably be required for us to
support soon. Similarly we should change getIpAndHost and split them up into
getIps and getHost calls.
2)
{code}
+ optional string ip = 7;
+ optional string host = 8;
{code}
Instead of named variables - can we make this a bit more generic and add
support for a simple list of string->list? Then the next time we need to send a
new variable, we don’t need to mess around with the protobuf pieces. Something
like
{code}
message StringListMapProto {
required string key = 1;
optional repeated string value = 2;
}
{code}
and in ContainerStatusProto
{code}
optional repeated StringListMapProto attributes_map = 7;
{code}
3)
{code}
+ public String[] getIpAndHost(Container container) {
+return null;
+ }
{code}
The default implementation shouldn’t return null. We should return the host
name and ip.
4)
{code}
+ public void setIpAndHost(String[] ipAndHost) {
+this.ip = ipAndHost[0];
+this.host = ipAndHost[1];
+ }
{code}
This may become a problem if the size of the array is more than 2(multiple ips)
5)
{code}
+ @Override
+ public String[] getIpAndHost(Container container) {
+String[] ipAndHost = new String[2];
+try {
+ InetAddress address = InetAddress.getLocalHost();
+ ipAndHost[0] =
[jira] [Updated] (YARN-5453) FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource
[
https://issues.apache.org/jira/browse/YARN-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee updated YARN-5453:
---
Attachment: YARN-5453.01.patch
> FairScheduler#update may skip update demand resource of child queue/app if
> current demand reached maxResource
> -
>
> Key: YARN-5453
> URL: https://issues.apache.org/jira/browse/YARN-5453
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: sandflee
>Assignee: sandflee
> Attachments: YARN-5453.01.patch
>
>
> {code}
> demand = Resources.createResource(0);
> for (FSQueue childQueue : childQueues) {
> childQueue.updateDemand();
> Resource toAdd = childQueue.getDemand();
> demand = Resources.add(demand, toAdd);
> demand = Resources.componentwiseMin(demand, maxRes);
> if (Resources.equals(demand, maxRes)) {
> break;
> }
> }
> {code}
> if one singe queue's demand resource exceed maxRes, the other queue's demand
> resource will not update.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406094#comment-15406094
]
Varun Vasudev edited comment on YARN-5430 at 8/3/16 3:46 PM:
-
Thanks for the patch [~jianhe]!
1)
{code}
+ @Private
+ @Unstable
+ public abstract void setIp(String ip);
+
{code}
We can’t return just one ip. In the non-docker scenario, multiple ips for a
host are common and Docker itself has support for multiple ips for a container.
We don’t currently support that but it’ll probably be required for us to
support soon. Similarly we should change getIpAndHost and split them up into
getIps and getHost calls.
2)
{code}
+ optional string ip = 7;
+ optional string host = 8;
{code}
Instead of named variables - can we make this a bit more generic and add
support for a simple list of string->list? Then the next time we need to send a
new variable, we don’t need to mess around with the protobuf pieces. Something
like
{code}
message StringListMapProto {
required string key = 1;
optional repeated string value = 2;
}
{code}
and in ContainerStatusProto
{code}
optional repeated StringListMapProto attributes_map = 7;
{code}
3)
{code}
+ public String[] getIpAndHost(Container container) {
+return null;
+ }
{code}
The default implementation shouldn’t return null. We should return the host
name and ip.
4)
{code}
+ public void setIpAndHost(String[] ipAndHost) {
+this.ip = ipAndHost[0];
+this.host = ipAndHost[1];
+ }
{code}
This may become a problem if the size of the array is more than 2(multiple ips)
5)
{code}
+ @Override
+ public String[] getIpAndHost(Container container) {
+String[] ipAndHost = new String[2];
+try {
+ InetAddress address = InetAddress.getLocalHost();
+ ipAndHost[0] = address.getHostAddress();
+ ipAndHost[1] = address.getHostName();
+} catch (UnknownHostException e) {
+ LOG.error("Unable to get Local hostname and ip for " + container
+ .getContainerId());
+}
+return ipAndHost;
+ }
}
{code}
Instead of this, we should just use NetUtils.getLocalHostname() and
NetUtils.getIPs. we’ll have to modify getIps() because today it expects a
subnet and we want all ips irrespective of subnet.
6)
{code}
+ LOG.info("Docker inspect output for " + containerId + ": " + output);
{code}
Change log level to debug?
7)
{code}
+// Be sure to not use space in the argument, otherwise the
+// extract_values_delim method in container-executor binary
+// cannot parse the arguments correctly.
+super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}"
++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'");
+super.addCommandArguments(containerName);
+return this;
{code}
Ugh. This is an ugly one. I thought there needed to be a space between range
and NetworkSettings because the code passed is meant to be a go template. Use
this instead - {code}--format='{{range(.NetworkSettings.Networks)}}{code}
8)
{code}
+ if (docker_binary == NULL) {
+docker_binary = "docker";
+ }
{code}
Can we make this a common function - the same code is in
launch_docker_container_as_user?
was (Author: vvasudev):
Thanks for the patch [~jianhe]!
1)
{code}
+ @Private
+ @Unstable
+ public abstract void setIp(String ip);
+
{code}
We can’t return just one ip. In the non-docker scenario, multiple ips for a
host are common and Docker itself has support for multiple ips for a container.
We don’t currently support that but it’ll probably be required for us to
support soon. Similarly we should change getIpAndHost and split them up into
getIps and getHost calls.
2)
{code}
+ optional string ip = 7;
+ optional string host = 8;
{code}
Instead of named variables - can we make this a bit more generic and add
support for a simple list of string->list? Then the next time we need to send a
new variable, we don’t need to mess around with the protobuf pieces. Something
like
{code}
message StringListMapProto {
required string key = 1;
optional repeated string value = 2;
}
and in ContainerStatusProto
{code}
optional repeated StringListMapProto attributes_map = 7;
{code}
3)
{code}
+ public String[] getIpAndHost(Container container) {
+return null;
+ }
{code}
The default implementation shouldn’t return null. We should return the host
name and ip.
4)
{code}
+ public void setIpAndHost(String[] ipAndHost) {
+this.ip = ipAndHost[0];
+this.host = ipAndHost[1];
+ }
{code}
This may become a problem if the size of the array is more than 2(multiple ips)
5)
{code}
+ @Override
+ public String[] getIpAndHost(Container container) {
+String[] ipAndHost = new String[2];
+try {
+ InetAddress address = InetAddress.getLocalHost();
+ ipAndHost[0] = address.getHostAddress();
+ ipAndHost[1] = address.getHostName();
+} catch (UnknownHostException e) {
+
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406094#comment-15406094
]
Varun Vasudev commented on YARN-5430:
-
Thanks for the patch [~jianhe]!
1)
{code}
+ @Private
+ @Unstable
+ public abstract void setIp(String ip);
+
{code}
We can’t return just one ip. In the non-docker scenario, multiple ips for a
host are common and Docker itself has support for multiple ips for a container.
We don’t currently support that but it’ll probably be required for us to
support soon. Similarly we should change getIpAndHost and split them up into
getIps and getHost calls.
2)
{code}
+ optional string ip = 7;
+ optional string host = 8;
{code}
Instead of named variables - can we make this a bit more generic and add
support for a simple list of string->list? Then the next time we need to send a
new variable, we don’t need to mess around with the protobuf pieces. Something
like
{code}
message StringListMapProto {
required string key = 1;
optional repeated string value = 2;
}
and in ContainerStatusProto
{code}
optional repeated StringListMapProto attributes_map = 7;
{code}
3)
{code}
+ public String[] getIpAndHost(Container container) {
+return null;
+ }
{code}
The default implementation shouldn’t return null. We should return the host
name and ip.
4)
{code}
+ public void setIpAndHost(String[] ipAndHost) {
+this.ip = ipAndHost[0];
+this.host = ipAndHost[1];
+ }
{code}
This may become a problem if the size of the array is more than 2(multiple ips)
5)
{code}
+ @Override
+ public String[] getIpAndHost(Container container) {
+String[] ipAndHost = new String[2];
+try {
+ InetAddress address = InetAddress.getLocalHost();
+ ipAndHost[0] = address.getHostAddress();
+ ipAndHost[1] = address.getHostName();
+} catch (UnknownHostException e) {
+ LOG.error("Unable to get Local hostname and ip for " + container
+ .getContainerId());
+}
+return ipAndHost;
+ }
}
{code}
Instead of this, we should just use NetUtils.getLocalHostname() and
NetUtils.getIPs. we’ll have to modify getIps() because today it expects a
subnet and we want all ips irrespective of subnet.
6)
{code}
+ LOG.info("Docker inspect output for " + containerId + ": " + output);
{code}
Change log level to debug?
7)
{code}
+// Be sure to not use space in the argument, otherwise the
+// extract_values_delim method in container-executor binary
+// cannot parse the arguments correctly.
+super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}"
++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'");
+super.addCommandArguments(containerName);
+return this;
{code}
Ugh. This is an ugly one. I thought there needed to be a space between range
and NetworkSettings because the code passed is meant to be a go template. Use
this instead - --format='{{range(.NetworkSettings.Networks)}}
8)
{code}
+ if (docker_binary == NULL) {
+docker_binary = "docker";
+ }
{code}
Can we make this a common function - the same code is in
launch_docker_container_as_user?
> Get container's ip and host from NM
> ---
>
> Key: YARN-5430
> URL: https://issues.apache.org/jira/browse/YARN-5430
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch
>
>
> In YARN-4757, we introduced a DNS mechanism for containers. That's based on
> the assumption that we can get the container's ip and host information and
> store it in the registry-service. This jira aims to get the container's ip
> and host from the NM, primarily docker container
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406060#comment-15406060
]
Jun Gong commented on YARN-5333:
Thanks [~jianhe].
Attach a new patch to address above comments. It also fix the checkstyle error.
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5333:
---
Attachment: YARN-5333.07.patch
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch,
> YARN-5333.06.patch, YARN-5333.07.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406049#comment-15406049 ] Arun Suresh commented on YARN-5079: --- We were going thru SLIDER-787. With regard to the state of slider as it is today, especially with regard to upgarde, we had a couple of questions/clarifications: # Every YARN Container started by the Slider AM consists of an Agent + the Component itself. # Given 1) is true, I assume, the startContainers call to the NM would actually be starting the Agent (as the root process) and the agent starts the Component process. # Given 1) and 2), I assume component ‘upgrade’ is a yarn out-of-band protocol between the slider AM and the Agent where the component process is brought down and restarted. This also implies that currently the AM does not lose the allocation of the actual Container (unless the Agent goes down) # Is a Slider upgrade (AM + Agent) supported currently? I assume if only AM needs to be upgraded, then work preserving AM restart must be enabled to allow the existing containers to bind to the new AM. IF the Agent itself needs to upgraded, I guess there is no other option but to lose the Container allocations (which I guess YARN-4876 might solve ?) # From the docs, it also looks like the actual coordination / orchestration of the upgrade is currently not the responsibility of Slider: An external tool can explicitly decide which container (group of containers / roles) to upgrade, and can issue the necessary Slider commands. Right? # As per the discussions on this JIRA, the proposal is to NOT include the Slider agent in the YARN merge. Is there a plan to provide a similar Agent like component within YARN? We would still need an Agent like component that can (re)configure and (re)start processes as well as doing fancy stuff like port allocations etc. Are you guys planning on opening up JIRAs for the purpose (are they already there)? > [Umbrella] Native YARN framework layer for services and beyond > -- > > Key: YARN-5079 > URL: https://issues.apache.org/jira/browse/YARN-5079 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > (See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.1 to track the specific sub-item.) > (This is a companion to YARN-4793 in our effort to simplify the entire story, > but focusing on APIs) > So far, YARN by design has restricted itself to having a very low-level API > that can support any type of application. Frameworks like Apache Hadoop > MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix > and others ended up exposing higher level APIs that end-users can directly > leverage to build their applications on top of YARN. On the services side, > Apache Slider has done something similar. > With our current attention on making services first-class and simplified, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. Beyond the functionality that I outlined in the > previous sections in the doc on how NodeManagers can be enhanced to help > services, the biggest missing piece is the framework itself. There is a lot > of very important functionality that a services' framework can own together > with YARN in executing services end-to-end. > In this JIRA I propose we look at having a native Apache Hadoop framework for > running services natively on YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406001#comment-15406001 ] Jason Lowe commented on YARN-5451: -- No, rather it's more an issue that there's no concept of heartbeat timeouts coming from the container localizer. The localization service starts a localizer then expects that localizer to phone in via heartbeats to control it. It doesn't track the pid or any other way to normally control it. The problem with using a timeout interval at the Shell level is that it puts a hard limit on how long to localize something, and there may be cases where we want to keep localizing because good progress is being made (e.g.: large docker container download). The other issue is that during NM restarts we no longer have a Shell instance associated with those processes since they are no longer children of the new NM process. IMHO the clear indication something is amiss is that startLocalizer returns void. There's no pid or cookie of some kind to associate with the localizer instance. It's fire-and-forget. Once we start the localizer we can't stop it without it voluntarily heartbeating, and if the localizer goes rogue and refuses to heartbeat or respond properly to heartbeat commands the NM can't force it to stop like it can with normal containers. The localizer should be tracked like containers are tracked so we can control them like we can control containers. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-5469:
--
Attachment: YARN-5469.001.patch
Attaching patch to increase timeout to 10 seconds.
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
> Attachments: YARN-5469.001.patch
>
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at
[jira] [Moved] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne moved HADOOP-13462 to YARN-5469:
---
Key: YARN-5469 (was: HADOOP-13462)
Project: Hadoop YARN (was: Hadoop Common)
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Priority: Minor
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
[jira] [Assigned] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[
https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger reassigned YARN-5469:
-
Assignee: Eric Badger
> Increase timeout of TestAmFilter.testFilter
> ---
>
> Key: YARN-5469
> URL: https://issues.apache.org/jira/browse/YARN-5469
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
>
> Timeout is currently only 1 second. Saw a timeout failure
> {noformat}
> java.lang.Exception: test timed out after 1000 milliseconds
> at java.util.zip.ZipFile.getEntry(Native Method)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:311)
> at java.util.jar.JarFile.getEntry(JarFile.java:240)
> at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
> at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841)
> at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at
[jira] [Commented] (YARN-5456) container-executor support for FreeBSD, NetBSD, and others if conf path is absolute
[ https://issues.apache.org/jira/browse/YARN-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405911#comment-15405911 ] Allen Wittenauer commented on YARN-5456: Thanks! > container-executor support for FreeBSD, NetBSD, and others if conf path is > absolute > --- > > Key: YARN-5456 > URL: https://issues.apache.org/jira/browse/YARN-5456 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, security >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: security > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5456.00.patch, YARN-5456.01.patch > > > YARN-5121 fixed quite a few portability issues, but it also changed how it > determines it's location to be very operating specific for security reasons. > We should add support for FreeBSD to unbreak it's ports entry, NetBSD (the > sysctl options are just in a different order), and for operating systems that > do not have a defined method, an escape hatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405857#comment-15405857 ] Ying Zhang commented on YARN-5287: -- Thanks [~Naganarasimha] and [~vvasudev]. Comments addressed and patch updated. Please have a look and kindly help to commit it. > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405835#comment-15405835 ] Shane Kumpf commented on YARN-5428: --- Just to add that in the YARN model, we run the docker commands as root and hence the file needs to be owned by root with 400 permissions minimum. YARN is also not aware of the actual username and password, only where the client config is stored. The administrator or user would need to run "docker login" interactively at least once to generate the config.json file and distribute it across the cluster if they wanted to leverage the client config for credentials. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[
https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405821#comment-15405821
]
Sunil G commented on YARN-5334:
---
Thanks [~Sreenath] for the patch and for working on same. Its really a nice
feature and helpful add-on while using the UI.
Few comments from my side:
1. In {{cluster-overview.js}} , I think we also need to unload *yarn-app* as it
will get fresh set of RUNNING apps.
2. Also in {{yarn-apps.js}}, we can unload *yarn-app*.
3. In {{yarn-nodes.js}}, we can unload *yarn-rm-node*
4. in {{yarn-queue-apps.js}}, we can unload *yarn-app*.
5. In {{yarn-queue.js}}, could we remove the unwanted console logs which prints
all items. Similarly in {{yarn-queues.js}}
6. {{Refresh}} if we can have custom button which can take
property like {{btn-sm btn-primary}}, it ll be helpful. Yes, we can remove
whole line in various templates, but its a big changes in all routes (for the
common action) and templates. We could come with an optimization and
improvement ticket later for same.
I also did an initial sanity tests. Looks good. Will do some more test and
update if I find some issues.
> [YARN-3368] Introduce REFRESH button in various UI pages
>
>
> Key: YARN-5334
> URL: https://issues.apache.org/jira/browse/YARN-5334
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Reporter: Sunil G
>Assignee: Sreenath Somarajapuram
> Attachments: YARN-5334-YARN-3368-0001.patch
>
>
> It will be better if we have a common Refresh button in all pages to get the
> latest information in all tables such as apps/nodes/queue etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[
https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405776#comment-15405776
]
Hadoop QA commented on YARN-5334:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 3s {color}
| {color:red} Docker failed to build yetus/hadoop:6d3a5f5. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821818/YARN-5334-YARN-3368-0001.patch
|
| JIRA Issue | YARN-5334 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12626/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Introduce REFRESH button in various UI pages
>
>
> Key: YARN-5334
> URL: https://issues.apache.org/jira/browse/YARN-5334
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Reporter: Sunil G
>Assignee: Sreenath Somarajapuram
> Attachments: YARN-5334-YARN-3368-0001.patch
>
>
> It will be better if we have a common Refresh button in all pages to get the
> latest information in all tables such as apps/nodes/queue etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated YARN-5334: - Attachment: YARN-5334-YARN-3368-0001.patch [~sunilg] [~leftnoteasy] Please help. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4602) Simple and Scalable Message Service for YARN application
[ https://issues.apache.org/jira/browse/YARN-4602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405770#comment-15405770 ] Arun Suresh commented on YARN-4602: --- [~djp], wondering if you've taken a look at apache REEF. It uses an event driven framework called https://reef.apache.org/wake.html that also supports messaging. It uses Netty under the hood. > Simple and Scalable Message Service for YARN application > > > Key: YARN-4602 > URL: https://issues.apache.org/jira/browse/YARN-4602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications, resourcemanager >Reporter: Junping Du >Assignee: Junping Du > > We are proposing to support MR AM restart with work preserving in > MAPREDUCE-6608 (https://issues.apache.org/jira/browse/MAPREDUCE-6608) that > when AM get failed for some reason, the inflight tasks will keep > running/pending until new AM attempt comes back to continue. One of > prerequisite is tasks should know where the new AM attempt get launched so > TaskUmbilicalProtocol can get retry between clients and new server. > There could be the same requirement for other applications running on YARN > too. Some application decide to handle message delivery itself, e.g. Long > running services can leverage Slider agent to notify messages back and forth. > However, vanilla applications on YARN is hard to achieve this because Hadoop > RPC mechanism essentially is a single way of communication. Although two > directions mechanism like heartbeats (between NM-RM or AM-RM) can get built > on top of it, it make less sense to build the same mechanism between AM and > its application containers - or it need to handle massive of client > connections in AM which could be the new bottleneck for scalability and very > complicated in state maintaining. Instead, we need a new message mechanism > that is simple and scalable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5460) Change container runtime type logging in DelegatingLinuxContainerRuntime to debug
[
https://issues.apache.org/jira/browse/YARN-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405767#comment-15405767
]
Shane Kumpf commented on YARN-5460:
---
Thanks, [~vvasudev]!
> Change container runtime type logging in DelegatingLinuxContainerRuntime to
> debug
> -
>
> Key: YARN-5460
> URL: https://issues.apache.org/jira/browse/YARN-5460
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Trivial
> Fix For: 2.9.0
>
> Attachments: YARN-5460.001.patch
>
>
> When a container is reacquired, signalContainer is called via
> DelegatingLinuxContainerRuntime every second to validate that the container
> is still running, until the container completes or is killed, resulting in
> many of the following log entries:
> {code}
> 2016-08-01 15:28:14,529 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DefaultLinuxContainerRuntime
> 2016-08-01 15:28:14,533 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DockerLinuxContainerRuntime
> 2016-08-01 15:28:15,537 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DefaultLinuxContainerRuntime
> 2016-08-01 15:28:15,540 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DockerLinuxContainerRuntime
> 2016-08-01 15:28:16,550 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DefaultLinuxContainerRuntime
> 2016-08-01 15:28:16,553 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DockerLinuxContainerRuntime
> 2016-08-01 15:28:17,561 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DefaultLinuxContainerRuntime
> 2016-08-01 15:28:17,561 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DockerLinuxContainerRuntime
> 2016-08-01 15:28:18,570 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DefaultLinuxContainerRuntime
> 2016-08-01 15:28:18,570 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime:
> Using container runtime: DockerLinuxContainerRuntime
> {code}
> The following should be changed to debug:
> {code}if (LOG.isInfoEnabled()) {
> LOG.info("Using container runtime: " + runtime.getClass()
> .getSimpleName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405755#comment-15405755 ] Shane Kumpf commented on YARN-5428: --- To be clear, this patch is enabling the ability to set the docker client's configuration directory. It is up to the administrator if they want to store credentials in that configuration. I'm not advocating a global credential store in any regard. Other client configuration is also stored in config.json. To answer your question though, one scenario where this is useful is to allow YARN to automatically pull the image from a docker private repository as needed. A read-only user can be created at Docker hub and given read access to the images they require. By storing this read-only credential in config.json, and setting the property provided in this patch, the administrator or end user no longer needs to "pre-pull" the image on all machines, which is an administrative burden. As Zhankun points out, it is common that the config is owned by root and perms set to 700, but again, it is up to the administrator how they want to leverage the docker client config. Hope that helps. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[
https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405751#comment-15405751
]
Hadoop QA commented on YARN-5287:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 8s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821811/YARN-5287.005.patch |
| JIRA Issue | YARN-5287 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 11e214400a4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d848184 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12625/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12625/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> LinuxContainerExecutor fails to set proper permission
> -
>
> Key: YARN-5287
> URL: https://issues.apache.org/jira/browse/YARN-5287
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Ying Zhang
>Assignee: Ying Zhang
>Priority: Minor
> Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch,
> YARN-5287.004.patch, YARN-5287.005.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> LinuxContainerExecutor fails to set the proper permissions on the local
> directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster
> has been configured with a restrictive umask, e.g.: umask 077. Job failed due
> to the following reason:
> Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has
> permission 700 but needs permission 750
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Zhang updated YARN-5287: - Attachment: YARN-5287.005.patch > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[
https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405656#comment-15405656
]
Weiwei Yang commented on YARN-5402:
---
Hello [~sunilg]
Some updates
I just tried latest trunk, there is no such issue.
I setup a 1 node cluster and tried again with 3368 branch, this did not happen
to me as well. (Previously I was setting a 3 nodes cluster, not sure if that
matters)
I'd like to keep this open for a while in case I am able to reproduce it again.
> Fix NoSuchMethodError in ClusterMetricsInfo
> ---
>
> Key: YARN-5402
> URL: https://issues.apache.org/jira/browse/YARN-5402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Affects Versions: YARN-3368
>Reporter: Weiwei Yang
> Attachments: YARN-5402.YARN-3368.001.patch
>
>
> When trying out new UI on a cluster, the index page failed to load because of
> error {code}java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405655#comment-15405655 ] Varun Vasudev commented on YARN-5307: - +1. Let me know if I should commit or you can take care of it. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch, > YARN-5307-YARN-2915-v4.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405645#comment-15405645 ] Zhankun Tang commented on YARN-5428: The admin provided creds in config.json is to avoid interactive docker pull I think. Indeed, an admin can store several auths for individual repositories. In this case, the application should provide both repo and image name for YARN. For the security of global creds, because the config.json can only be owned by root and set permission to 700, I guess no one else can steal it. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[
https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405620#comment-15405620
]
Varun Vasudev commented on YARN-3664:
-
Thanks for the updated patch [~subru].
{code}
- addReservationResourcesToProto();
+ addSubClusterInfosToProto();
{code}
Is this correct? Should both be added to the proto or are you fixing a bug in
the earlier version?
+1 apart from the comment above. Let me know if I need to commit the patch or
you'll take care of it.
> Federation PolicyStore internal APIs
>
>
> Key: YARN-3664
> URL: https://issues.apache.org/jira/browse/YARN-3664
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3664-YARN-2915-v0.patch,
> YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch,
> YARN-3664-YARN-2915-v3.patch
>
>
> The federation Policy Store contains information about the capacity
> allocations made by users, their mapping to sub-clusters and the policies
> that each of the components (Router, AMRMPRoxy, RMs) should enforce
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5042) Mount /sys/fs/cgroup into Docker containers as read only mount
[
https://issues.apache.org/jira/browse/YARN-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405606#comment-15405606
]
Varun Vasudev commented on YARN-5042:
-
[~luhuichun] - thanks for the latest patch. Some things need to be fixed -
1)
{code}
+if (new File("/sys/fs/cgroup").exists())
+ runCommand.addMountLocation("/sys/fs/cgroup", "/sys/fs/cgroup:ro");
+
{code}
Instead of checking if the location exists in DockerLinuxContainerRuntime.java
- can you add a new function in DockerRunCommand.java which takes the paths and
a boolean as a param - which allows the user to toggle whether the mount should
happen if the source doesn't exist.
Something like -
{code}
public DockerRunCommand addMountLocation(String sourcePath, String
destinationPath, boolean createSource) {
boolean sourceExists = new File(sourcePath).exists();
if(!sourceExists && !createSource) {
return this;
}
super.addCommandArguments("-v", sourcePath + ":" + destinationPath);
return this;
}
{code}
You should also modify the existing mountCommand to call this version with
createSource as true because that's the current behavior. Adding the new
function should also fix the FindBugs warning.
2)
Please fix the failing unit test in TestDockerContainerRuntime.java.
> Mount /sys/fs/cgroup into Docker containers as read only mount
> --
>
> Key: YARN-5042
> URL: https://issues.apache.org/jira/browse/YARN-5042
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Varun Vasudev
>Assignee: luhuichun
> Attachments: YARN-5042.001.patch, YARN-5042.002.patch,
> YARN-5042.003.patch
>
>
> Containers running systemd need access to /sys/fs/cgroup. We should mount it
> into the container as a read only mount.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405539#comment-15405539
]
Jian He commented on YARN-5333:
---
bq. then call rm.handleTransitionToStandBy.
I think the ActiveStandbyElector will handle reJoin automatically if exception
thrown from the transtionToActive method.
We can use the fail fast config for this scenario.
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, YARN-5333.06.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405503#comment-15405503
]
Jun Gong commented on YARN-5333:
Hi [~jianhe], I think the
[comment|https://issues.apache.org/jira/browse/YARN-3893?focusedCommentId=14708471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14708471]
in YARN-3893 makes sense. How about it?
{quote}
If refreshAll() fails, BOTH RM will be in ACTIVE state as per this defect.
Continuing RM services with invalid configuration does not good idea. Moreover
invalid configurations should be notified to user immediately. So it would be
better to make use of fail-fast configuration to exit the RM JVM. If this
configuration is set to false , then call rm.handleTransitionToStandBy.
{quote}
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch, YARN-5333.02.patch,
> YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, YARN-5333.06.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will put recovered apps into default queue because
> it might have not loaded the new {{fair-scheduler.xml}}. We need call
> {{initScheduler}} before start active services or bring {{refreshAll()}} in
> front of {{rm.transitionToActive()}}. *It seems it is also important for
> other scheduler*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5042) Mount /sys/fs/cgroup into Docker containers as read only mount
[
https://issues.apache.org/jira/browse/YARN-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405486#comment-15405486
]
Hadoop QA commented on YARN-5042:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 12s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 5s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| | Hard coded reference to an absolute pathname in
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
At DockerLinuxContainerRuntime.java:absolute pathname in
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
At DockerLinuxContainerRuntime.java:[line 349] |
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821784/YARN-5042.003.patch |
| JIRA Issue | YARN-5042 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 632d92a35025 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d848184 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
