date:20150625

[jira] [Commented] (YARN-2871) TestRMRestart#testRMRestartGetApplicationList sometime fails in trunk

2015-06-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600817#comment-14600817
 ] 

Hadoop QA commented on YARN-2871:
-

\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  15m 51s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:green}+1{color} | javac |   7m 31s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 34s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 46s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   1m 25s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |  50m 49s | Tests passed in 
hadoop-yarn-server-resourcemanager. |
| | |  88m 30s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12741794/YARN-2871.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a815cc1 |
| hadoop-yarn-server-resourcemanager test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8341/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/8341/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/8341/console |


This message was automatically generated.

 TestRMRestart#testRMRestartGetApplicationList sometime fails in trunk
 -

 Key: YARN-2871
 URL: https://issues.apache.org/jira/browse/YARN-2871
 Project: Hadoop YARN
  Issue Type: Test
Reporter: Ted Yu
Assignee: zhihai xu
Priority: Minor
 Attachments: YARN-2871.000.patch, YARN-2871.001.patch


 From trunk build #746 (https://builds.apache.org/job/Hadoop-Yarn-trunk/746):
 {code}
 Failed tests:
   TestRMRestart.testRMRestartGetApplicationList:957
 rMAppManager.logApplicationSummary(
 isA(org.apache.hadoop.yarn.api.records.ApplicationId)
 );
 Wanted 3 times:
 - at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:957)
 But was 2 times:
 - at 
 org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.handle(RMAppManager.java:66)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3849) Too much of preemption activity causing continuos killing of containers across queues

2015-06-25 Thread Wangda Tan (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14601848#comment-14601848
]

Wangda Tan commented on YARN-3849:
--

[~sunilg],
Trying to understand this issue, when the toObtainResource becomes 10,0, and
assume container size are c1=2,1, c2=5,3, c3=4,2, c4=2,1. Preemption
policy will kill c1..c3, my understanding of this problem is preemption policy
can preempt one of the resource type (CPU/Memory) more than needed, but I'm not
sure why it preempts all containers except AM.

Too much of preemption activity causing continuos killing of containers
across queues
-

Key: YARN-3849
URL: https://issues.apache.org/jira/browse/YARN-3849
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Sunil G
Assignee: Sunil G
Priority: Critical

Two queues are used. Each queue has given a capacity of 0.5. Dominant
Resource policy is used.
1. An app is submitted in QueueA which is consuming full cluster capacity
2. After submitting an app in QueueB, there are some demand and invoking
preemption in QueueA
3. Instead of killing the excess of 0.5 guaranteed capacity, we observed that
all containers other than AM is getting killed in QueueA
4. Now the app in QueueB is trying to take over cluster with the current free
space. But there are some updated demand from the app in QueueA which lost
its containers earlier, and preemption is kicked in QueueB now.
Scenario in step 3 and 4 continuously happening in loop. Thus none of the
apps are completing.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-3793) Several NPEs when deleting local files on NM recovery

2015-06-25 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3793:
---
Attachment: YARN-3793.01.patch

 Several NPEs when deleting local files on NM recovery
 -

 Key: YARN-3793
 URL: https://issues.apache.org/jira/browse/YARN-3793
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.6.0
Reporter: Karthik Kambatla
Assignee: Varun Saxena
Priority: Critical
 Attachments: YARN-3793.01.patch


 When NM work-preserving restart is enabled, we see several NPEs on recovery. 
 These seem to correspond to sub-directories that need to be deleted. I wonder 
 if null pointers here mean incorrect tracking of these resources and a 
 potential leak. This JIRA is to investigate and fix anything required.
 Logs show:
 {noformat}
 2015-05-18 07:06:10,225 INFO 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting 
 absolute path : null
 2015-05-18 07:06:10,224 ERROR 
 org.apache.hadoop.yarn.server.nodemanager.DeletionService: Exception during 
 execution of task in DeletionService
 java.lang.NullPointerException
 at 
 org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:274)
 at org.apache.hadoop.fs.FileContext.delete(FileContext.java:755)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.deleteAsUser(DefaultContainerExecutor.java:458)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:293)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-3793) Several NPEs when deleting local files on NM recovery

2015-06-25 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-3793:
---
Priority: Critical  (was: Major)

 Several NPEs when deleting local files on NM recovery
 -

 Key: YARN-3793
 URL: https://issues.apache.org/jira/browse/YARN-3793
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.6.0
Reporter: Karthik Kambatla
Assignee: Varun Saxena
Priority: Critical
 Attachments: YARN-3793.01.patch


 When NM work-preserving restart is enabled, we see several NPEs on recovery. 
 These seem to correspond to sub-directories that need to be deleted. I wonder 
 if null pointers here mean incorrect tracking of these resources and a 
 potential leak. This JIRA is to investigate and fix anything required.
 Logs show:
 {noformat}
 2015-05-18 07:06:10,225 INFO 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting 
 absolute path : null
 2015-05-18 07:06:10,224 ERROR 
 org.apache.hadoop.yarn.server.nodemanager.DeletionService: Exception during 
 execution of task in DeletionService
 java.lang.NullPointerException
 at 
 org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:274)
 at org.apache.hadoop.fs.FileContext.delete(FileContext.java:755)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.deleteAsUser(DefaultContainerExecutor.java:458)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:293)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (YARN-3855) If acl is enabled and http.authentication.type is simple, user cannot view the app page in default setup

2015-06-25 Thread Jian He (JIRA)

[
https://issues.apache.org/jira/browse/YARN-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602341#comment-14602341
]

Jian He edited comment on YARN-3855 at 6/26/15 4:35 AM:

I believe what you suggested is a general good practice to setup secure
cluster. Btw, the patch did not enable/enforce any of this. People can config
whatever they want for the http authentication regardless how the rest
components are setup before this jira. The point of this jira is to prevent
this scenario that user cannot view any application (even for its own
application) in whatever way unless the daemon is restarted.

was (Author: jianhe):
I believe what you suggested is a general good practice to setup secure
cluster. Btw, the patch did not enable/enforce any of this. People can config
whatever they want for the http authentication regardless how the rest
components are setup before this jira. The point of this jira is to prevent
this scenario that user cannot view the applications in whatever way unless the
daemon is restarted.

If acl is enabled and http.authentication.type is simple, user cannot view
the app page in default setup

Key: YARN-3855
URL: https://issues.apache.org/jira/browse/YARN-3855
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3855.1.patch, YARN-3855.2.patch

If all ACLs (admin acl, queue-admin-acls etc.) are setup properly and
http.authentication.type is 'simple' in secure mode , user cannot view the
application web page in default setup because the incoming user is always
considered as dr.who . User also cannot pass user.name to indicate the
incoming user name, because AuthenticationFilterInitializer is not enabled by
default. This is inconvenient from user's perspective.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-3826) Race condition in ResourceTrackerService leads to wrong diagnostics messages

2015-06-25 Thread Devaraj K (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated YARN-3826:


The timed out test is not related to the patch.

+1, will commit it shortly.

 Race condition in ResourceTrackerService leads to wrong diagnostics messages
 

 Key: YARN-3826
 URL: https://issues.apache.org/jira/browse/YARN-3826
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
 Attachments: YARN-3826.01.patch, YARN-3826.02.patch, 
 YARN-3826.03.patch


 Since we are calling {{setDiagnosticsMessage}} in {{nodeHeartbeat}}, which 
 can be called concurrently, the static {{resync}} and {{shutdown}} may have 
 wrong diagnostics messages in some cases.
 On the other side, these static members can hardly save any memory, since the 
 normal heartbeat responses are created for each heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-3745) SerializedException should also try to instantiate internal exception with the default constructor

2015-06-25 Thread Devaraj K (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated YARN-3745:

Hadoop Flags: Reviewed

+1, latest patch looks good to me, will commit it shortly.

 SerializedException should also try to instantiate internal exception with 
 the default constructor
 --

 Key: YARN-3745
 URL: https://issues.apache.org/jira/browse/YARN-3745
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Lavkesh Lahngir
Assignee: Lavkesh Lahngir
 Attachments: YARN-3745.1.patch, YARN-3745.2.patch, YARN-3745.3.patch, 
 YARN-3745.patch


 While deserialising a SerializedException it tries to create internal 
 exception in instantiateException() with cn = 
 cls.getConstructor(String.class).
 if cls does not has a constructor with String parameter it throws 
 Nosuchmethodexception
 for example ClosedChannelException class.  
 We should also try to instantiate exception with default constructor so that 
 inner exception can to propagated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3826) Race condition in ResourceTrackerService: potential wrong diagnostics messages

2015-06-25 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600981#comment-14600981
 ] 

Hadoop QA commented on YARN-3826:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  16m 46s | Pre-patch trunk has 3 extant 
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |   7m 35s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 35s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   1m 14s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 26s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |   0m 24s | Tests passed in 
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests |  61m  0s | Tests failed in 
hadoop-yarn-server-resourcemanager. |
| | | 101m 33s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12741817/YARN-3826.03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a815cc1 |
| Pre-patch Findbugs warnings | 
https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
 |
| hadoop-yarn-server-common test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
 |
| hadoop-yarn-server-resourcemanager test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/8342/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/8342/console |


This message was automatically generated.

 Race condition in ResourceTrackerService: potential wrong diagnostics messages
 --

 Key: YARN-3826
 URL: https://issues.apache.org/jira/browse/YARN-3826
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.7.0
Reporter: Chengbing Liu
Assignee: Chengbing Liu
 Attachments: YARN-3826.01.patch, YARN-3826.02.patch, 
 YARN-3826.03.patch


 Since we are calling {{setDiagnosticsMessage}} in {{nodeHeartbeat}}, which 
 can be called concurrently, the static {{resync}} and {{shutdown}} may have 
 wrong diagnostics messages in some cases.
 On the other side, these static members can hardly save any memory, since the 
 normal heartbeat responses are created for each heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 >

1 - 100 of 131 matches

Mail list logo