[jira] [Updated] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[ https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4000: --- Attachment: YARN-4000.06.patch > RM crashes with NPE if leaf queue becomes parent queue during restart > - > > Key: YARN-4000 > URL: https://issues.apache.org/jira/browse/YARN-4000 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-4000.01.patch, YARN-4000.02.patch, > YARN-4000.03.patch, YARN-4000.04.patch, YARN-4000.05.patch, YARN-4000.06.patch > > > This is a similar situation to YARN-2308. If an application is active in > queue A and then the RM restarts with a changed capacity scheduler > configuration where queue A becomes a parent queue to other subqueues then > the RM will crash with a NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[
https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957075#comment-14957075
]
Hadoop QA commented on YARN-4262:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 55s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 57s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 51s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 4s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 8m 52s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 59m 27s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766551/YARN-4262.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d6c8bad |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9442/console |
This message was automatically generated.
> Allow admins to run privileged docker containers.
> --
>
> Key: YARN-4262
> URL: https://issues.apache.org/jira/browse/YARN-4262
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-4262.001.patch
>
>
> There are scenarios where privileged containers are necessary in order to run
> certain kinds of applications (one example is trying to run postresql/oracle
> inside containers). However, given the security implications, we should
> ensure that :
> 1) privileged containers are disabled by default, even for admins
> 2) if enabled, only admins should be allowed to launch such containers and
> 3) Not all containers launched by admin users need to be privileged
> containers : admin users need to explicitly request that a privileged
> container be launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4170) AM need to be notified with priority in AllocateResponse
[
https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-4170:
--
Attachment: 0007-YARN-4170.patch
{{TestCapacityScheduler.testPreemptionInfo}} failure is not related to this
patch. Test case can run successfully in its local runs. However this is a
known sporadic issue and YARN-3533 fixed it in {{MockRM#launchAM}}. But
{{TestCapacityScheduler.testPreemptionInfo}} is using a local {{launchAM}}
within TestCapacityScheduler which is similar to MockRM api and doesnt have the
fix in YARN-3533. I feel we do not need this local method, rather
{{MockRM#launchAM}} can be used instead.
I am fixing this also along with this new patch. Please suggest if needed to be
tracked separately.
> AM need to be notified with priority in AllocateResponse
> -
>
> Key: YARN-4170
> URL: https://issues.apache.org/jira/browse/YARN-4170
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch,
> 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch,
> 0006-YARN-4170.patch, 0007-YARN-4170.patch
>
>
> As discussed in MAPREDUCE-5870, Application Master need to be notified with
> priority in Allocate heartbeat. This will help AM to know the priority and
> can update JobStatus when client asks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957002#comment-14957002 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/CHANGES.txt > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4258) Add support for controlling capabilities for docker containers
[
https://issues.apache.org/jira/browse/YARN-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957023#comment-14957023
]
Hadoop QA commented on YARN-4258:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 16s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 18s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 54s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 5s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 8m 54s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 60m 21s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766513/YARN-4258.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d6c8bad |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9441/console |
This message was automatically generated.
> Add support for controlling capabilities for docker containers
> --
>
> Key: YARN-4258
> URL: https://issues.apache.org/jira/browse/YARN-4258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-4258.001.patch, YARN-4258.002.patch,
> YARN-4258.003.patch, YARN-4258.004.patch
>
>
> Docker allows us to specify which capabilities a container is assigned at
> launch time. We need a mechanism (admin controlled) that will allow us to
> define this set of capabilities that docker containers are launched with.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957119#comment-14957119 ] Sidharta Seethana commented on YARN-4262: - The checkstyle issue refers to the length of YarnConfiguration.java (there isn't much that can be done about that at this point). Pre-patch failure is unrelated to this patch. > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957127#comment-14957127 ] Allen Wittenauer commented on YARN-4262: "admin" is the wrong thing to do here. It really should be a different list of users so that the two feature sets can have separation of privileges. > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4258) Add support for controlling capabilities for docker containers
[ https://issues.apache.org/jira/browse/YARN-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957096#comment-14957096 ] Varun Vasudev commented on YARN-4258: - +1. I'll commit this tomorrow if no one objects. > Add support for controlling capabilities for docker containers > -- > > Key: YARN-4258 > URL: https://issues.apache.org/jira/browse/YARN-4258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4258.001.patch, YARN-4258.002.patch, > YARN-4258.003.patch, YARN-4258.004.patch > > > Docker allows us to specify which capabilities a container is assigned at > launch time. We need a mechanism (admin controlled) that will allow us to > define this set of capabilities that docker containers are launched with. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4009) CORS support for ResourceManager REST API
[ https://issues.apache.org/jira/browse/YARN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957098#comment-14957098 ] Varun Vasudev commented on YARN-4009: - The test failures are unrelated to the patch. > CORS support for ResourceManager REST API > - > > Key: YARN-4009 > URL: https://issues.apache.org/jira/browse/YARN-4009 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prakash Ramachandran >Assignee: Varun Vasudev > Attachments: YARN-4009.001.patch, YARN-4009.002.patch, > YARN-4009.003.patch, YARN-4009.004.patch, YARN-4009.005.patch, > YARN-4009.006.patch, YARN-4009.007.patch > > > Currently the REST API's do not have CORS support. This means any UI (running > in browser) cannot consume the REST API's. For ex Tez UI would like to use > the REST API for getting application, application attempt information exposed > by the API's. > It would be very useful if CORS is enabled for the REST API's. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4221) Store user in app to flow table
[ https://issues.apache.org/jira/browse/YARN-4221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957171#comment-14957171 ] Varun Saxena commented on YARN-4221: Addition of javadocs for REST APIs' can be done as part of YARN-4224 > Store user in app to flow table > --- > > Key: YARN-4221 > URL: https://issues.apache.org/jira/browse/YARN-4221 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-4221-YARN-2928.01.patch, > YARN-4221-YARN-2928.02.patch > > > We should store user as well in in app to flow table. > For queries where user is not supplied and flow context can be retrieved from > app to flow table, we should take the user from app to flow table instead of > considering UGI as default user. > This is as per discussion on YARN-3864 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-3216: -- Attachment: 0007-YARN-3216.patch Hi [~leftnoteasy] Attaching patch addressing most of the comments except "why computing max-queue limit". I have share my thoughts in earlier comment, kindly help to check the same. As needed, I will update subsequent patch based on discussion. > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957355#comment-14957355
]
Hudson commented on YARN-4250:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #495 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/495/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[ https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957370#comment-14957370 ] Rohith Sharma K S commented on YARN-2729: - Overall patch look good to me. I have one doubt/comment. If I am missing anything correct me, # NodeLabelsProvider is abstract class. Why it was not made as Interface? # And for each NodeLabelsProvide implementing any other way in future which require modification in NodeManager to create an instance of NodeLabelsProvider. Instead of that loading implemented class would be easier and also it is pluggable. Any thoughts? cc :/[~leftnoteasy] > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup > --- > > Key: YARN-2729 > URL: https://issues.apache.org/jira/browse/YARN-2729 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch, > YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch, > YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch, > YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch, > YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch, > YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch, > YARN-2729.20150925-1.patch, YARN-2729.20151310-1.patch, > YARN-2729.20151310-2.patch > > > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3224) Notify AM with containers (on decommissioning node) could be preempted after timeout.
[ https://issues.apache.org/jira/browse/YARN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957377#comment-14957377 ] Sunil G commented on YARN-3224: --- Hi [~djp] Could you please help to check this. I have now implemented this patch without updating the preemption timeout to RM. The dependent JIRA is YARN-3784 looks gone stale. Could you please help to advise how we can proceed here. May be I feel we can get this in w/o updating preemption timeout. And once that ticket gets in, we can try making the update for DECOMMISION scenario. Kindly advice. > Notify AM with containers (on decommissioning node) could be preempted after > timeout. > - > > Key: YARN-3224 > URL: https://issues.apache.org/jira/browse/YARN-3224 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Sunil G > Attachments: 0001-YARN-3224.patch, 0002-YARN-3224.patch > > > We should leverage YARN preemption framework to notify AM that some > containers will be preempted after a timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4264) In-VM test ATS instances fail with metrics already registered
Steve Loughran created YARN-4264: Summary: In-VM test ATS instances fail with metrics already registered Key: YARN-4264 URL: https://issues.apache.org/jira/browse/YARN-4264 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.8.0 Reporter: Steve Loughran Testing my SPARK-1537 code against branch-2 triggers stack traces due to failed attempts to re-register metrics. This is with code which works against 2.7.1, so it's a regression. Either the timeline server needs to unregister its metrics on shutdown, or ATS adds an option to disable metrics for test purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[
https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957392#comment-14957392
]
Rohith Sharma K S commented on YARN-2729:
-
Adding to above
# From the below description, I assume that NodeLabelsProvider instance can be
created only if configuratin-type is distributed. And I think before creating
an instance for provider, there should be check for configuration-type also.
{noformat}
When node labels "yarn.node-labels.configuration-type" is of type "distributed"
Administrators can configure the source of the node labels provider by
configuring this parameter
{noformat}
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup
> ---
>
> Key: YARN-2729
> URL: https://issues.apache.org/jira/browse/YARN-2729
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch,
> YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch,
> YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch,
> YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch,
> YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch,
> YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch,
> YARN-2729.20150925-1.patch, YARN-2729.20151310-1.patch,
> YARN-2729.20151310-2.patch
>
>
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[ https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957293#comment-14957293 ] Varun Saxena commented on YARN-4000: Checkstyle is related to file length. I guess no need to fix it. Should I upload another patch for whitespace error or you will take care while applying patch during commit(if not further comments) ? > RM crashes with NPE if leaf queue becomes parent queue during restart > - > > Key: YARN-4000 > URL: https://issues.apache.org/jira/browse/YARN-4000 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-4000.01.patch, YARN-4000.02.patch, > YARN-4000.03.patch, YARN-4000.04.patch, YARN-4000.05.patch, YARN-4000.06.patch > > > This is a similar situation to YARN-2308. If an application is active in > queue A and then the RM restarts with a changed capacity scheduler > configuration where queue A becomes a parent queue to other subqueues then > the RM will crash with a NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4227) FairScheduler: RM quits processing expired container from a removed node
[
https://issues.apache.org/jira/browse/YARN-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957372#comment-14957372
]
Anubhav Dhoot commented on YARN-4227:
-
The previous statement should also be updated to handle a null node to avoid a
NPE inside it
{noformat}application.unreserve(rmContainer.getReservedPriority(),
node);{noformat}
This may need to still process some portion FSAppAttempt#unreserveInternal
instead of skipping the entire processing.
The test seems ok. Should we rename blacklist -> remove?
Overall the fix looks ok. Just another bug which indicates until we restructure
the code we will have to keep adding bandaids.
> FairScheduler: RM quits processing expired container from a removed node
>
>
> Key: YARN-4227
> URL: https://issues.apache.org/jira/browse/YARN-4227
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.3.0, 2.5.0, 2.7.1
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Critical
> Attachments: YARN-4227.2.patch, YARN-4227.3.patch, YARN-4227.4.patch,
> YARN-4227.patch
>
>
> Under some circumstances the node is removed before an expired container
> event is processed causing the RM to exit:
> {code}
> 2015-10-04 21:14:01,063 INFO
> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> Expired:container_1436927988321_1307950_01_12 Timed out after 600 secs
> 2015-10-04 21:14:01,063 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_1436927988321_1307950_01_12 Container Transitioned from
> ACQUIRED to EXPIRED
> 2015-10-04 21:14:01,063 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp:
> Completed container: container_1436927988321_1307950_01_12 in state:
> EXPIRED event:EXPIRE
> 2015-10-04 21:14:01,063 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=system_op
>OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS
> APPID=application_1436927988321_1307950
> CONTAINERID=container_1436927988321_1307950_01_12
> 2015-10-04 21:14:01,063 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event type CONTAINER_EXPIRED to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.completedContainer(FairScheduler.java:849)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1273)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:122)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:585)
> at java.lang.Thread.run(Thread.java:745)
> 2015-10-04 21:14:01,063 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
> {code}
> The stack trace is from 2.3.0 but the same issue has been observed in 2.5.0
> and 2.6.0 by different customers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4260) Check NodeManager's validity based on hostname(IP) and port
[ https://issues.apache.org/jira/browse/YARN-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaehoon ko updated YARN-4260: - Attachment: YARN-4260.003.patch patch against 40cac59248f17c59fc819f4145cdeac9db309626 (current trunk) > Check NodeManager's validity based on hostname(IP) and port > --- > > Key: YARN-4260 > URL: https://issues.apache.org/jira/browse/YARN-4260 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.1 >Reporter: jaehoon ko > Attachments: YARN-4260.001.patch, YARN-4260.002.patch, > YARN-4260.003.patch > > > There can be situations where multiple NMs are running on the same host; > emulating multi-host YARN cluster on one host, deploying dockerized NMs that > share network stack. In such cases, RM should be able to distinguish NMs > running on the same host. However, current implementation checks validity of > NM based only on hostname (NodesListManager.isValidNode()). So it is not > possible to distinguish a specific NM from other NMs on the same host. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4260) Check NodeManager's validity based on hostname(IP) and port
[
https://issues.apache.org/jira/browse/YARN-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956345#comment-14956345
]
Hadoop QA commented on YARN-4260:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 3s | The patch command could not apply
the patch. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766474/YARN-4260.002.patch |
| Optional Tests | shellcheck javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 2a98724 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9436/console |
This message was automatically generated.
> Check NodeManager's validity based on hostname(IP) and port
> ---
>
> Key: YARN-4260
> URL: https://issues.apache.org/jira/browse/YARN-4260
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: jaehoon ko
> Attachments: YARN-4260.001.patch, YARN-4260.002.patch
>
>
> There can be situations where multiple NMs are running on the same host;
> emulating multi-host YARN cluster on one host, deploying dockerized NMs that
> share network stack. In such cases, RM should be able to distinguish NMs
> running on the same host. However, current implementation checks validity of
> NM based only on hostname (NodesListManager.isValidNode()). So it is not
> possible to distinguish a specific NM from other NMs on the same host.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4260) Check NodeManager's validity based on hostname(IP) and port
[
https://issues.apache.org/jira/browse/YARN-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956519#comment-14956519
]
Hadoop QA commented on YARN-4260:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 16s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 49s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 27s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 55m 35s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 96m 0s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
|
| |
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766490/YARN-4260.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2a98724 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9437/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9437/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9437/console |
This message was automatically generated.
> Check NodeManager's validity based on hostname(IP) and port
> ---
>
> Key: YARN-4260
> URL: https://issues.apache.org/jira/browse/YARN-4260
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: jaehoon ko
> Attachments: YARN-4260.001.patch, YARN-4260.002.patch,
> YARN-4260.003.patch
>
>
> There can be situations where multiple NMs are running on the same host;
> emulating multi-host YARN cluster on one host, deploying dockerized NMs that
> share network stack. In such cases, RM should be able to distinguish NMs
> running on the same host. However, current implementation checks validity of
> NM based only on hostname (NodesListManager.isValidNode()). So it is not
> possible to distinguish a specific NM from other NMs on the same host.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4258) Add support for controlling capabilities for docker containers
[
https://issues.apache.org/jira/browse/YARN-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956588#comment-14956588
]
Hadoop QA commented on YARN-4258:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 52s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 12s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 53s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 28s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 7s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 8m 52s | Tests failed in
hadoop-yarn-server-nodemanager. |
| | | 59m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766492/YARN-4258.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2a98724 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9438/console |
This message was automatically generated.
> Add support for controlling capabilities for docker containers
> --
>
> Key: YARN-4258
> URL: https://issues.apache.org/jira/browse/YARN-4258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-4258.001.patch, YARN-4258.002.patch,
> YARN-4258.003.patch
>
>
> Docker allows us to specify which capabilities a container is assigned at
> launch time. We need a mechanism (admin controlled) that will allow us to
> define this set of capabilities that docker containers are launched with.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4258) Add support for controlling capabilities for docker containers
[ https://issues.apache.org/jira/browse/YARN-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4258: Attachment: YARN-4258.003.patch > Add support for controlling capabilities for docker containers > -- > > Key: YARN-4258 > URL: https://issues.apache.org/jira/browse/YARN-4258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4258.001.patch, YARN-4258.002.patch, > YARN-4258.003.patch > > > Docker allows us to specify which capabilities a container is assigned at > launch time. We need a mechanism (admin controlled) that will allow us to > define this set of capabilities that docker containers are launched with. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4256) YARN fair scheduler vcores with decimal values
[
https://issues.apache.org/jira/browse/YARN-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956594#comment-14956594
]
Jun Gong commented on YARN-4256:
Thanks [~Prabhu Joseph] for reporting. How about considering the value before
decimal? If OK, I will attach a patch.
> YARN fair scheduler vcores with decimal values
> --
>
> Key: YARN-4256
> URL: https://issues.apache.org/jira/browse/YARN-4256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Prabhu Joseph
>Priority: Minor
> Fix For: 2.7.2
>
>
> When the queue with vcores is in decimal value, the value after the decimal
> point is taken as vcores by FairScheduler.
> For the below queue,
> 2 mb,20 vcores,20.25 disks
> 3 mb,40.2 vcores,30.25 disks
> When many applications submitted parallely into queue, all were in PENDING
> state as the vcores is taken as 2 skipping the value 40.
> The code FairSchedulerConfiguration.java to Pattern match the vcores has to
> be improved in such a way either throw
> AllocationConfigurationException("Missing resource") or consider the value
> before decimal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956597#comment-14956597 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-trunk-Commit #8629 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8629/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/CHANGES.txt > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956416#comment-14956416
]
Hadoop QA commented on YARN-4250:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 57s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 68m 48s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 113m 44s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766472/YARN-4250-004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 40cac59 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9435/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9435/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9435/console |
This message was automatically generated.
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[
https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957666#comment-14957666
]
Hadoop QA commented on YARN-3216:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 8s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 10m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 14m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 48s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 41s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 11s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 2m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 51s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 61m 45s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 115m 16s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766591/0007-YARN-3216.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 56dc777 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9446/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9446/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9446/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9446/console |
This message was automatically generated.
> Max-AM-Resource-Percentage should respect node labels
> -
>
> Key: YARN-3216
> URL: https://issues.apache.org/jira/browse/YARN-3216
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Critical
> Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch,
> 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch,
> 0006-YARN-3216.patch, 0007-YARN-3216.patch
>
>
> Currently, max-am-resource-percentage considers default_partition only. When
> a queue can access multiple partitions, we should be able to compute
> max-am-resource-percentage based on that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4243) Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit
[
https://issues.apache.org/jira/browse/YARN-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957530#comment-14957530
]
Hadoop QA commented on YARN-4243:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 53s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 20s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 8s | The applied patch generated 2
new checkstyle issues (total was 211, now 212). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 8m 24s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 57m 23s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 121m 33s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766368/YARN-4243.3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0d77e85 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9445/console |
This message was automatically generated.
> Add retry on establishing Zookeeper conenction in
> EmbeddedElectorService#serviceInit
>
>
> Key: YARN-4243
> URL: https://issues.apache.org/jira/browse/YARN-4243
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-4243.1.patch, YARN-4243.2.1.patch,
> YARN-4243.2.patch, YARN-4243.3.patch
>
>
> Right now, the RM would shut down if the zk connection is down when the RM do
> the initialization. We need to add retry on this part
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[
https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957617#comment-14957617
]
Naganarasimha G R commented on YARN-2729:
-
Thanks for the comments [~rohithsharma],
bq. 1.NodeLabelsProvider is abstract class. Why it was not made as Interface?
Good point i think making it as interface should be sufficient and in
NodeManager we can do {{addIfService(nodeLabelsProvider)}}, this will will give
the implementations of the NodeLabelsProvider to be much more flexible. will
update current patch to incorporate these modifications shortly
bq. 2. And for each NodeLabelsProvide implementing any other way in future
which require modification in NodeManager to create an instance of
NodeLabelsProvider. Instead of that loading implemented class would be easier
and also it is pluggable.
I presume you might of got it wrong over configuration of NodeLabelsProvider in
NM. Basically there is whitelist of script and config and if this doesnt match
then we are assuming its a qualified classname and try load it as you
mentioned. Hence still its pluggable. correct me if i am wrong in understanding
your comment here
bq. From the below description, I assume that NodeLabelsProvider instance can
be created only if configuration-type is distributed. And I think before
creating an instance for provider, there should be check for configuration-type
also.
This was a conscious decision which was taken to minimize the number of
configurations on the NM side. It was sufficient to configure provider only as
in NM, if configuration-type is configured and provider is not configured then
we need to throw exception. {{configuration-type}} is mainly for the RM to deal
with NM's HB/registrations and blocking CLI and REST interactions.
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup
> ---
>
> Key: YARN-2729
> URL: https://issues.apache.org/jira/browse/YARN-2729
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch,
> YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch,
> YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch,
> YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch,
> YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch,
> YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch,
> YARN-2729.20150925-1.patch, YARN-2729.20151310-1.patch,
> YARN-2729.20151310-2.patch
>
>
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957635#comment-14957635 ] Allen Wittenauer commented on YARN-4262: But admin also exposes functionality on the RM. bq. should we expose such functionality to anybody who is not in the 'admin' role for the cluster? No, which is why it should be a separate list. This isn't an "either/or". You need three lists: regular users, users who can run docker in priv mode, and admin level privs. This is particular relevant when you think about OSes that aren't Linux that support Docker container formats but do support roles... > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957535#comment-14957535 ] Sidharta Seethana commented on YARN-4262: - Hi [~aw], I did consider using a separate list. Running a privileged container in some ways provides the equivalent of superuser access to the underlying node. So, the question here would be : should we expose such functionality to anybody who is not in the 'admin' role for the cluster? Thoughts? thanks, -Sidharta > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
Li Lu created YARN-4265: --- Summary: Provide new timeline plugin storage to support fine-grained entity caching Key: YARN-4265 URL: https://issues.apache.org/jira/browse/YARN-4265 Project: Hadoop YARN Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu To support the newly proposed APIs in YARN-4234, we need to create a new plugin timeline store. The store may have similar behavior as the EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id granularity, instead of application id granularity. Let's have this storage as a standalone one, instead of updating EntityFileTimelineStore, to keep the existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4234) New put APIs in TimelineClient for ats v1.5
[ https://issues.apache.org/jira/browse/YARN-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4234: Attachment: YARN-4234.2.patch > New put APIs in TimelineClient for ats v1.5 > --- > > Key: YARN-4234 > URL: https://issues.apache.org/jira/browse/YARN-4234 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4234.1.patch, YARN-4234.2.patch > > > In this ticket, we will add new put APIs in timelineClient to let > clients/applications have the option to use ATS v1.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4243) Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit
[ https://issues.apache.org/jira/browse/YARN-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957903#comment-14957903 ] Xuan Gong commented on YARN-4243: - [~kasha] Could you review the latest patch, please ? > Add retry on establishing Zookeeper conenction in > EmbeddedElectorService#serviceInit > > > Key: YARN-4243 > URL: https://issues.apache.org/jira/browse/YARN-4243 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4243.1.patch, YARN-4243.2.1.patch, > YARN-4243.2.patch, YARN-4243.3.patch > > > Right now, the RM would shut down if the zk connection is down when the RM do > the initialization. We need to add retry on this part -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4219: Attachment: YARN-4219-trunk.001.patch Let me quickly flip the patch to address some findbugs warnings. > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4219-trunk.001.patch > > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4219: Attachment: YARN-4219-trunk.001.patch Upload a patch to add the leveldb cache storage. To organize this storage, I refactored MemoryTimelineStore, moved the abstract logic to operate on a general map of entities into a new MapTimelineStore. Both MemoryTimelineStore and LevelDbCacheTimelineStore are MapTimelineStores, communicating through the abstract logic through a TimelineStoreMapAdapter. > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4219-trunk.001.patch > > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4162) Scheduler info in REST, is currently not displaying partition specific queue information similar to UI
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4162: Attachment: YARN-4162.v2.005.patch Hi [~wangda] I am attaching a patch with fixing following comments bq. In the webui i will keep it as it is i.e. but in REST will keep it blank. bq. Could you update ResourceUsageInfo to make only user and leaf-queue have am-used-resource? Also have verified the same in local cluster. > Scheduler info in REST, is currently not displaying partition specific queue > information similar to UI > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4219: Attachment: (was: YARN-4219-trunk.001.patch) > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[ https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957904#comment-14957904 ] Jian He commented on YARN-4000: --- Forgot about my previous comment: bq. actually, I think this will be a problem in regular case. Consider this scenario : 1) application is recovered and added into scheduler, some slow NM has not re-registered back, so those containers are not yet recovered. 2) User kills this app 3) CapacityScheduler#doneApplicationAttempt is called, containers tracked by RM so far are killed. Note that CapacityScheduler#doneApplication is not called, so scheduler still has the SchedulerApplication in memory 4) Slow NM now re-registers and try to recover the containers. These containers will be recovered even though application is in the process of being killed. These container will not be killed later on. Hence, these containers are leaked. > RM crashes with NPE if leaf queue becomes parent queue during restart > - > > Key: YARN-4000 > URL: https://issues.apache.org/jira/browse/YARN-4000 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-4000.01.patch, YARN-4000.02.patch, > YARN-4000.03.patch, YARN-4000.04.patch, YARN-4000.05.patch, YARN-4000.06.patch > > > This is a similar situation to YARN-2308. If an application is active in > queue A and then the RM restarts with a changed capacity scheduler > configuration where queue A becomes a parent queue to other subqueues then > the RM will crash with a NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4065) container-executor error should include effective user id
[
https://issues.apache.org/jira/browse/YARN-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957973#comment-14957973
]
Casey Brotherton commented on YARN-4065:
Have worked on this, in the middle of testing, I realized that the ordering of
the checks was wrong.
The code checks if the program is owned by root, and setuid after trying to
access the configuration file.
I am changing the ordering, and will still add an error message, however, it
likely doesn't need to print the effective user-id.
> container-executor error should include effective user id
> -
>
> Key: YARN-4065
> URL: https://issues.apache.org/jira/browse/YARN-4065
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Casey Brotherton
>Assignee: Casey Brotherton
>Priority: Trivial
>
> When container-executor fails to access it's config file, the following
> message will be thrown:
> {code}
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container executor initialization is : 24
> ExitCodeException exitCode=24: Invalid conf file provided :
> /etc/hadoop/conf/container-executor.cfg
> {code}
> The real problem may be a change in the container-executor not running as set
> uid root.
> From:
> https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
> {quote}
> The container-executor program must be owned by root and have the permission
> set ---sr-s---.
> {quote}
> The error message could be improved by printing out the effective user id
> with the error message, and possibly the executable trying to access the
> config file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956628#comment-14956628 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-trunk-Commit #8630 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8630/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/CHANGES.txt > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956627#comment-14956627 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-trunk-Commit #8630 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8630/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * hadoop-yarn-project/CHANGES.txt > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956689#comment-14956689 ] Sidharta Seethana commented on YARN-4252: - Thanks! > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956690#comment-14956690 ] Sidharta Seethana commented on YARN-4253: - Thanks! > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956746#comment-14956746 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/CHANGES.txt > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956745#comment-14956745 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * hadoop-yarn-project/CHANGES.txt > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956747#comment-14956747 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1263 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1263/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956674#comment-14956674 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * hadoop-yarn-project/CHANGES.txt > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956676#comment-14956676 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4258) Add support for controlling capabilities for docker containers
[ https://issues.apache.org/jira/browse/YARN-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4258: Attachment: YARN-4258.004.patch > Add support for controlling capabilities for docker containers > -- > > Key: YARN-4258 > URL: https://issues.apache.org/jira/browse/YARN-4258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4258.001.patch, YARN-4258.002.patch, > YARN-4258.003.patch, YARN-4258.004.patch > > > Docker allows us to specify which capabilities a container is assigned at > launch time. We need a mechanism (admin controlled) that will allow us to > define this set of capabilities that docker containers are launched with. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4256) YARN fair scheduler vcores with decimal values
[
https://issues.apache.org/jira/browse/YARN-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong reassigned YARN-4256:
--
Assignee: Jun Gong
> YARN fair scheduler vcores with decimal values
> --
>
> Key: YARN-4256
> URL: https://issues.apache.org/jira/browse/YARN-4256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Prabhu Joseph
>Assignee: Jun Gong
>Priority: Minor
> Fix For: 2.7.2
>
> Attachments: YARN-4256.001.patch
>
>
> When the queue with vcores is in decimal value, the value after the decimal
> point is taken as vcores by FairScheduler.
> For the below queue,
> 2 mb,20 vcores,20.25 disks
> 3 mb,40.2 vcores,30.25 disks
> When many applications submitted parallely into queue, all were in PENDING
> state as the vcores is taken as 2 skipping the value 40.
> The code FairSchedulerConfiguration.java to Pattern match the vcores has to
> be improved in such a way either throw
> AllocationConfigurationException("Missing resource") or consider the value
> before decimal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4256) YARN fair scheduler vcores with decimal values
[
https://issues.apache.org/jira/browse/YARN-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-4256:
---
Attachment: YARN-4256.001.patch
> YARN fair scheduler vcores with decimal values
> --
>
> Key: YARN-4256
> URL: https://issues.apache.org/jira/browse/YARN-4256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Prabhu Joseph
>Priority: Minor
> Fix For: 2.7.2
>
> Attachments: YARN-4256.001.patch
>
>
> When the queue with vcores is in decimal value, the value after the decimal
> point is taken as vcores by FairScheduler.
> For the below queue,
> 2 mb,20 vcores,20.25 disks
> 3 mb,40.2 vcores,30.25 disks
> When many applications submitted parallely into queue, all were in PENDING
> state as the vcores is taken as 2 skipping the value 40.
> The code FairSchedulerConfiguration.java to Pattern match the vcores has to
> be improved in such a way either throw
> AllocationConfigurationException("Missing resource") or consider the value
> before decimal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4250:
Affects Version/s: 2.8.0
Priority: Major (was: Blocker)
Component/s: scheduler
resourcemanager
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956691#comment-14956691 ] Sidharta Seethana commented on YARN-4255: - Thanks! > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956697#comment-14956697
]
Hudson commented on YARN-4250:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8631 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8631/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956728#comment-14956728 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4262) Allow admins to run privileged docker containers.
Sidharta Seethana created YARN-4262: --- Summary: Allow admins to run privileged docker containers. Key: YARN-4262 URL: https://issues.apache.org/jira/browse/YARN-4262 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sidharta Seethana Assignee: Sidharta Seethana There are scenarios where privileged containers are necessary in order to run certain kinds of applications (one example is trying to run postresql/oracle inside containers). However, given the security implications, we should ensure that : 1) privileged containers are disabled by default, even for admins 2) if enabled, only admins should be allowed to launch such containers and 3) Not all containers launched by admin users need to be privileged containers : admin users need to explicitly request that a privileged container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4223) Findbugs warnings in hadoop-yarn-server-nodemanager project
[
https://issues.apache.org/jira/browse/YARN-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956718#comment-14956718
]
Varun Saxena commented on YARN-4223:
[~rohithsharma], kindly review
> Findbugs warnings in hadoop-yarn-server-nodemanager project
> ---
>
> Key: YARN-4223
> URL: https://issues.apache.org/jira/browse/YARN-4223
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: FindBugs Report.html, YARN-4223.01.patch
>
>
> {noformat}
> classname='org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher'>
>message='Unchecked/unconfirmed cast from
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEvent
> to
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.SignalContainersLauncherEvent
> in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher.handle(ContainersLauncherEvent)'
> lineNumber='146'/>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956727#comment-14956727 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956729#comment-14956729 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2475 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2475/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/CHANGES.txt > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4219) New levelDB cache storage for timeline v1.5
[
https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958083#comment-14958083
]
Hadoop QA commented on YARN-4219:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 8m 10s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 26s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 19s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 30s | The applied patch generated
15 new checkstyle issues (total was 59, now 74). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 2s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 3m 31s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| | | 42m 47s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766656/YARN-4219-trunk.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / be7a0ad |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/artifact/patchprocess/diffJavacWarnings.txt
|
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/artifact/patchprocess/diffcheckstylehadoop-yarn-server-applicationhistoryservice.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9447/console |
This message was automatically generated.
> New levelDB cache storage for timeline v1.5
> ---
>
> Key: YARN-4219
> URL: https://issues.apache.org/jira/browse/YARN-4219
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-4219-trunk.001.patch
>
>
> We need to have an "offline" caching storage for timeline server v1.5 after
> the changes in YARN-3942. The in memory timeline storage may run into OOM
> issues when used as a cache storage for entity file timeline storage. We can
> refactor the code and have a level db based caching storage for this use
> case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4237) Support additional queries for ATSv2 Web UI
[ https://issues.apache.org/jira/browse/YARN-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958117#comment-14958117 ] Li Lu commented on YARN-4237: - Sorry for the delay folks. Patch LGTM. I tried it locally and it behaves correctly on the web UI related functions. +1. Will commit tomorrow morning if no objections. > Support additional queries for ATSv2 Web UI > --- > > Key: YARN-4237 > URL: https://issues.apache.org/jira/browse/YARN-4237 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-4237-YARN-2928.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4264) In-VM test ATS instances fail with metrics already registered
[
https://issues.apache.org/jira/browse/YARN-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957988#comment-14957988
]
Steve Loughran commented on YARN-4264:
--
stack
{code}
Exception encountered when invoking run on a nested suite - Metrics source
TimelineDataManagerMetrics already exists! *** ABORTED ***
org.apache.hadoop.metrics2.MetricsException: Metrics source
TimelineDataManagerMetrics already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:143)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:120)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManagerMetrics.create(TimelineDataManagerMetrics.java:100)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.serviceInit(TimelineDataManager.java:74)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
...
{code}
> In-VM test ATS instances fail with metrics already registered
> -
>
> Key: YARN-4264
> URL: https://issues.apache.org/jira/browse/YARN-4264
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> Testing my SPARK-1537 code against branch-2 triggers stack traces due to
> failed attempts to re-register metrics. This is with code which works against
> 2.7.1, so it's a regression.
> Either the timeline server needs to unregister its metrics on shutdown, or
> ATS adds an option to disable metrics for test purposes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4237) Support additional queries for ATSv2 Web UI
[ https://issues.apache.org/jira/browse/YARN-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958114#comment-14958114 ] Sangjin Lee commented on YARN-4237: --- The patch looks good to me. Others? > Support additional queries for ATSv2 Web UI > --- > > Key: YARN-4237 > URL: https://issues.apache.org/jira/browse/YARN-4237 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-4237-YARN-2928.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4256) YARN fair scheduler vcores with decimal values
[
https://issues.apache.org/jira/browse/YARN-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956833#comment-14956833
]
Hadoop QA commented on YARN-4256:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 47s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 55s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 63m 3s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 108m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d6c8bad |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9440/console |
This message was automatically generated.
> YARN fair scheduler vcores with decimal values
> --
>
> Key: YARN-4256
> URL: https://issues.apache.org/jira/browse/YARN-4256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Prabhu Joseph
>Assignee: Jun Gong
>Priority: Minor
> Fix For: 2.7.2
>
> Attachments: YARN-4256.001.patch
>
>
> When the queue with vcores is in decimal value, the value after the decimal
> point is taken as vcores by FairScheduler.
> For the below queue,
> 2 mb,20 vcores,20.25 disks
> 3 mb,40.2 vcores,30.25 disks
> When many applications submitted parallely into queue, all were in PENDING
> state as the vcores is taken as 2 skipping the value 40.
> The code FairSchedulerConfiguration.java to Pattern match the vcores has to
> be improved in such a way either throw
> AllocationConfigurationException("Missing resource") or consider the value
> before decimal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4263) Capacity scheduler 60%-40% formatting floating point issue
Adrian Kalaszi created YARN-4263:
Summary: Capacity scheduler 60%-40% formatting floating point issue
Key: YARN-4263
URL: https://issues.apache.org/jira/browse/YARN-4263
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.7.1
Reporter: Adrian Kalaszi
Priority: Trivial
If capacity scheduler is set with two queues to 60% and 40% capacity, due to a
java float floating representation issue
{code}
> hadoop queue -list
==
Queue Name : default
Queue State : running
Scheduling Info : Capacity: 40.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0
==
Queue Name : large
Queue State : running
Scheduling Info : Capacity: 60.04, MaximumCapacity: 100.0, CurrentCapacity:
0.0
{code}
Because
{code} System.err.println((0.6f) * 100); {code}
results in 60.04.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956958#comment-14956958 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/CHANGES.txt > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956957#comment-14956957 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * hadoop-yarn-project/CHANGES.txt > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956959#comment-14956959 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #494 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/494/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[ https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956963#comment-14956963 ] Varun Saxena commented on YARN-4000: [~jianhe], Discussed offline with Rohith. Diagnostic information is needed by him for another JIRA he is working on.. So if YARN-4000 is good enough to go in, I guess I can simply rebase this patch and let it go in. And we can close YARN-4111 as duplicate. > RM crashes with NPE if leaf queue becomes parent queue during restart > - > > Key: YARN-4000 > URL: https://issues.apache.org/jira/browse/YARN-4000 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Varun Saxena > Attachments: YARN-4000.01.patch, YARN-4000.02.patch, > YARN-4000.03.patch, YARN-4000.04.patch, YARN-4000.05.patch > > > This is a similar situation to YARN-2308. If an application is active in > queue A and then the RM restarts with a changed capacity scheduler > configuration where queue A becomes a parent queue to other subqueues then > the RM will crash with a NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956661#comment-14956661 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4252) Log container-executor invocation details when exit code is non-zero
[ https://issues.apache.org/jira/browse/YARN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956663#comment-14956663 ] Hudson commented on YARN-4252: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/]) YARN-4252. Log container-executor invocation details when exit code is (vvasudev: rev fb9c1519b30cf8113cf9b8deed9b9901db5e9693) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java > Log container-executor invocation details when exit code is non-zero > > > Key: YARN-4252 > URL: https://issues.apache.org/jira/browse/YARN-4252 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4252.001.patch > > > It would be useful for debugging/troubleshooting purposes to know the > invocation parameters for container-executor (used in LinuxContainerExecutor) > if there is a failures. These invocation parameters should be logged in the > NM logs at WARN/ERROR level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956662#comment-14956662 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #527 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/527/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4261) fix the order of timelinereader in yarn/yarn.cmd
[ https://issues.apache.org/jira/browse/YARN-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-4261: -- Assignee: Varun Saxena > fix the order of timelinereader in yarn/yarn.cmd > > > Key: YARN-4261 > URL: https://issues.apache.org/jira/browse/YARN-4261 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Trivial > > The order of the timelinereader command is not correct in yarn/yarn.cmd. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956675#comment-14956675 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #539 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/539/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * hadoop-yarn-project/CHANGES.txt > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[
https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958283#comment-14958283
]
Shiwei Guo commented on YARN-3448:
--
[~jeagles], I noticed that in patch.12, have following changes compared to
patch.10:
1.
{code:title=RollingLevelDBTimelineStore#putEntities}
// write entity marker
// ... ignore code to prepare markerKey
byte[] markerValue = writeReverseOrderedLong(startAndInsertTime
.insertTime);
writeBatch.put(markerKey, markerValue);
{code}
is changed to
{code}
// write entity marker
// ... ignore code to prepare markerKey
writeBatch.put(markerKey, EMPTY_BYTES);
{code}
and 2.
{code:title=TestRollingLevelDBTimelineStore.java}
@Test
public void testGetEntitiesWithFromTs() throws IOException {
super.testGetEntitiesWithFromTs();
}
{code}
is changed to
{code}
@Test
public void testGetEntitiesWithFromTs() throws IOException {
// feature not supported
}
{code}
What's the reason to make this change? Cause I found that if we keep the
patch.10 version of these code, plus a small bug fix code(as following code),
the testGetEntitiesWithFromTs test case can pass, so we can support the
GetEntitiesWithFromTs feature. Maybe I'm missing some other things.
BUGFIX code in getEntityByTime
{code:title= RollingLevelDBTimelineStore#getEntityByTime}
if (fromTs != null) {
long insertTime = readReverseOrderedLong(iterator.peekNext()
.getValue(), 0);
if (insertTime > fromTs) {
byte[] firstKey = key;
while (iterator.hasNext()) {
key = iterator.peekNext().getKey();
// BUGFIX code block
iterator.next();
if (!prefixMatches(firstKey, kp.getOffset(), key)) {
break;
}
}
continue;
}
}
{code}
change to
{code}
if (fromTs != null) {
long insertTime = readReverseOrderedLong(iterator.peekNext()
.getValue(), 0);
if (insertTime > fromTs) {
byte[] firstKey = key;
while (iterator.hasNext()) {
key = iterator.peekNext().getKey();
// BUGFIX code block
if (!prefixMatches(firstKey, kp.getOffset(), key)) {
break;
}
iterator.next();
}
continue;
}
}
{code}
> Add Rolling Time To Lives Level DB Plugin Capabilities
> --
>
> Key: YARN-3448
> URL: https://issues.apache.org/jira/browse/YARN-3448
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Fix For: 2.8.0
>
> Attachments: YARN-3448.1.patch, YARN-3448.10.patch,
> YARN-3448.12.patch, YARN-3448.13.patch, YARN-3448.14.patch,
> YARN-3448.15.patch, YARN-3448.16.patch, YARN-3448.17.patch,
> YARN-3448.2.patch, YARN-3448.3.patch, YARN-3448.4.patch, YARN-3448.5.patch,
> YARN-3448.7.patch, YARN-3448.8.patch, YARN-3448.9.patch
>
>
> For large applications, the majority of the time in LeveldbTimelineStore is
> spent deleting old entities record at a time. An exclusive write lock is held
> during the entire deletion phase which in practice can be hours. If we are to
> relax some of the consistency constraints, other performance enhancing
> techniques can be employed to maximize the throughput and minimize locking
> time.
> Split the 5 sections of the leveldb database (domain, owner, start time,
> entity, index) into 5 separate databases. This allows each database to
> maximize the read cache effectiveness based on the unique usage patterns of
> each database. With 5 separate databases each lookup is much faster. This can
> also help with I/O to have the entity and index databases on separate disks.
> Rolling DBs for entity and index DBs. 99.9% of the data are in these two
> sections 4:1 ration (index to entity) at least for tez. We replace DB record
> removal with file system removal if we create a rolling set of databases that
> age out and can be efficiently removed. To do this we must place a constraint
> to always place an entity's events into it's correct rolling db instance
> based on start time. This allows us to stitching the data back together while
> reading and artificial paging.
> Relax the synchronous writes constraints. If we are willing to accept losing
> some records that we not flushed in the operating system during a crash, we
> can use async writes that can be much faster.
> Prefer Sequential writes. sequential writes can be several times faster than
> random writes. Spend some small effort arranging the writes in such a way
> that will trend
[jira] [Commented] (YARN-4000) RM crashes with NPE if leaf queue becomes parent queue during restart
[
https://issues.apache.org/jira/browse/YARN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957285#comment-14957285
]
Hadoop QA commented on YARN-4000:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 28s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 8m 12s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated 2
new checkstyle issues (total was 632, now 615). |
| {color:red}-1{color} | whitespace | 0m 29s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 63m 9s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 104m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766562/YARN-4000.06.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d6c8bad |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9444/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9444/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9444/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9444/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9444/console |
This message was automatically generated.
> RM crashes with NPE if leaf queue becomes parent queue during restart
> -
>
> Key: YARN-4000
> URL: https://issues.apache.org/jira/browse/YARN-4000
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Varun Saxena
> Attachments: YARN-4000.01.patch, YARN-4000.02.patch,
> YARN-4000.03.patch, YARN-4000.04.patch, YARN-4000.05.patch, YARN-4000.06.patch
>
>
> This is a similar situation to YARN-2308. If an application is active in
> queue A and then the RM restarts with a changed capacity scheduler
> configuration where queue A becomes a parent queue to other subqueues then
> the RM will crash with a NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956919#comment-14956919
]
Hudson commented on YARN-4250:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #528 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/528/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956911#comment-14956911
]
Hudson commented on YARN-4250:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #1264 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1264/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4162) Scheduler info in REST, is currently not displaying partition specific queue information similar to UI
[
https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958165#comment-14958165
]
Hadoop QA commented on YARN-4162:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 8s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 50s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 18s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 28s | The applied patch generated 1
new checkstyle issues (total was 2, now 3). |
| {color:red}-1{color} | checkstyle | 1m 52s | The applied patch generated
29 new checkstyle issues (total was 222, now 247). |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 13s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 57m 37s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 103m 45s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766643/YARN-4162.v2.005.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / be7a0ad |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
https://builds.apache.org/job/PreCommit-YARN-Build/9448/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9448/console |
This message was automatically generated.
> Scheduler info in REST, is currently not displaying partition specific queue
> information similar to UI
> --
>
> Key: YARN-4162
> URL: https://issues.apache.org/jira/browse/YARN-4162
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch,
> YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch,
> YARN-4162.v2.005.patch, restAndJsonOutput.zip
>
>
> When Node Labels are enabled then REST Scheduler Information should also
> provide partition specific queue information similar to the existing Web UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956987#comment-14956987
]
Hudson commented on YARN-4250:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2476 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2476/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4250) NPE in AppSchedulingInfo#isRequestLabelChanged
[
https://issues.apache.org/jira/browse/YARN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956999#comment-14956999
]
Hudson commented on YARN-4250:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/])
YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy
(rohithsharmaks: rev d6c8bad86964dbad3cc810914f786c7c477a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt
> NPE in AppSchedulingInfo#isRequestLabelChanged
> --
>
> Key: YARN-4250
> URL: https://issues.apache.org/jira/browse/YARN-4250
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.8.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-4250-002.patch, YARN-4250-003.patch,
> YARN-4250-004.patch, YARN-4250.patch
>
>
> *Trace*
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.isRequestLabelChanged(AppSchedulingInfo.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:342)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.allocate(FifoScheduler.java:350)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart$MyFifoScheduler.allocate(TestAMRMClientOnRMRestart.java:544)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:507)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart.testAMRMClientResendsRequestsOnRMRestart(TestAMRMClientOnRMRestart.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4253) Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957001#comment-14957001 ] Hudson commented on YARN-4253: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/]) YARN-4253. Standardize on using PrivilegedOperationExecutor for all (vvasudev: rev 8d5929308998869933982bd425102c3a9488ee8f) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java > Standardize on using PrivilegedOperationExecutor for all invocations of > container-executor in LinuxContainerExecutor > > > Key: YARN-4253 > URL: https://issues.apache.org/jira/browse/YARN-4253 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0 > > Attachments: YARN-4253-branch-2.001.patch, YARN-4253.001.patch > > > YARN-3443 introduced PrivilegedOperationExecutor and PrivilegedOperation(s) > which are meant to wrap invocations to the container-executor binary. > However, not all invocations of container-executor in LinuxContainerExecutor > use the PrivilegedOperationExecutor. We should change all such invocations to > use PrivilegedOperationExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4255) container-executor does not clean up docker operation command files.
[ https://issues.apache.org/jira/browse/YARN-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957000#comment-14957000 ] Hudson commented on YARN-4255: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2432 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2432/]) YARN-4255. container-executor does not clean up docker operation command (vvasudev: rev da1ee078f9d3c2c25c51d0b392b0925821c42ad3) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c > container-executor does not clean up docker operation command files. > - > > Key: YARN-4255 > URL: https://issues.apache.org/jira/browse/YARN-4255 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4255.001.patch > > > container-executor leaves behind docker command files that are used to run > docker commands. These need to be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4262: Attachment: YARN-4262.001.patch Uploading a patch to allow admins to run privileged containers. This patch has a dependency on YARN-4258 without which it will not compile. For the time being, I have included changes for YARN-4258 in this patch. I'll upload a new version once YARN-4258 completes its review cycle. [~vvasudev], could you please give this patch a look? Thank you. > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[ https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958362#comment-14958362 ] Shiwei Guo commented on YARN-3448: -- I may found the answer that the insertTime is removed in patch.12. Will it be better to throw a exception when fromTs != null, cause this feature is not supported yet? > Add Rolling Time To Lives Level DB Plugin Capabilities > -- > > Key: YARN-3448 > URL: https://issues.apache.org/jira/browse/YARN-3448 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.8.0 > > Attachments: YARN-3448.1.patch, YARN-3448.10.patch, > YARN-3448.12.patch, YARN-3448.13.patch, YARN-3448.14.patch, > YARN-3448.15.patch, YARN-3448.16.patch, YARN-3448.17.patch, > YARN-3448.2.patch, YARN-3448.3.patch, YARN-3448.4.patch, YARN-3448.5.patch, > YARN-3448.7.patch, YARN-3448.8.patch, YARN-3448.9.patch > > > For large applications, the majority of the time in LeveldbTimelineStore is > spent deleting old entities record at a time. An exclusive write lock is held > during the entire deletion phase which in practice can be hours. If we are to > relax some of the consistency constraints, other performance enhancing > techniques can be employed to maximize the throughput and minimize locking > time. > Split the 5 sections of the leveldb database (domain, owner, start time, > entity, index) into 5 separate databases. This allows each database to > maximize the read cache effectiveness based on the unique usage patterns of > each database. With 5 separate databases each lookup is much faster. This can > also help with I/O to have the entity and index databases on separate disks. > Rolling DBs for entity and index DBs. 99.9% of the data are in these two > sections 4:1 ration (index to entity) at least for tez. We replace DB record > removal with file system removal if we create a rolling set of databases that > age out and can be efficiently removed. To do this we must place a constraint > to always place an entity's events into it's correct rolling db instance > based on start time. This allows us to stitching the data back together while > reading and artificial paging. > Relax the synchronous writes constraints. If we are willing to accept losing > some records that we not flushed in the operating system during a crash, we > can use async writes that can be much faster. > Prefer Sequential writes. sequential writes can be several times faster than > random writes. Spend some small effort arranging the writes in such a way > that will trend towards sequential write performance over random write > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958364#comment-14958364 ] Sidharta Seethana commented on YARN-4262: - bq, should we expose such functionality to anybody who is not in the 'admin' role for the cluster? What I meant here is : if users A, B, C are admins in a cluster, should *any* users apart from A, B, C be allowed to run privileged containers? In other words, should the list for docker privileged mode be entirely orthogonal/unrelated to the admin list or should it be a subset? If I understand you correctly, you are suggesting that the list should be completely unrelated to the admin role. I see the value in separation of privileges but I thought tying this to the admin role might lead to this feature being used more carefully - hence the path chosen in the first version of the patch. I'll upload a new patch using a different list as you suggested. I'll update the description accordingly. > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default, even for admins > 2) if enabled, only admins should be allowed to launch such containers and > 3) Not all containers launched by admin users need to be privileged > containers : admin users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4262) Allow admins to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4262: Description: (Updated based on discussion in the JIRA) There are scenarios where privileged containers are necessary in order to run certain kinds of applications (one example is trying to run postresql/oracle inside containers). However, given the security implications, we should ensure that : 1) privileged containers are disabled by default 2) if enabled, only a whitelisted set of users should be allowed to launch such containers and 3) Not all containers launched by whitelisted users need to be privileged containers : whitelisted users need to explicitly request that a privileged container be launched. was: There are scenarios where privileged containers are necessary in order to run certain kinds of applications (one example is trying to run postresql/oracle inside containers). However, given the security implications, we should ensure that : 1) privileged containers are disabled by default, even for admins 2) if enabled, only admins should be allowed to launch such containers and 3) Not all containers launched by admin users need to be privileged containers : admin users need to explicitly request that a privileged container be launched. > Allow admins to run privileged docker containers. > -- > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch > > > (Updated based on discussion in the JIRA) > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default > 2) if enabled, only a whitelisted set of users should be allowed to launch > such containers and > 3) Not all containers launched by whitelisted users need to be privileged > containers : whitelisted users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
