[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15789079#comment-15789079
]
Hadoop QA commented on YARN-6041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 40 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
39s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
55s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
39s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
8s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
28s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 28s{color}
| {color:red} root-jdk1.8.0_111 with JDK v1.8.0_111 generated 1 new + 860
unchanged - 1 fixed = 861 total (was 861) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 19s{color} | {color:orange} root: The patch generated 18 new + 3470
unchanged - 42 fixed = 3488 total (was 3512) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (YARN-5838) windows - environement variables aren't accessible on Yarn 3.0 alpha-1
[
https://issues.apache.org/jira/browse/YARN-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15788415#comment-15788415
]
Chris Nauroth commented on YARN-5838:
-
Hello [~rekha.du...@gmail.com]. Are you possibly looking for the
{{yarn.nodemanager.admin-env}} setting in yarn-site.xml? Here is a copy-paste
of the default as defined in yarn-default.xml:
{code}
Environment variables that should be forwarded from the
NodeManager's environment to the container's.
yarn.nodemanager.admin-env
MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX
{code}
> windows - environement variables aren't accessible on Yarn 3.0 alpha-1
> --
>
> Key: YARN-5838
> URL: https://issues.apache.org/jira/browse/YARN-5838
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
> Environment: windows 7
>Reporter: Kanthirekha
>
> windows environment variables aren't accessible on Yarn 3.0 alpha-1
> tried fetching %Path% from Application master and on the container script
> (after a container is allocated by application master for task executions)
> echo %Path%
> result : is echo is on
> it returns blank .
> Could you please let us know what are the necessary steps to access env
> variables from yarn 3.0 aplha1 version ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5931) Document timeout interfaces CLI and REST APIs
[ https://issues.apache.org/jira/browse/YARN-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15788045#comment-15788045 ] Hudson commented on YARN-5931: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11059 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11059/]) YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by (templedf: rev 165d01a73e5f328108a9e876c7a751a3c2cec4a8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/RMAppLifetimeMonitor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationTimeout.java > Document timeout interfaces CLI and REST APIs > - > > Key: YARN-5931 > URL: https://issues.apache.org/jira/browse/YARN-5931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: ResourceManagerRest.html, YARN-5931.0.patch, > YARN-5931.1.patch, YARN-5931.2.patch, YARN-5931.3.patch, YARN-5931.4.patch, > YARN-5931.5.patch, YARN-5931.6.patch, YarnCommands.html > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6041: -- Attachment: YARN-6041-branch-2.002.patch Thanks for the review [~leftnoteasy] Uploading new patch: # Added the extra @Unstable annotations to ContainerState and YarnConfiguration you had requested. # Fixed some unused imports On the NM side, there really are no API changes pre se (except the addition of the SCHEDULED ContainerState). > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6046) Documentation correction in YarnApplicationSecurity
Bibin A Chundatt created YARN-6046:
--
Summary: Documentation correction in YarnApplicationSecurity
Key: YARN-6046
URL: https://issues.apache.org/jira/browse/YARN-6046
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Priority: Trivial
Few documentation correction required in
hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
{code}
1. Suring AM startup, log in to Kerberos.
{code}
{code}
Don’t. Rely on the lifespan of the
{code}
{code}
renewed automatically; the AM pushes out
{code}
{code}
In an insecure cluster, the application will run as the identity of the account
of the node manager, typically something such as yarn or mapred. By default,
the application will access HDFS as that user, with a different home directory,
and with a different user identified in audit logs and on file system owner
attributes.
{code}
Need to reframe sentence.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787934#comment-15787934
]
Arun Suresh edited comment on YARN-6041 at 12/30/16 4:17 PM:
-
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The whitespace errors are not actually errors, since they pertain to the
documentation markdown file.
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
was (Author: asuresh):
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
> Opportunistic containers : Combined patch for branch-2
> ---
>
> Key: YARN-6041
> URL: https://issues.apache.org/jira/browse/YARN-6041
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6041-branch-2.001.patch
>
>
> This is a combined patch targeting branch-2 of the following JIRAs which have
> already been committed to trunk :
> YARN-5938. Refactoring OpportunisticContainerAllocator to use
> SchedulerRequestKey instead of Priority and other misc fixes
> YARN-5646. Add documentation and update config parameter names for scheduling
> of OPPORTUNISTIC containers.
> YARN-5982. Simplify opportunistic container parameters and metrics.
> YARN-5918. Handle Opportunistic scheduling allocate request failure when NM
> is lost.
> YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager
> container lifecycle.
> YARN-5823. Update NMTokens in case of requests with only opportunistic
> containers.
> YARN-5377. Fix
> TestQueuingContainerManager.testKillMultipleOpportunisticContainers.
> YARN-2995. Enhance UI to show cluster resource utilization of various
> container Execution types.
> YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http
> Address.
> YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method
> to handle OPPORTUNISTIC container requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787934#comment-15787934
]
Arun Suresh commented on YARN-6041:
---
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
> Opportunistic containers : Combined patch for branch-2
> ---
>
> Key: YARN-6041
> URL: https://issues.apache.org/jira/browse/YARN-6041
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6041-branch-2.001.patch
>
>
> This is a combined patch targeting branch-2 of the following JIRAs which have
> already been committed to trunk :
> YARN-5938. Refactoring OpportunisticContainerAllocator to use
> SchedulerRequestKey instead of Priority and other misc fixes
> YARN-5646. Add documentation and update config parameter names for scheduling
> of OPPORTUNISTIC containers.
> YARN-5982. Simplify opportunistic container parameters and metrics.
> YARN-5918. Handle Opportunistic scheduling allocate request failure when NM
> is lost.
> YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager
> container lifecycle.
> YARN-5823. Update NMTokens in case of requests with only opportunistic
> containers.
> YARN-5377. Fix
> TestQueuingContainerManager.testKillMultipleOpportunisticContainers.
> YARN-2995. Enhance UI to show cluster resource utilization of various
> container Execution types.
> YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http
> Address.
> YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method
> to handle OPPORTUNISTIC container requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: TensorFlow_on_YARN_design.pdf > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > Attachments: TensorFlow_on_YARN_design.pdf > > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: (was: TensorFlow_on_YARN_proposal.pdf) > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong reassigned YARN-4090: -- Assignee: zhangshilong > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin >Assignee: zhangshilong > Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch, > YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, > YARN-4090.004.patch, sampling1.jpg, sampling2.jpg > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787353#comment-15787353 ] zhangshilong commented on YARN-4090: no problem, thank you very much for your patch, a great help to me. > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin > Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch, > YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, > YARN-4090.004.patch, sampling1.jpg, sampling2.jpg > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787286#comment-15787286 ] Xianyin Xin commented on YARN-4090: --- Hi [~zsl2007], sorry i have moved to another project, and don't have enough time. Change it to unassigned, and anyone who wants to take over is welcome. :) > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin > Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch, > YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, > YARN-4090.004.patch, sampling1.jpg, sampling2.jpg > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshilong updated YARN-4090: --- Attachment: YARN-4090.004.patch fix 2.6 deadlock in FSParentQueue.java > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin > Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch, > YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, > YARN-4090.004.patch, sampling1.jpg, sampling2.jpg > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-4090: -- Assignee: (was: Xianyin Xin) > Make Collections.sort() more efficient in FSParentQueue.java > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Xianyin Xin > Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch, > YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, sampling1.jpg, > sampling2.jpg > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient in FSParentQueue.java
[
https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787258#comment-15787258
]
zhangshilong commented on YARN-4090:
I see. In branch-2.6,
FSParentQueue.java: will lock FSQueue. But the code changes in version 2.7.1
{code:java}
@Override
public synchronized List getQueueUserAclInfo(
UserGroupInformation user) {
List userAcls = new ArrayList();
// Add queue acls
userAcls.add(getUserAclInfo(user));
// Add children queue acls
for (FSQueue child : childQueues) {
userAcls.addAll(child.getQueueUserAclInfo(user));
}
return userAcls;
}
{code}
> Make Collections.sort() more efficient in FSParentQueue.java
>
>
> Key: YARN-4090
> URL: https://issues.apache.org/jira/browse/YARN-4090
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Xianyin Xin
>Assignee: Xianyin Xin
> Attachments: YARN-4090-TestResult.pdf, YARN-4090-preview.patch,
> YARN-4090.001.patch, YARN-4090.002.patch, YARN-4090.003.patch, sampling1.jpg,
> sampling2.jpg
>
>
> Collections.sort() consumes too much time in a scheduling round.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787181#comment-15787181
]
Hadoop QA commented on YARN-6041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
50s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 40 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
34s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
52s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
25s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
59s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
28s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 28s{color}
| {color:red} root-jdk1.8.0_111 with JDK v1.8.0_111 generated 1 new + 860
unchanged - 1 fixed = 861 total (was 861) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 20s{color} | {color:orange} root: The patch generated 19 new + 3469
unchanged - 42 fixed = 3488 total (was 3511) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: TensorFlow_on_YARN_proposal.pdf Proposal attached. And thanks [~drankye], [~jingcheng.du], [~HuafengWang], [~mauzhang], [~zhongmuzhong] > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > Attachments: TensorFlow_on_YARN_proposal.pdf > > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
