[jira] [Commented] (YARN-1474) Make schedulers services
[
https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000914#comment-14000914
]
Hadoop QA commented on YARN-1474:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645405/YARN-1474.13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-tools/hadoop-sls
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3763//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3763//console
This message is automatically generated.
> Make schedulers services
>
>
> Key: YARN-1474
> URL: https://issues.apache.org/jira/browse/YARN-1474
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Sandy Ryza
>Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1474.1.patch, YARN-1474.10.patch,
> YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.13.patch,
> YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch, YARN-1474.5.patch,
> YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch, YARN-1474.9.patch
>
>
> Schedulers currently have a reinitialize but no start and stop. Fitting them
> into the YARN service model would make things more coherent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1993) Cross-site scripting vulnerability in TextView.java
[
https://issues.apache.org/jira/browse/YARN-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000915#comment-14000915
]
Hadoop QA commented on YARN-1993:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644792/YARN-1993.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3765//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3765//console
This message is automatically generated.
> Cross-site scripting vulnerability in TextView.java
> ---
>
> Key: YARN-1993
> URL: https://issues.apache.org/jira/browse/YARN-1993
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: Ted Yu
> Attachments: YARN-1993.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/TextView.java
> , method echo() e.g. :
> {code}
> for (Object s : args) {
> out.print(s);
> }
> {code}
> Printing s to an HTML page allows cross-site scripting, because it was not
> properly sanitized for context HTML attribute name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2027) YARN ignores host-specific resource requests
[
https://issues.apache.org/jira/browse/YARN-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000910#comment-14000910
]
Kam Kasravi commented on YARN-2027:
---
I tried this on a 5 node cluster in AWS EC2 instances. I set relaxLocality to
false. Hadoop-2.2. Capacity Scheudler. Did not work. I tried setting rack to
null. I tried allocating 3 container request as well as just 1 container
request.
3 container requests:
containers += new ContainerRequest(capability, broker.nodes, null, pri,
false)
containers += new ContainerRequest(capability, null,
Seq[String]("/default-rack").toArray[String], pri, false)
containers += new ContainerRequest(capability, null, null, pri, false)
1 container request:
containers += new ContainerRequest(capability, broker.nodes, null, pri,
false)
where broker.nodes is an array of one FQDM host. Did not work. I can try Hong
Zhiguo's distributed shell patch on the same AWS cluster and report findings.
I'll run run hadoop-2.4 instead of hadoop-2.2.
> YARN ignores host-specific resource requests
>
>
> Key: YARN-2027
> URL: https://issues.apache.org/jira/browse/YARN-2027
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.4.0
> Environment: RHEL 6.1
> YARN 2.4
>Reporter: Chris Riccomini
>
> YARN appears to be ignoring host-level ContainerRequests.
> I am creating a container request with code that pretty closely mirrors the
> DistributedShell code:
> {code}
> protected def requestContainers(memMb: Int, cpuCores: Int, containers: Int)
> {
> info("Requesting %d container(s) with %dmb of memory" format (containers,
> memMb))
> val capability = Records.newRecord(classOf[Resource])
> val priority = Records.newRecord(classOf[Priority])
> priority.setPriority(0)
> capability.setMemory(memMb)
> capability.setVirtualCores(cpuCores)
> // Specifying a host in the String[] host parameter here seems to do
> nothing. Setting relaxLocality to false also doesn't help.
> (0 until containers).foreach(idx => amClient.addContainerRequest(new
> ContainerRequest(capability, null, null, priority)))
> }
> {code}
> When I run this code with a specific host in the ContainerRequest, YARN does
> not honor the request. Instead, it puts the container on an arbitrary host.
> This appears to be true for both the FifoScheduler and the CapacityScheduler.
> Currently, we are running the CapacityScheduler with the following settings:
> {noformat}
>
>
> yarn.scheduler.capacity.maximum-applications
> 1
>
> Maximum number of applications that can be pending and running.
>
>
>
> yarn.scheduler.capacity.maximum-am-resource-percent
> 0.1
>
> Maximum percent of resources in the cluster which can be used to run
> application masters i.e. controls number of concurrent running
> applications.
>
>
>
> yarn.scheduler.capacity.resource-calculator
>
> org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
>
> The ResourceCalculator implementation to be used to compare
> Resources in the scheduler.
> The default i.e. DefaultResourceCalculator only uses Memory while
> DominantResourceCalculator uses dominant-resource to compare
> multi-dimensional resources such as Memory, CPU etc.
>
>
>
> yarn.scheduler.capacity.root.queues
> default
>
> The queues at the this level (root is the root queue).
>
>
>
> yarn.scheduler.capacity.root.default.capacity
> 100
> Samza queue target capacity.
>
>
> yarn.scheduler.capacity.root.default.user-limit-factor
> 1
>
> Default queue user limit a percentage from 0.0 to 1.0.
>
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> 100
>
> The maximum capacity of the default queue.
>
>
>
> yarn.scheduler.capacity.root.default.state
> RUNNING
>
> The state of the default queue. State can be one of RUNNING or STOPPED.
>
>
>
> yarn.scheduler.capacity.root.default.acl_submit_applications
> *
>
> The ACL of who can submit jobs to the default queue.
>
>
>
> yarn.scheduler.capacity.root.default.acl_administer_queue
> *
>
> The ACL of who can administer jobs on the default queue.
>
>
>
> yarn.scheduler.capacity.node-locality-delay
> 40
>
> Number of missed scheduling opportunities after which the
> CapacityScheduler
> attempts to schedule rack-local containers.
> Typically this should be set to number of nodes in the cluster, By
> default is setting
> approximately number of nodes in one ra
[jira] [Commented] (YARN-2071) Enforce more restricted permissions for the directory of Leveldb store
[
https://issues.apache.org/jira/browse/YARN-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000907#comment-14000907
]
Hadoop QA commented on YARN-2071:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645372/YARN-2071.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3764//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3764//console
This message is automatically generated.
> Enforce more restricted permissions for the directory of Leveldb store
> --
>
> Key: YARN-2071
> URL: https://issues.apache.org/jira/browse/YARN-2071
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Attachments: YARN-2071.1.patch
>
>
> We need to enforce more restricted permissions for the directory of Leveldb
> store, as w did for filesystem generic history store.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2068) FairScheduler uses the same ResourceCalculator for all policies
[ https://issues.apache.org/jira/browse/YARN-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-2068: - Assignee: Karthik Kambatla (was: Sandy Ryza) > FairScheduler uses the same ResourceCalculator for all policies > --- > > Key: YARN-2068 > URL: https://issues.apache.org/jira/browse/YARN-2068 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > FairScheduler uses the same ResourceCalculator for all policies including > DRF. Need to fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2068) FairScheduler uses the same ResourceCalculator for all policies
[ https://issues.apache.org/jira/browse/YARN-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000854#comment-14000854 ] Sandy Ryza commented on YARN-2068: -- I think this is not the case. The code is slightly confusing, because FairScheduler.java has a ResourceCalculator that it uses only for preemption. This is going to go away with YARN-596. The scheduling policies don't make use of this ResourceCalculator - the DominantResourceFairnessPolicy for example doesn't use a ResourceCalculator at all because it needs to save some intermediate results. > FairScheduler uses the same ResourceCalculator for all policies > --- > > Key: YARN-2068 > URL: https://issues.apache.org/jira/browse/YARN-2068 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > > FairScheduler uses the same ResourceCalculator for all policies including > DRF. Need to fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (YARN-2068) FairScheduler uses the same ResourceCalculator for all policies
[ https://issues.apache.org/jira/browse/YARN-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned YARN-2068: Assignee: Sandy Ryza (was: Karthik Kambatla) > FairScheduler uses the same ResourceCalculator for all policies > --- > > Key: YARN-2068 > URL: https://issues.apache.org/jira/browse/YARN-2068 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Sandy Ryza > > FairScheduler uses the same ResourceCalculator for all policies including > DRF. Need to fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1993) Cross-site scripting vulnerability in TextView.java
[
https://issues.apache.org/jira/browse/YARN-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000807#comment-14000807
]
Hadoop QA commented on YARN-1993:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644792/YARN-1993.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3762//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3762//console
This message is automatically generated.
> Cross-site scripting vulnerability in TextView.java
> ---
>
> Key: YARN-1993
> URL: https://issues.apache.org/jira/browse/YARN-1993
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: Ted Yu
> Attachments: YARN-1993.patch
>
>
> In
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/TextView.java
> , method echo() e.g. :
> {code}
> for (Object s : args) {
> out.print(s);
> }
> {code}
> Printing s to an HTML page allows cross-site scripting, because it was not
> properly sanitized for context HTML attribute name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2041) Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
[ https://issues.apache.org/jira/browse/YARN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000747#comment-14000747 ] Tsuyoshi OZAWA commented on YARN-2041: -- [~nravi], what scheduler are you using? If you use CapacityScheduler, YARN-893 can be related issue. > Hard to co-locate MR2 and Spark jobs on the same cluster in YARN > > > Key: YARN-2041 > URL: https://issues.apache.org/jira/browse/YARN-2041 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.3.0 >Reporter: Nishkam Ravi > > Performance of MR2 jobs falls drastically as YARN config parameter > yarn.nodemanager.resource.memory-mb is increased beyond a certain value. > Performance of Spark falls drastically as the value of > yarn.nodemanager.resource.memory-mb is decreased beyond a certain value for a > large data set. > This makes it hard to co-locate MR2 and Spark jobs in YARN. > The experiments are being conducted on a 6-node cluster. The following > workloads are being run: TeraGen, TeraSort, TeraValidate, WordCount, > ShuffleText and PageRank. > Will add more details to this JIRA over time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: YARN-1474.13.patch > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0, 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.10.patch, > YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.13.patch, > YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch, YARN-1474.5.patch, > YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: (was: YARN-1474.13.patch) > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0, 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.10.patch, > YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.2.patch, YARN-1474.3.patch, > YARN-1474.4.patch, YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch, > YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000738#comment-14000738 ] Tsuyoshi OZAWA commented on YARN-1474: -- [~kkambatl] thank you for the point. I agree with you to preserve backward compatibility. Updated a patch to address the problem. Can you check a latest patch? > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0, 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.10.patch, > YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.13.patch, > YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch, YARN-1474.5.patch, > YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1474) Make schedulers services
[ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-1474: - Attachment: YARN-1474.13.patch > Make schedulers services > > > Key: YARN-1474 > URL: https://issues.apache.org/jira/browse/YARN-1474 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.3.0, 2.4.0 >Reporter: Sandy Ryza >Assignee: Tsuyoshi OZAWA > Attachments: YARN-1474.1.patch, YARN-1474.10.patch, > YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.13.patch, > YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch, YARN-1474.5.patch, > YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch, YARN-1474.9.patch > > > Schedulers currently have a reinitialize but no start and stop. Fitting them > into the YARN service model would make things more coherent. -- This message was sent by Atlassian JIRA (v6.2#6252)
