[jira] [Commented] (YARN-7137) Resource profile: Move new added APIs to unstable
[ https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162569#comment-16162569 ] Sunil G commented on YARN-7137: --- +1 on latest patch. I ll commit this later today if no objections. > Resource profile: Move new added APIs to unstable > - > > Key: YARN-7137 > URL: https://issues.apache.org/jira/browse/YARN-7137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7137.YARN-3926.001.patch, > YARN-7137.YARN-3926.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7163) RMContext need not to be injected to webapp and other Always Running services.
[ https://issues.apache.org/jira/browse/YARN-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-7163: Attachment: YARN-7163-branch-2.addednum.patch I just observed in branch-2 patch that one more place I had missed to modify the change. I am updating addednum patch for branch-2. cc:/ [~sunilg] > RMContext need not to be injected to webapp and other Always Running services. > -- > > Key: YARN-7163 > URL: https://issues.apache.org/jira/browse/YARN-7163 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Blocker > Fix For: 2.9.0, 3.0.0-beta1, 3.1.0 > > Attachments: suspect-1.png, suspect-2.png, YARN-7163.01.patch, > YARN-7163.02.patch, YARN-7163.03.patch, YARN-7163.03.patch, > YARN-7163-branch-2.01.patch, YARN-7163-branch-2.addednum.patch > > > It is observed that RM crashes with heap space OOM in secure cluster(http > authentication is kerborse) when RM HA is enabled. > Scenario is > 1. Start RM in HA secure mode. Lets say RM1 is active mode. > 2. Run many applications so that it uses greater than 50% of heap space > configured. Lets say, if heap space is 2GB, then run applications that occupy > 1.5GB of heap space. > 3. Switch RM to StandBy and bring back to Active! While recovering > applications from state store, RM crashes with OOM. > *Note* : This issue will happen only when RM is started as ACTIVE directly. > (not switched from standby to active during start of JVM) > Heap dump shows that RMAuthenticationFilter holds 60% heap space! And other > 40% held by RMAppState which is during recovering from state store. This > exceeds the heap space and crashes with OOM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162518#comment-16162518
]
Yeliang Cang commented on YARN-7132:
Thanks [~haibochen] for the committing!
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yeliang Cang
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang reassigned YARN-7132:
--
Assignee: Yeliang Cang
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yeliang Cang
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3409) Support Node Attribute functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162494#comment-16162494 ] Naganarasimha G R commented on YARN-3409: - [~sunilg], As per our offline discussion, have recreated the branch "yarn-3409" and the latest patch seems to get applied compiled properly in the local env. > Support Node Attribute functionality > > > Key: YARN-3409 > URL: https://issues.apache.org/jira/browse/YARN-3409 > Project: Hadoop YARN > Issue Type: New Feature > Components: api, client, RM >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: 3409-apiChanges_v2.pdf (4).pdf, > Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch > > > Specify only one label for each node (IAW, partition a cluster) is a way to > determinate how resources of a special set of nodes could be shared by a > group of entities (like teams, departments, etc.). Partitions of a cluster > has following characteristics: > - Cluster divided to several disjoint sub clusters. > - ACL/priority can apply on partition (Only market team / marke team has > priority to use the partition). > - Percentage of capacities can apply on partition (Market team has 40% > minimum capacity and Dev team has 60% of minimum capacity of the partition). > Attributes are orthogonal to partition, they’re describing features of node’s > hardware/software just for affinity. Some example of attributes: > - glibc version > - JDK version > - Type of CPU (x86_64/i686) > - Type of OS (windows, linux, etc.) > With this, application can be able to ask for resource has (glibc.version >= > 2.20 && JDK.version >= 8u20 && x86_64). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162491#comment-16162491
]
Devaraj K commented on YARN-6620:
-
Thanks [~leftnoteasy] for the patch, Great work!
There are some comments on the patch.
1. XML file reading in GpuDeviceInformationParser.java, can we use the existing
libraries like javax.xml.bind.JAXBContext to unmarshall the XML document to a
Java Object instead of reading tag by tag?
2. If you don't agree to use the existing libraries for reading xml file, 'in'
stream may have to be closed after reading/parsing.
{code:xml}
InputStream in = IOUtils.toInputStream(sanitizeXmlInput(xmlStr), "UTF-8");
doc = builder.parse(in);
{code}
3. Instead of hardcoding the BINARY_NAME, can it be included as part of
DEFAULT_NM_GPU_PATH_TO_EXEC as a default value, so that it can be also becomes
configurable if incase users want to change it.
{code:xml}
public static final String DEFAULT_NM_GPU_PATH_TO_EXEC = "";
protected static final String BINARY_NAME = "nvidia-smi";
{code}
4. Please change the inline comment here accordingly.
{code:xml}
+ /**
+ * Disk as a resource is disabled by default.
+ **/
+ @Private
+ public static final boolean DEFAULT_NM_GPU_RESOURCE_ENABLED = false;
{code}
5. Can we use spaces instead of tab characters for indentation in
nvidia-smi-sample-output.xml?
6. Are we going to support multiple containers/processes(limited number)
sharing the same GPU device?
7.
{code:title=GpuResourceAllocator.java|borderStyle=solid}
for (int deviceNum : allowedGpuDevices) {
if (!usedDevices.containsKey(deviceNum)) {
usedDevices.put(deviceNum, containerId);
assignedGpus.add(deviceNum);
if (assignedGpus.size() == numRequestedGpuDevices) {
break;
}
}
}
// Record in state store if we allocated anything
if (!assignedGpus.isEmpty()) {
List allocatedDevices = new ArrayList<>();
for (int gpu : assignedGpus) {
allocatedDevices.add(String.valueOf(gpu));
}
{code}
Can you merge these two for loops into a one like below,
{code:xml}
usedDevices.put(deviceNum, containerId);
assignedGpus.add(deviceNum);
allocatedDevices.add(String.valueOf(deviceNum));
{code}
And also if the condition *if (assignedGpus.size() == numRequestedGpuDevices)*
doesn't meet, do we need to throw an exception or log the error?
8. I see that getGpuDeviceInformation() is getting invoked twice which intern
executes shell command and parses the xml file which are costly operations. Do
we need to execute it twice here?
{code:title=GpuResourceDiscoverPlugin.java|borderStyle=solid}
GpuDeviceInformation info = getGpuDeviceInformation();
LOG.info("Trying to discover GPU information ...");
GpuDeviceInformation info = getGpuDeviceInformation();
{code}
And also I don't convince that having the logic other than assigning conf in
setConf() method.
{code:xml}
public synchronized void setConf(Configuration conf) {
this.conf = conf;
numOfErrorExecutionSinceLastSucceed = 0;
featureEnabled = conf.getBoolean(YarnConfiguration.NM_GPU_RESOURCE_ENABLED,
YarnConfiguration.DEFAULT_NM_GPU_RESOURCE_ENABLED);
if (featureEnabled) {
String dir = conf.get(YarnConfiguration.NM_GPU_PATH_TO_EXEC,
.
{code}
And also there are Hadoop QA reported comments which needs to be fixed.
> [YARN-6223] NM Java side code changes to support isolate GPU devices by using
> CGroups
> -
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch,
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162471#comment-16162471
]
Hadoop QA commented on YARN-7185:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 97 unchanged - 0 fixed = 99 total (was 97) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 53s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7185 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886570/YARN-7185.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d40cf1ee9cb2 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fa53178 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/17411/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17411/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17411/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17411/testReport/ |
|
[jira] [Commented] (YARN-7173) Container update RM-NM communication fix for backward compatibility
[
https://issues.apache.org/jira/browse/YARN-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162472#comment-16162472
]
Hudson commented on YARN-7173:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12842 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12842/])
YARN-7173. Container update RM-NM communication fix for backward (wangda: rev
e74d1be04be47969943b0501a4f335b0b5188287)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeUpdateContainerEvent.java
> Container update RM-NM communication fix for backward compatibility
> ---
>
> Key: YARN-7173
> URL: https://issues.apache.org/jira/browse/YARN-7173
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-7173.001.patch, YARN-7173-branch-2.001.patch
>
>
> This is based on discussions with [~leftnoteasy] in YARN-6979.
> In YARN-6979, the {{getContainersToDecrease()}} and
> {{addAllContainersToDecrease()}} methods were removed from the
> NodeHeartbeatResponse (although the actual protobuf fields were still
> retained). We need to ensure that for clusters that upgrade from 2.8.x to
> 2.9.0, the decreased containers should also be sent to the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162460#comment-16162460
]
Hudson commented on YARN-7132:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12841 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12841/])
YARN-7132. FairScheduler.initScheduler() contains a surprising unary
(haibochen: rev fa531788fd7c9eee70808818a2f6aec9bdc3bec7)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162456#comment-16162456
]
Arun Suresh commented on YARN-7185:
---
Thanks for the patch [~wangda],
In general, it LGTM.
Minor nit though: Can you see if you can add the testcase to
{{TestContainerSchedulerQueuing}} ?
If the setup/teardown is not too complicated.
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch, YARN-7185.002.patch,
> YARN-7185.003.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162453#comment-16162453
]
Haibo Chen commented on YARN-7132:
--
Thanks [~Cyl] for your contribution. I have committed the patch to trunk and
branch-3.0
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7173) Container update RM-NM communication fix for backward compatibility
[
https://issues.apache.org/jira/browse/YARN-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7173:
-
Summary: Container update RM-NM communication fix for backward
compatibility (was: Container Update Backward compatibility fix for RM-NM
communication)
> Container update RM-NM communication fix for backward compatibility
> ---
>
> Key: YARN-7173
> URL: https://issues.apache.org/jira/browse/YARN-7173
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7173.001.patch, YARN-7173-branch-2.001.patch
>
>
> This is based on discussions with [~leftnoteasy] in YARN-6979.
> In YARN-6979, the {{getContainersToDecrease()}} and
> {{addAllContainersToDecrease()}} methods were removed from the
> NodeHeartbeatResponse (although the actual protobuf fields were still
> retained). We need to ensure that for clusters that upgrade from 2.8.x to
> 2.9.0, the decreased containers should also be sent to the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7173) Container Update Backward compatibility fix for RM-NM communication
[
https://issues.apache.org/jira/browse/YARN-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7173:
-
Summary: Container Update Backward compatibility fix for RM-NM
communication (was: Container Update Backward compatibility fix for upgrades)
> Container Update Backward compatibility fix for RM-NM communication
> ---
>
> Key: YARN-7173
> URL: https://issues.apache.org/jira/browse/YARN-7173
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7173.001.patch, YARN-7173-branch-2.001.patch
>
>
> This is based on discussions with [~leftnoteasy] in YARN-6979.
> In YARN-6979, the {{getContainersToDecrease()}} and
> {{addAllContainersToDecrease()}} methods were removed from the
> NodeHeartbeatResponse (although the actual protobuf fields were still
> retained). We need to ensure that for clusters that upgrade from 2.8.x to
> 2.9.0, the decreased containers should also be sent to the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Attachment: YARN-7185.003.patch
Attached ver.3 patch. Fixed issues reported by unit tests.
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch, YARN-7185.002.patch,
> YARN-7185.003.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162410#comment-16162410
]
Yeliang Cang commented on YARN-7132:
Thanks [~haibochen] for the review!
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162356#comment-16162356
]
Hadoop QA commented on YARN-7185:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 37s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7185 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886552/YARN-7185.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1b7a68c77348 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 661f5eb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/17410/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17410/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17410/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17410/testReport/ |
| mod
[jira] [Commented] (YARN-7186) Add examples in yarn-service
[
https://issues.apache.org/jira/browse/YARN-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162338#comment-16162338
]
Hadoop QA commented on YARN-7186:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
48s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 2s{color} | {color:orange} root: The patch generated 8 new + 87 unchanged -
4 fixed = 95 total (was 91) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
38s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 56s{color} |
{color:black} {color} |
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162318#comment-16162318 ] Zhankun Tang commented on YARN-6852: [~wangda], agree with you. I'll refactor both native and java side code. But still a concern is that how can we determine which resource should be treated as first-class citizen? I'm afraid a flexible way with minimal code changes to enable more hardware accelerator resource type is needed? For instance, just a .so for every device module? > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-beta1 > > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, > YARN-6852.009.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Attachment: YARN-7185.002.patch
Attached ver.2 patch.
[~asuresh] please review.
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch, YARN-7185.002.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
[
https://issues.apache.org/jira/browse/YARN-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162302#comment-16162302
]
Hadoop QA commented on YARN-6991:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| | Redundant nullcheck of callerUGI, which is known to be non-null in
org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block)
Redundant null check at AppBlock.java:is known to be non-null in
org.apache.hadoop.yarn.server.webapp.AppBlock.render(HtmlBlock$Block)
Redundant null check at AppBlock.java:[line 147] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6991 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886547/YARN-6991.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8eed4a43302f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 661f5eb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17409/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| findbugs |
http
[jira] [Updated] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
[ https://issues.apache.org/jira/browse/YARN-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-6991: --- Attachment: YARN-6991.002.patch Rebased with trunk > "Kill application" button does not show error if other user tries to kill the > application for secure cluster > > > Key: YARN-6991 > URL: https://issues.apache.org/jira/browse/YARN-6991 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Suma Shivaprasad > Attachments: YARN-6991.001.patch, YARN-6991.002.patch > > > 1. Submit an application by user 1 > 2. log into RM UI as user 2 > 3. Kill the application submitted by user 1 > 4. Even though application does not get killed, there is no error/info dialog > box being shown to let the user that "user doesnot have permissions to kill > application of other user" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
[
https://issues.apache.org/jira/browse/YARN-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162280#comment-16162280
]
Hadoop QA commented on YARN-6991:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6991 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6991 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886542/YARN-6991.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17408/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> "Kill application" button does not show error if other user tries to kill the
> application for secure cluster
>
>
> Key: YARN-6991
> URL: https://issues.apache.org/jira/browse/YARN-6991
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Suma Shivaprasad
> Attachments: YARN-6991.001.patch
>
>
> 1. Submit an application by user 1
> 2. log into RM UI as user 2
> 3. Kill the application submitted by user 1
> 4. Even though application does not get killed, there is no error/info dialog
> box being shown to let the user that "user doesnot have permissions to kill
> application of other user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7186) Add examples in yarn-service
[ https://issues.apache.org/jira/browse/YARN-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162274#comment-16162274 ] Jian He commented on YARN-7186: --- A new command "yarn service demo " is added for users to easily bring up pre-built example service > Add examples in yarn-service > - > > Key: YARN-7186 > URL: https://issues.apache.org/jira/browse/YARN-7186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7186.yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
[ https://issues.apache.org/jira/browse/YARN-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162270#comment-16162270 ] Suma Shivaprasad edited comment on YARN-6991 at 9/12/17 12:17 AM: -- Attached patch which validates logged in user vs application user before displaying "Kill Application" Validated this on a secure cluster with secure UI was (Author: suma.shivaprasad): Attached patch which validates logged in user vs application user before displaying "Kill Application" > "Kill application" button does not show error if other user tries to kill the > application for secure cluster > > > Key: YARN-6991 > URL: https://issues.apache.org/jira/browse/YARN-6991 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Suma Shivaprasad > Attachments: YARN-6991.001.patch > > > 1. Submit an application by user 1 > 2. log into RM UI as user 2 > 3. Kill the application submitted by user 1 > 4. Even though application does not get killed, there is no error/info dialog > box being shown to let the user that "user doesnot have permissions to kill > application of other user" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7186) Add examples in yarn-service
[ https://issues.apache.org/jira/browse/YARN-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7186: -- Attachment: YARN-7186.yarn-native-services.01.patch > Add examples in yarn-service > - > > Key: YARN-7186 > URL: https://issues.apache.org/jira/browse/YARN-7186 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7186.yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
[ https://issues.apache.org/jira/browse/YARN-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-6991: --- Attachment: YARN-6991.001.patch Attached patch which validates logged in user vs application user before displaying "Kill Application" > "Kill application" button does not show error if other user tries to kill the > application for secure cluster > > > Key: YARN-6991 > URL: https://issues.apache.org/jira/browse/YARN-6991 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Suma Shivaprasad > Attachments: YARN-6991.001.patch > > > 1. Submit an application by user 1 > 2. log into RM UI as user 2 > 3. Kill the application submitted by user 1 > 4. Even though application does not get killed, there is no error/info dialog > box being shown to let the user that "user doesnot have permissions to kill > application of other user" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7186) Add examples in yarn-service
Jian He created YARN-7186: - Summary: Add examples in yarn-service Key: YARN-7186 URL: https://issues.apache.org/jira/browse/YARN-7186 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162257#comment-16162257
]
Arun Suresh commented on YARN-7185:
---
Had an offline discussion with [~leftnoteasy]
The main issue here is that prior to YARN-6706, the NM would blindly trust the
RM's guaranteed container allocations and not perform pre-checks before
starting the container (if opp scheduling is turned off). Post YARN-6706, we
make the check for every container. [~leftnoteasy]'s fix here would revert it
to the old behavior.
IMHO, as Vinod suggested, the correct solution should be to force the NM and RM
to use the same Resource Calculator. Unfortunately, given that the
ResourceCaculator is currently a per-queue, per-scheduler setting, it is
difficult to set a "global" resource calculator. Although, I am not sure if
anybody actually uses a configuration with different Resource Calculators at
different levels of the Queue Hierarchy.
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162242#comment-16162242 ] Hudson commented on YARN-6622: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12840 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12840/]) YARN-6622. Document Docker work as experimental (Contributed by Varun (templedf: rev 6651cbcc72d92caf86b744fa76fba513b36b12c7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7128) The error message in TimelineSchemaCreator is not enough to find out the error.
[
https://issues.apache.org/jira/browse/YARN-7128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162243#comment-16162243
]
Hudson commented on YARN-7128:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12840 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12840/])
YARN-7128. The error message in TimelineSchemaCreator is not enough to
(haibochen: rev 661f5eb0c6791148f2d15d5730635ccb668601e3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineSchemaCreator.java
> The error message in TimelineSchemaCreator is not enough to find out the
> error.
> ---
>
> Key: YARN-7128
> URL: https://issues.apache.org/jira/browse/YARN-7128
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
>Reporter: Jinjiang Ling
>Assignee: Jinjiang Ling
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7128.patch
>
>
> When I run the schema creator tool of timeline service v2, then an error
> happened with the error message below:
> {quote}
> 17/08/30 16:04:18 INFO zookeeper.ClientCnxn: Session establishment complete
> on server zdh175/10.43.183.175:2181, sessionid = 0x15e0d065a330019,
> negotiated timeout = 18
> 17/08/30 16:04:18 WARN util.HeapMemorySizeUtil:
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 17/08/30 16:04:18 ERROR storage.TimelineSchemaCreator: Error in creating
> hbase tables: java.lang.reflect.InvocationTargetException
> 17/08/30 16:04:18 WARN storage.TimelineSchemaCreator: Schema creation
> finished with the following exceptions
> 17/08/30 16:04:18 WARN storage.TimelineSchemaCreator:
> java.lang.reflect.InvocationTargetException
> {quote}
> But I can't tell which error caused this exception with message.
> Then I find it only print the exception message in TimelineSchemaCreator
> {code}
> try {
> ..
> } catch (IOException e) {
> LOG.error("Error in creating hbase tables: " + e.getMessage());
> exceptions.add(e);
> }
> {code}
> I change it like this
> {code}
> try {
> ..
> } catch (IOException e) {
> LOG.error("Error in creating hbase tables: ", e);
> exceptions.add(e);
> }
> {code}
> finally, I get the detail of this error, and it's caused by a directory of
> hbase not created.
> {quote}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
> ... 10 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /data1/zdh/hbase/tmp/RegionServer/local/jars, DynamicClassLoader failed to
> init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:244)
> ... 15 more
> 17/08/30 16:03:42 WARN storage.TimelineSchemaCreator: Schema creation
> finished with the following exceptions
> 17/08/30 16:03:42 WARN storage.TimelineSchemaCreator:
> java.lang.reflect.InvocationTargetException
> {quote}
> As the exception message is not enough to find out the error, so I think
> print all the information of the exception in TimelineSchemaCreator is better.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6022) Revert changes of AbstractResourceRequest
[
https://issues.apache.org/jira/browse/YARN-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162241#comment-16162241
]
Hudson commented on YARN-6022:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12840 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12840/])
Revert "YARN-6022. Document Docker work as experimental (Contributed by
(templedf: rev 07951b586e273ed8596c5f24b1e8d7f627d0104c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
> Revert changes of AbstractResourceRequest
> -
>
> Key: YARN-6022
> URL: https://issues.apache.org/jira/browse/YARN-6022
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-6022.001.patch, YARN-6022.002.patch,
> YARN-6022.003.patch, YARN-6022.004.patch, YARN-6022.005.patch,
> YARN-6022.branch-2.005.patch, YARN-6022.branch-2.006.patch
>
>
> YARN-5774 added AbstractResourceRequest to make easier internal scheduler
> change, this is not a correct approach: For example, with this change, we
> need to make AbstractResourceRequest to be public/stable. And end users can
> use it like:
> {code}
> AbstractResourceRequest request = ...
> request.setCapability(...)
> {code}
> But AbstractResourceRequest should not be visible by application at all.
> We need to revert it from branch-2.8 / branch-2 / trunk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7128) The error message in TimelineSchemaCreator is not enough to find out the error.
[
https://issues.apache.org/jira/browse/YARN-7128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162216#comment-16162216
]
Haibo Chen commented on YARN-7128:
--
+1 on the patch. Thanks for the contribution [~lingjinjiang]!
> The error message in TimelineSchemaCreator is not enough to find out the
> error.
> ---
>
> Key: YARN-7128
> URL: https://issues.apache.org/jira/browse/YARN-7128
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
>Reporter: Jinjiang Ling
>Assignee: Jinjiang Ling
> Attachments: YARN-7128.patch
>
>
> When I run the schema creator tool of timeline service v2, then an error
> happened with the error message below:
> {quote}
> 17/08/30 16:04:18 INFO zookeeper.ClientCnxn: Session establishment complete
> on server zdh175/10.43.183.175:2181, sessionid = 0x15e0d065a330019,
> negotiated timeout = 18
> 17/08/30 16:04:18 WARN util.HeapMemorySizeUtil:
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 17/08/30 16:04:18 ERROR storage.TimelineSchemaCreator: Error in creating
> hbase tables: java.lang.reflect.InvocationTargetException
> 17/08/30 16:04:18 WARN storage.TimelineSchemaCreator: Schema creation
> finished with the following exceptions
> 17/08/30 16:04:18 WARN storage.TimelineSchemaCreator:
> java.lang.reflect.InvocationTargetException
> {quote}
> But I can't tell which error caused this exception with message.
> Then I find it only print the exception message in TimelineSchemaCreator
> {code}
> try {
> ..
> } catch (IOException e) {
> LOG.error("Error in creating hbase tables: " + e.getMessage());
> exceptions.add(e);
> }
> {code}
> I change it like this
> {code}
> try {
> ..
> } catch (IOException e) {
> LOG.error("Error in creating hbase tables: ", e);
> exceptions.add(e);
> }
> {code}
> finally, I get the detail of this error, and it's caused by a directory of
> hbase not created.
> {quote}
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
> ... 10 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /data1/zdh/hbase/tmp/RegionServer/local/jars, DynamicClassLoader failed to
> init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:244)
> ... 15 more
> 17/08/30 16:03:42 WARN storage.TimelineSchemaCreator: Schema creation
> finished with the following exceptions
> 17/08/30 16:03:42 WARN storage.TimelineSchemaCreator:
> java.lang.reflect.InvocationTargetException
> {quote}
> As the exception message is not enough to find out the error, so I think
> print all the information of the exception in TimelineSchemaCreator is better.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7063) TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on TSv2 branch2
[
https://issues.apache.org/jira/browse/YARN-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162210#comment-16162210
]
Haibo Chen commented on YARN-7063:
--
bq. Do we have any other option?
Agree with you there is no other option.
bq. this code would have to be retained.
Do you mind removing the last sentence of the comment in this case?
> TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on
> TSv2 branch2
> -
>
> Key: YARN-7063
> URL: https://issues.apache.org/jira/browse/YARN-7063
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Attachments: YARN-7063-YARN-5355_branch2.01.patch
>
>
> Seeing NoClassDefFound on the branch2 at runtime
> Stack trace
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/security/AuthenticationWithProxyUserFilter
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderAuthenticationFilterInitializer.initFilter(TimelineReaderAuthenticationFilterInitializer.java:49)
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:393)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:344)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:104)
> at
> org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:292)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderWebApp(TimelineReaderServer.java:181)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceStart(TimelineReaderServer.java:124)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.AbstractTimelineReaderHBaseTestBase.initialize(AbstractTimelineReaderHBaseTestBase.java:91)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setupBeforeClass(TestTimelineReaderWebServicesHBaseStorage.java:79)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7063) TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on TSv2 branch2
[
https://issues.apache.org/jira/browse/YARN-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162210#comment-16162210
]
Haibo Chen edited comment on YARN-7063 at 9/11/17 11:17 PM:
bq. Do we have any other option?
Agree with you there is no other option.
bq. this code would have to be retained.
Do you mind removing the last sentence of the comment in this case?
Otherwise, patch LGTM.
was (Author: haibochen):
bq. Do we have any other option?
Agree with you there is no other option.
bq. this code would have to be retained.
Do you mind removing the last sentence of the comment in this case?
> TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on
> TSv2 branch2
> -
>
> Key: YARN-7063
> URL: https://issues.apache.org/jira/browse/YARN-7063
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Attachments: YARN-7063-YARN-5355_branch2.01.patch
>
>
> Seeing NoClassDefFound on the branch2 at runtime
> Stack trace
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/security/AuthenticationWithProxyUserFilter
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderAuthenticationFilterInitializer.initFilter(TimelineReaderAuthenticationFilterInitializer.java:49)
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:393)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:344)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:104)
> at
> org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:292)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderWebApp(TimelineReaderServer.java:181)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceStart(TimelineReaderServer.java:124)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.AbstractTimelineReaderHBaseTestBase.initialize(AbstractTimelineReaderHBaseTestBase.java:91)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setupBeforeClass(TestTimelineReaderWebServicesHBaseStorage.java:79)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6022) Revert changes of AbstractResourceRequest
[
https://issues.apache.org/jira/browse/YARN-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162202#comment-16162202
]
Hudson commented on YARN-6022:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12838 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12838/])
YARN-6022. Document Docker work as experimental (Contributed by Varun
(templedf: rev 66be6c848b47ae7248b0658e3bb2149d2387cebd)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
> Revert changes of AbstractResourceRequest
> -
>
> Key: YARN-6022
> URL: https://issues.apache.org/jira/browse/YARN-6022
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-6022.001.patch, YARN-6022.002.patch,
> YARN-6022.003.patch, YARN-6022.004.patch, YARN-6022.005.patch,
> YARN-6022.branch-2.005.patch, YARN-6022.branch-2.006.patch
>
>
> YARN-5774 added AbstractResourceRequest to make easier internal scheduler
> change, this is not a correct approach: For example, with this change, we
> need to make AbstractResourceRequest to be public/stable. And end users can
> use it like:
> {code}
> AbstractResourceRequest request = ...
> request.setCapability(...)
> {code}
> But AbstractResourceRequest should not be visible by application at all.
> We need to revert it from branch-2.8 / branch-2 / trunk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7137) Resource profile: Move new added APIs to unstable
[
https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162203#comment-16162203
]
Hadoop QA commented on YARN-7137:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
48s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
54s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 59 unchanged - 1 fixed = 62 total (was 60) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 4s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7137 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886468/YARN-7137.YARN-3926.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3d4fae3aa7ea 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / f6ca20b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17406/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162189#comment-16162189
]
Vinod Kumar Vavilapalli commented on YARN-7185:
---
Even if opportunistic scheduling is enabled
# We shouldn't do such checks at all for guaranteed containers? {{There are no
sufficient resources to start guaranteed container_ at the moment.}}
# We should look at vcores only if scheduler is looking at vcores for
scheduling?
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162173#comment-16162173 ] Chris Douglas commented on YARN-6622: - Sure, it's better than nothing. Thanks, [~templedf]. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Attachment: YARN-7185.001.patch
[~asuresh], could you check the patch before I add more tests?
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162152#comment-16162152
]
Wangda Tan commented on YARN-7185:
--
A simple fix for this issue in my mind is only do available resource check when
max opportunistic queue len configured to > 0. Please see attached ver.001
patch.
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Attachments: YARN-7185.001.patch
>
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162146#comment-16162146 ] Daniel Templeton commented on YARN-6622: Since the original review comments were asking for richer docs, I'm going to go ahead and +1 this patch and commit it. I'll file a second JIRA to enrich the docs as mentioned above. Any issue with that, [~chris.douglas]? > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Description:
Found an issue:
When DefaultContainerCalculator is enabled and opportunistic container
allocation is disabled. It is possible that for a NM:
{code}
Σ(allocated-container.vcores) > nm.configured-vores.
{code}
When this happens, ContainerScheduler will report errors like:
bq. ContainerScheduler
(ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are no
sufficient resources to start guaranteed.
This will be an incompatible change after 2.8 because before YARN-6706, we can
start containers when DefaultContainerCalculator is configured and vcores is
overallocated.
was:
Application fails to go to FINISHED state or sometimes to RUNNING state.
In the nodemanager, we can see the following warnings
{Code}
WARN scheduler.ContainerScheduler
(ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are no
sufficient resources to start guaranteed container_ at the moment.
Opportunistic containers are in the process ofbeing killed to make room
{Code}
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Found an issue:
> When DefaultContainerCalculator is enabled and opportunistic container
> allocation is disabled. It is possible that for a NM:
> {code}
> Σ(allocated-container.vcores) > nm.configured-vores.
> {code}
> When this happens, ContainerScheduler will report errors like:
> bq. ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed.
> This will be an incompatible change after 2.8 because before YARN-6706, we
> can start containers when DefaultContainerCalculator is configured and vcores
> is overallocated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6977) Node information is not provided for non am containers in RM logs
[
https://issues.apache.org/jira/browse/YARN-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162133#comment-16162133
]
Wangda Tan commented on YARN-6977:
--
+1, will commit it later, thanks [~suma.shivaprasad]!
> Node information is not provided for non am containers in RM logs
> -
>
> Key: YARN-6977
> URL: https://issues.apache.org/jira/browse/YARN-6977
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sumana Sathish
>Assignee: Suma Shivaprasad
> Labels: newbie
> Attachments: YARN-6977.patch
>
>
> There is no information on which node non am container is being assigned in
> the trunk for hadoop 3.0
> Earlier we used to have logs for non am container in the similar way
> {code}
> Assigned container container_ of capacity on host
> , which has 1 containers, used and
> available after allocation
> {code}
> 3.0 has information for am container alone in the following way
> {code}
> Done launching container Container: [ContainerId: container_,
> AllocationRequestId: 0, Version: 0, NodeId:nodeID, NodeHttpAddress:
> nodeAddress, Resource: , Priority: 0, Token: Token {
> kind: ContainerToken, service: service}, ExecutionType: GUARANTEED, ] for AM
> appattempt_
> {code}
> Can we please have similar message for Non am container too ??
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) ContainerScheduler should only look at availableResource for GUARANTEED containers when opportunistic scheduling is enabled
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Summary: ContainerScheduler should only look at availableResource for
GUARANTEED containers when opportunistic scheduling is enabled (was: Fix
miscellaneous issues in ContainerScheduler)
> ContainerScheduler should only look at availableResource for GUARANTEED
> containers when opportunistic scheduling is enabled
>
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162103#comment-16162103 ] Daniel Templeton commented on YARN-6622: Not having docs doesn't close the security hole. Now, a lot of what the docs document isn't available in 2.8, so we can't just backport YARN-5258, but we still need to make it clear that Docker support is not production ready. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162099#comment-16162099 ] Junping Du commented on YARN-6622: -- bq. Then let's backport YARN-5258. Enabling docker support in branch-2 effectively gives any user the capability to run processes as root on cluster machines. I agree we should do this for branch-2, but I am conservative on backporting it to branch-2.8.2. Most target for branch-2.8.2 is to finalize existing features/improvements since 2.8.0, especially this is in RC stage now. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162090#comment-16162090 ] Chris Douglas commented on YARN-6622: - Then let's backport YARN-5258. Enabling docker support in branch-2 effectively gives any user the capability to run processes as root on cluster machines. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7057) FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption
[ https://issues.apache.org/jira/browse/YARN-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162085#comment-16162085 ] Yufei Gu commented on YARN-7057: Thanks [~templedf] for the review. [~kasha], this patch doesn't apply to trunk. Can you rebase? or I can do the rebase if you are OK with that. > FSAppAttempt#getResourceUsage doesn't need to consider resources queued for > preemption > -- > > Key: YARN-7057 > URL: https://issues.apache.org/jira/browse/YARN-7057 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: YARN-7057.001.patch > > > FSAppAttempt#getResourceUsage excludes resources that are currently allocated > to the app but are about to be preempted. This inconsistency shows in the UI > and can affect scheduling of containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-6622: - Target Version/s: 2.9.0, 3.0.0-beta1 (was: 3.0.0-beta1, 2.8.3) > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162072#comment-16162072 ] Junping Du commented on YARN-6622: -- I just find that YARN-5258 (documentation for docker containers) is only backport to 2.9 and 3.0 only which means this jira's work doesn't apply on branch-2.8 releases. Drop 2.8.3 and replace 2.9 instead. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7146) Many RM unit tests failing with FairScheduler
[
https://issues.apache.org/jira/browse/YARN-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162070#comment-16162070
]
Robert Kanter commented on YARN-7146:
-
Test failures unrelated: YARN-7044, YARN6747
> Many RM unit tests failing with FairScheduler
> -
>
> Key: YARN-7146
> URL: https://issues.apache.org/jira/browse/YARN-7146
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7146.001.patch, YARN-7146.002.patch,
> YARN-7146.003.patch
>
>
> Many of the RM unit tests are failing when using the FairScheduler.
> Here is a list of affected test classes:
> {noformat}
> TestYarnClient
> TestApplicationCleanup
> TestApplicationMasterLauncher
> TestDecommissioningNodesWatcher
> TestKillApplicationWithRMHA
> TestNodeBlacklistingOnAMFailures
> TestRM
> TestRMAdminService
> TestRMRestart
> TestResourceTrackerService
> TestWorkPreservingRMRestart
> TestAMRMRPCNodeUpdates
> TestAMRMRPCResponseId
> TestAMRestart
> TestApplicationLifetimeMonitor
> TestNodesListManager
> TestRMContainerImpl
> TestAbstractYarnScheduler
> TestSchedulerUtils
> TestFairOrderingPolicy
> TestAMRMTokens
> TestDelegationTokenRenewer
> {noformat}
> Most of the test methods in these classes are failing, though some do succeed.
> There's two main categories of issues:
> # The test submits an application to the {{MockRM}} and waits for it to enter
> a specific state, which it never does, and the test times out. We need to
> call {{update()}} on the scheduler.
> # The test throws a {{ClassCastException}} on {{FSQueueMetrics}} to
> {{CSQueueMetrics}}. This is because {{QueueMetrics}} metrics are static, and
> a previous test using FairScheduler initialized it, and the current test is
> using CapacityScheduler. We need to reset the metrics.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Fix miscellaneous issues in ContainerScheduler
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Summary: Fix miscellaneous issues in ContainerScheduler (was: Application
fails to go to FINISHED state or sometimes to RUNNING state)
> Fix miscellaneous issues in ContainerScheduler
> --
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162024#comment-16162024 ] Wangda Tan commented on YARN-7136: -- [~templedf], thanks for committing the patch and thanks all the helps and suggestions from [~jlowe]/[~sunilg]/[~templedf]! > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: YARN-3926 > > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, > YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch, > YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch, > YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch, > YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch, > YARN-7136.YARN-3926.016.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7185:
-
Target Version/s: 2.9.0, 3.0.0-beta1 (was: 3.0.0-beta1)
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162012#comment-16162012
]
Junping Du commented on YARN-7185:
--
Container Scheduler should get disabled by default. Bump up this to blocker.
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-7185:
-
Target Version/s: 3.0.0-beta1
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-7185:
-
Fix Version/s: (was: 3.0.0)
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-7185:
-
Priority: Blocker (was: Critical)
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
[
https://issues.apache.org/jira/browse/YARN-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumana Sathish updated YARN-7185:
-
Fix Version/s: 3.0.0
> Application fails to go to FINISHED state or sometimes to RUNNING state
> ---
>
> Key: YARN-7185
> URL: https://issues.apache.org/jira/browse/YARN-7185
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Critical
> Fix For: 3.0.0
>
>
> Application fails to go to FINISHED state or sometimes to RUNNING state.
> In the nodemanager, we can see the following warnings
> {Code}
> WARN scheduler.ContainerScheduler
> (ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are
> no sufficient resources to start guaranteed container_ at the moment.
> Opportunistic containers are in the process ofbeing killed to make room
> {Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7185) Application fails to go to FINISHED state or sometimes to RUNNING state
Sumana Sathish created YARN-7185:
Summary: Application fails to go to FINISHED state or sometimes to
RUNNING state
Key: YARN-7185
URL: https://issues.apache.org/jira/browse/YARN-7185
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Sumana Sathish
Assignee: Tan, Wangda
Priority: Critical
Application fails to go to FINISHED state or sometimes to RUNNING state.
In the nodemanager, we can see the following warnings
{Code}
WARN scheduler.ContainerScheduler
(ContainerScheduler.java:pickOpportunisticContainersToKill(458)) - There are no
sufficient resources to start guaranteed container_ at the moment.
Opportunistic containers are in the process ofbeing killed to make room
{Code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[
https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161993#comment-16161993
]
Hadoop QA commented on YARN-7136:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
24s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
20s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
6s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
24s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 277 unchanged - 8 fixed = 293 total (was 285) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 32s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-yarn-server-tests in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 42s{color} |
{color:black} {color}
[jira] [Commented] (YARN-7146) Many RM unit tests failing with FairScheduler
[
https://issues.apache.org/jira/browse/YARN-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161947#comment-16161947
]
Hadoop QA commented on YARN-7146:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 19 new + 467 unchanged - 5 fixed = 486 total (was 472) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
55s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7146 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886477/YARN-7146.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0d2658ab9622 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / de9994b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17405/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161926#comment-16161926 ] Daniel Templeton commented on YARN-7136: LGTM +1 > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, > YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch, > YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch, > YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch, > YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch, > YARN-7136.YARN-3926.016.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161913#comment-16161913
]
Hadoop QA commented on YARN-7102:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 28 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 32s{color} | {color:orange} root: The patch generated 1 new + 1230 unchanged
- 4 fixed = 1231 total (was 1234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 55s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 30s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
14s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
42s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}166m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
| | hadoop.yarn.server.TestContainerManagerSecurity |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7102 |
| JIRA Patch URL |
https
[jira] [Commented] (YARN-7181) CPUTimeTracker.updateElapsedJiffies can report negative usage
[
https://issues.apache.org/jira/browse/YARN-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161889#comment-16161889
]
Hudson commented on YARN-7181:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12834 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12834/])
YARN-7181. CPUTimeTracker.updateElapsedJiffies can report negative (haibochen:
rev d2d8f665e62a6ddfe20457fb3a07cc2bea9a856b)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/CpuTimeTracker.java
* (add)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestCpuTimeTracker.java
> CPUTimeTracker.updateElapsedJiffies can report negative usage
> -
>
> Key: YARN-7181
> URL: https://issues.apache.org/jira/browse/YARN-7181
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7181.000.patch
>
>
> It happens, when the process exited and elapsedJiffies becomes 0 again.
> {code}
> public void updateElapsedJiffies(BigInteger elapsedJiffies, long newTime) {
> cumulativeCpuTime = elapsedJiffies.multiply(jiffyLengthInMillis);
> sampleTime = newTime;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7181) CPUTimeTracker.updateElapsedJiffies can report negative usage
[
https://issues.apache.org/jira/browse/YARN-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161851#comment-16161851
]
Haibo Chen commented on YARN-7181:
--
+1 Will commit it shortly.
> CPUTimeTracker.updateElapsedJiffies can report negative usage
> -
>
> Key: YARN-7181
> URL: https://issues.apache.org/jira/browse/YARN-7181
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7181.000.patch
>
>
> It happens, when the process exited and elapsedJiffies becomes 0 again.
> {code}
> public void updateElapsedJiffies(BigInteger elapsedJiffies, long newTime) {
> cumulativeCpuTime = elapsedJiffies.multiply(jiffyLengthInMillis);
> sampleTime = newTime;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161833#comment-16161833 ] Eric Badger commented on YARN-6622: --- bq. It's generally known to developers, but not to users. They're the target for this documentation. Unless they're familiar with both Docker and Hadoop, they're unlikely to understand the consequences of enabling this feature. I agree with this. Until I started working specifically with the docker support in hadoop, I didn't fully understand the kinds of trouble you can get into with enabling this feature. If you don't know what you're doing related to security (and/or don't care I guess), you should *not* enable Docker in its current state. This should be properly communicated to the users > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161795#comment-16161795 ] Chris Douglas commented on YARN-6622: - bq. docker container is alpha feature which is generally known. It's generally known to developers, but not to users. They're the target for this documentation. Unless they're familiar with both Docker and Hadoop, they're unlikely to understand the consequences of enabling this feature. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161771#comment-16161771 ] Junping Du commented on YARN-6622: -- This is out of my radar too as marked as major instead of blocker/critical. Like I mentioned in release thread, let's handle this release message given RC0 is out. Let's bump up to critical instead of blocker as docker container is alpha feature which is generally known. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor
[ https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161770#comment-16161770 ] Shane Kumpf commented on YARN-6726: --- Thanks [~wangda] [~sunilg] and [~ebadger]! > Fix issues with docker commands executed by container-executor > -- > > Key: YARN-6726 > URL: https://issues.apache.org/jira/browse/YARN-6726 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6726.001.patch, YARN-6726.002.patch, > YARN-6726.003.patch > > > docker inspect, rm, stop, etc are issued through container-executor. Commands > other than docker run are not functioning properly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7146) Many RM unit tests failing with FairScheduler
[
https://issues.apache.org/jira/browse/YARN-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-7146:
Attachment: YARN-7146.003.patch
The 003 patch:
- Adds a comment about assign multiple
- For {{update()}}, I ended up moving the {{UpdateThread}} mechanism from
{{FairScheduler}} to {{AbstractYarnScheduler}}. The {{update()}} method was
otherwise only called from tests and {{MockRM}}, so I couldn't really justify a
valid comment for it without moving it. Doing that mostly involved moving some
code around.
-- The {{UpdateThread}} will only start if the scheduler subclass sets the
{{updateInterval}} to a positive value, so currently only {{FairScheduler}}
actually uses it, but it is available to other schedulers.
-- {{TestContinuousScheduling}} had a test that checked the lifecycle of the
{{UpdateThread}} as well as the {{SchedulingThread}}. I changed that to only
do the {{SchedulingThread}} and added a similar test for the {{UpdateThread}}
to {{TestAbstractYarnScheduler}}.
> Many RM unit tests failing with FairScheduler
> -
>
> Key: YARN-7146
> URL: https://issues.apache.org/jira/browse/YARN-7146
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7146.001.patch, YARN-7146.002.patch,
> YARN-7146.003.patch
>
>
> Many of the RM unit tests are failing when using the FairScheduler.
> Here is a list of affected test classes:
> {noformat}
> TestYarnClient
> TestApplicationCleanup
> TestApplicationMasterLauncher
> TestDecommissioningNodesWatcher
> TestKillApplicationWithRMHA
> TestNodeBlacklistingOnAMFailures
> TestRM
> TestRMAdminService
> TestRMRestart
> TestResourceTrackerService
> TestWorkPreservingRMRestart
> TestAMRMRPCNodeUpdates
> TestAMRMRPCResponseId
> TestAMRestart
> TestApplicationLifetimeMonitor
> TestNodesListManager
> TestRMContainerImpl
> TestAbstractYarnScheduler
> TestSchedulerUtils
> TestFairOrderingPolicy
> TestAMRMTokens
> TestDelegationTokenRenewer
> {noformat}
> Most of the test methods in these classes are failing, though some do succeed.
> There's two main categories of issues:
> # The test submits an application to the {{MockRM}} and waits for it to enter
> a specific state, which it never does, and the test times out. We need to
> call {{update()}} on the scheduler.
> # The test throws a {{ClassCastException}} on {{FSQueueMetrics}} to
> {{CSQueueMetrics}}. This is because {{QueueMetrics}} metrics are static, and
> a previous test using FairScheduler initialized it, and the current test is
> using CapacityScheduler. We need to reset the metrics.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6726) Fix issues with docker commands executed by container-executor
[ https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6726: - Fix Version/s: 2.9.0 > Fix issues with docker commands executed by container-executor > -- > > Key: YARN-6726 > URL: https://issues.apache.org/jira/browse/YARN-6726 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6726.001.patch, YARN-6726.002.patch, > YARN-6726.003.patch > > > docker inspect, rm, stop, etc are issued through container-executor. Commands > other than docker run are not functioning properly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor
[ https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161747#comment-16161747 ] Wangda Tan commented on YARN-6726: -- [~ebadger] / [~shaneku...@gmail.com], I just committed the patch to branch-2 as well. > Fix issues with docker commands executed by container-executor > -- > > Key: YARN-6726 > URL: https://issues.apache.org/jira/browse/YARN-6726 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6726.001.patch, YARN-6726.002.patch, > YARN-6726.003.patch > > > docker inspect, rm, stop, etc are issued through container-executor. Commands > other than docker run are not functioning properly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6622: -- Priority: Blocker (was: Major) I'm going to raise this to a blocker for at least 3.0.0-beta1 based on Daniel's comment. Wasn't on my radar before. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6726) Fix issues with docker commands executed by container-executor
[ https://issues.apache.org/jira/browse/YARN-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161706#comment-16161706 ] Eric Badger commented on YARN-6726: --- YARN-6804 is in branch-2 now. [~leftnoteasy], can we backport this now? I did a quick check and it looks like the cherry-pick is pretty clean, only a few import inconsistencies. > Fix issues with docker commands executed by container-executor > -- > > Key: YARN-6726 > URL: https://issues.apache.org/jira/browse/YARN-6726 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Fix For: 3.0.0-beta1 > > Attachments: YARN-6726.001.patch, YARN-6726.002.patch, > YARN-6726.003.patch > > > docker inspect, rm, stop, etc are issued through container-executor. Commands > other than docker run are not functioning properly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter
[
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161703#comment-16161703
]
Wangda Tan commented on YARN-7172:
--
+1 to latest patch, [~templedf], could you help to commit the patch sooner than
later since I need to rebase YARN-3926 branch on top of this patch.
> ResourceCalculator.fitsIn() should not take a cluster resource parameter
>
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
> Labels: newbie
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch,
> YARN-7172.003.patch, YARN-7172.004.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}}
> (which involves a lock) to get a value to pass as the cluster resource
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably)
> ignored. We should remove the parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7137) Resource profile: Move new added APIs to unstable
[ https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7137: - Attachment: YARN-7137.YARN-3926.002.patch Attached ver.2 patch based on YARN-7136. > Resource profile: Move new added APIs to unstable > - > > Key: YARN-7137 > URL: https://issues.apache.org/jira/browse/YARN-7137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7137.YARN-3926.001.patch, > YARN-7137.YARN-3926.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7173) Container Update Backward compatibility fix for upgrades
[
https://issues.apache.org/jira/browse/YARN-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161680#comment-16161680
]
Wangda Tan commented on YARN-7173:
--
Thanks [~asuresh],
+1 to latest patch, will commit today if no opposite opinions.
> Container Update Backward compatibility fix for upgrades
>
>
> Key: YARN-7173
> URL: https://issues.apache.org/jira/browse/YARN-7173
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7173.001.patch, YARN-7173-branch-2.001.patch
>
>
> This is based on discussions with [~leftnoteasy] in YARN-6979.
> In YARN-6979, the {{getContainersToDecrease()}} and
> {{addAllContainersToDecrease()}} methods were removed from the
> NodeHeartbeatResponse (although the actual protobuf fields were still
> retained). We need to ensure that for clusters that upgrade from 2.8.x to
> 2.9.0, the decreased containers should also be sent to the NM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7136: - Attachment: YARN-7136.YARN-3926.016.patch Thanks [~templedf]. Done, please review 016. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, > YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch, > YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch, > YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch, > YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch, > YARN-7136.YARN-3926.016.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[
https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161629#comment-16161629
]
Daniel Templeton commented on YARN-7136:
That takes care of {{LightWeightResource}}, but {{Resource.compareTo()}} is
where the ternary has a performance cost. Can we fix that one, too?
> Additional Performance Improvement for Resource Profile Feature
> ---
>
> Key: YARN-7136
> URL: https://issues.apache.org/jira/browse/YARN-7136
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch,
> YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch,
> YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch,
> YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch,
> YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch,
> YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch,
> YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch,
> YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch
>
>
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache
> #known-resource-types. (Significant improvement).
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length
> every time. (Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize
> default Memory/VCores ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some
> improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor
> improvement).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161623#comment-16161623
]
Botong Huang edited comment on YARN-7102 at 9/11/17 5:23 PM:
-
V3 updated, fix more unit test failures around {{MiniYarnCluster}}. Removing
one unit test in TestMiniYarnClusterNodeUtilization because the other test
consumes this one.
was (Author: botong):
V3 updated, fix more unit test failures around {{MiniYarnCluster}}
> NM heartbeat stuck when responseId overflows MAX_INT
>
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Critical
> Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch,
> YARN-7102.v3.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM
> heartbeat in YARN-6640, please refer to YARN-6640 for details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT
[
https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Botong Huang updated YARN-7102:
---
Attachment: YARN-7102.v3.patch
V3 updated, fix more unit test failures around {{MiniYarnCluster}}
> NM heartbeat stuck when responseId overflows MAX_INT
>
>
> Key: YARN-7102
> URL: https://issues.apache.org/jira/browse/YARN-7102
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Critical
> Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch,
> YARN-7102.v3.patch
>
>
> ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM
> heartbeat in YARN-6640, please refer to YARN-6640 for details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7162) Remove XML excludes file format
[ https://issues.apache.org/jira/browse/YARN-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161588#comment-16161588 ] Robert Kanter commented on YARN-7162: - [~djp], can you take a look at the patch? FYI, [~mingma] voted for removing the XML format for now in [this comment|https://issues.apache.org/jira/browse/YARN-5536?focusedCommentId=16157204&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16157204] on YARN-5536. > Remove XML excludes file format > --- > > Key: YARN-7162 > URL: https://issues.apache.org/jira/browse/YARN-7162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: YARN-7162.001.patch, YARN-7162.branch-2.001.patch > > > YARN-5536 aims to replace the XML format for the excludes file with a JSON > format. However, it looks like we won't have time for that for Hadoop 3 Beta > 1. The concern is that if we release it as-is, we'll now have to support the > XML format as-is for all of Hadoop 3.x, which we're either planning on > removing, or rewriting using a pluggable framework. > [This comment in > YARN-5536|https://issues.apache.org/jira/browse/YARN-5536?focusedCommentId=16126194&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126194] > proposed two quick solutions to prevent this compat issue. In this JIRA, > we're going to remove the XML format. If we later want to add it back in, > YARN-5536 can add it back, rewriting it to be in the pluggable framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6622) Document Docker work as experimental
[ https://issues.apache.org/jira/browse/YARN-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161566#comment-16161566 ] Daniel Templeton commented on YARN-6622: [~djp], I don't think it's a good idea to bump this out to 2.8.3. This needs to be addressed before we push out another release. > Document Docker work as experimental > > > Key: YARN-6622 > URL: https://issues.apache.org/jira/browse/YARN-6622 > Project: Hadoop YARN > Issue Type: Task > Components: documentation >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6622.001.patch > > > We should update the Docker support documentation calling out the Docker work > as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Comment: was deleted (was: State Issue => YARN 2.8.1) > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: (was: 281.log.zip) > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: (was: state_issue_yarn_281.log.Z) > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: 281.log.zip, state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan reopened YARN-7183: -- > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: 281.log.zip, state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161511#comment-16161511 ] Anbu Cheeralan edited comment on YARN-7183 at 9/11/17 4:53 PM: --- Here is the relevant log from Yarn 2.8.1 RM [[^state_issue_yarn_281.log.zip]] ==> application id is application_1502985025699_2813 Here is the relevant log from Yarn 2.8.0 RM [^state_issue_yarn_280.log.zip] ==> application id is application_1501528954366_8919 was (Author: alunarbeach): Here is the relevant log from Yarn 2.8.1 RM [^state_issue_yarn_281.log.Z] ==> application id is application_1502985025699_2813 > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: 281.log.zip, state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: state_issue_yarn_281.log.zip State Issue => YARN 2.8.1 > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: 281.log.zip, state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.Z, state_issue_yarn_281.log.zip > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: 281.log.zip > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: 281.log.zip, state_issue_yarn_280.log.zip, > state_issue_yarn_281.log.Z > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: state_issue_yarn_280.log.zip > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: state_issue_yarn_280.log.zip, state_issue_yarn_281.log.Z > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7149) Cross-queue preemption sometimes starves an underserved queue
[
https://issues.apache.org/jira/browse/YARN-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161546#comment-16161546
]
Eric Payne commented on YARN-7149:
--
bq. You could check the unit test code to see if that matches your expectation.
I see that the patch for YARN-5889 needed to change the headroom usage in
{{TestLeafQueue}} for Assersions in {{testComputeUserLimitAndSetHeadroom}} and
{{testHeadroomWithMaxCap}}:
{code}
@@ -1123,9 +1129,9 @@ public void testComputeUserLimitAndSetHeadroom() throws
IOException {
//testcase3 still active - 2+2+6=10
assertEquals(10*GB, qb.getUsedResources().getMemorySize());
//app4 is user 0
-//maxqueue 16G, userlimit 13G, used 8G, headroom 5G
+//maxqueue 16G, userlimit 7G, used 8G, headroom 5G
//(8G used is 6G from this test case - app4, 2 from last test case, app_1)
-assertEquals(5*GB, app_4.getHeadroom().getMemorySize());
+assertEquals(0*GB, app_4.getHeadroom().getMemorySize());
}
@Test
@@ -1309,8 +1315,8 @@ public void testHeadroomWithMaxCap() throws Exception {
assertEquals(2*GB, app_0.getCurrentConsumption().getMemorySize());
assertEquals(0*GB, app_1.getCurrentConsumption().getMemorySize());
// TODO, fix headroom in the future patch
-assertEquals(1*GB, app_0.getHeadroom().getMemorySize());
- // User limit = 4G, 2 in use
+assertEquals(0*GB, app_0.getHeadroom().getMemorySize());
+ // User limit = 2G, 2 in use
assertEquals(0*GB, app_1.getHeadroom().getMemorySize());
// the application is not yet active
@@ -1322,15 +1328,15 @@ public void testHeadroomWithMaxCap() throws Exception {
assertEquals(3*GB, a.getUsedResources().getMemorySize());
assertEquals(2*GB, app_0.getCurrentConsumption().getMemorySize());
assertEquals(1*GB, app_1.getCurrentConsumption().getMemorySize());
-assertEquals(1*GB, app_0.getHeadroom().getMemorySize()); // 4G - 3G
-assertEquals(1*GB, app_1.getHeadroom().getMemorySize()); // 4G - 3G
+assertEquals(0*GB, app_0.getHeadroom().getMemorySize()); // 4G - 3G
+assertEquals(0*GB, app_1.getHeadroom().getMemorySize()); // 4G - 3G
// Submit requests for app_1 and set max-cap
a.setMaxCapacity(.1f);
app_2.updateResourceRequests(Collections.singletonList(
{code}
> Cross-queue preemption sometimes starves an underserved queue
> -
>
> Key: YARN-7149
> URL: https://issues.apache.org/jira/browse/YARN-7149
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-7149.demo.unit-test.patch
>
>
> In branch 2 and trunk, I am consistently seeing some use cases where
> cross-queue preemption does not happen when it should. I do not see this in
> branch-2.8.
> Use Case:
> | | *Size* | *Minimum Container Size* |
> |MyCluster | 20 GB | 0.5 GB |
> | *Queue Name* | *Capacity* | *Absolute Capacity* | *Minimum User Limit
> Percent (MULP)* | *User Limit Factor (ULF)* |
> |Q1 | 50% = 10 GB | 100% = 20 GB | 10% = 1 GB | 2.0 |
> |Q2 | 50% = 10 GB | 100% = 20 GB | 10% = 1 GB | 2.0 |
> - {{User1}} launches {{App1}} in {{Q1}} and consumes all resources (20 GB)
> - {{User2}} launches {{App2}} in {{Q2}} and requests 10 GB
> - _Note: containers are 0.5 GB._
> - Preemption monitor kills 2 containers (equals 1 GB) from {{App1}} in {{Q1}}.
> - Capacity Scheduler assigns 2 containers (equals 1 GB) to {{App2}} in {{Q2}}.
> - _No more containers are ever preempted, even though {{Q2}} is far
> underserved_
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161511#comment-16161511 ] Anbu Cheeralan commented on YARN-7183: -- Here is the relevant log from Yarn 2.8.1 RM [^state_issue_yarn_281.log.Z] ==> application id is application_1502985025699_2813 > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: state_issue_yarn_281.log.Z > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anbu Cheeralan updated YARN-7183: - Attachment: state_issue_yarn_281.log.Z > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > Attachments: state_issue_yarn_281.log.Z > > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-6059: - Assignee: Hitesh Sharma (was: Arun Suresh) > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma >Priority: Blocker > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-5216-YARN-6059.001.patch, > YARN-6059-YARN-5972.001.patch, YARN-6059-YARN-5972.002.patch, > YARN-6059-YARN-5972.003.patch, YARN-6059-YARN-5972.004.patch, > YARN-6059-YARN-5972.005.patch, YARN-6059-YARN-5972.006.patch, > YARN-6059-YARN-5972.007.patch, YARN-6059-YARN-5972.008.patch, > YARN-6059-YARN-5972.009.patch, YARN-6059-YARN-5972.010.patch, > YARN-6059-YARN-5972.011.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161414#comment-16161414 ] Rohith Sharma K S commented on YARN-7183: - Can you attach RM logs ? And point to application id? > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161404#comment-16161404 ] Anbu Cheeralan commented on YARN-7183: -- [~rohit12sh] Thanks for your time to answer this issue. This is an existing application which was running in production with Yarn 2.8.0. After upgrading the cluster to Yarn 2.8.1, on failure scenarios now we see the different state (FINISHED instead of FAILED). I believe this is a bug introduced in Yarn 2.8.1. The spark application in both case returning exit code 15. In both cases, AM was running before throwing the error. > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-7183. - Resolution: Invalid Closing as invalid! Please use user mailing lists to ask such doubts! > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7183) YARN - State vs Final Status - Discrepancy in 2.8.1
[ https://issues.apache.org/jira/browse/YARN-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161266#comment-16161266 ] Rohith Sharma K S commented on YARN-7183: - There are 2 different meanings for above mentioned cases # FinalStatus is "FAILED" and State is "FAILED" : AM unable to launch it self. It may be because of several reasons which need to be check for diagnosis message! # FinalStatus is "FAILED" and State is "FINISHED" : AM registered and started running successfully. But AM is unregistered with "FAILED" stated. It means AM has failed to run application. > YARN - State vs Final Status - Discrepancy in 2.8.1 > --- > > Key: YARN-7183 > URL: https://issues.apache.org/jira/browse/YARN-7183 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.8.1 > Environment: CENT OS >Reporter: Anbu Cheeralan > > Same Spark application results in different behavior between Hadoop 2.8.0 and > 2.8.1 > In 2.8.0 UI , FinalStatus is "FAILED" and State is "FAILED" > In 2.8.1 UI, FinalStatus is "FAILED" and State is "FINISHED". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
