[jira] [Commented] (YARN-5431) TimeLineReader daemon start should allow to pass its own reader opts
[
https://issues.apache.org/jira/browse/YARN-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396988#comment-15396988
]
Hudson commented on YARN-5431:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10166 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10166/])
YARN-5431. TimelineReader daemon start should allow to pass its own
(varunsaxena: rev 8d06bda337db45e1c8d6a8982ef83355c86bec6a)
* hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
* hadoop-yarn-project/hadoop-yarn/bin/yarn
* hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd
> TimeLineReader daemon start should allow to pass its own reader opts
>
>
> Key: YARN-5431
> URL: https://issues.apache.org/jira/browse/YARN-5431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts, timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5431.0.patch, YARN-5431.1.patch, YARN-5431.2.patch
>
>
> In yarn script , timelinereader doesn't allow to pass reader_opts.
> {code}
> timelinereader)
> HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
> HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5441) Fixing minor Scheduler test case failures
[
https://issues.apache.org/jira/browse/YARN-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396987#comment-15396987
]
Hudson commented on YARN-5441:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10166 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10166/])
YARN-5441. Fixing minor Scheduler test case failures (subru: rev
d2cbfd7de33fde526089a395550deafb4628fc6f)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
> Fixing minor Scheduler test case failures
> -
>
> Key: YARN-5441
> URL: https://issues.apache.org/jira/browse/YARN-5441
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Fix For: 2.9.0
>
> Attachments: YARN-5441-v1.patch
>
>
> YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}}
> comparator but the ResourceRequest object created via utility methods in few
> tests like *TestFifoScheduler* and *TestCapacityScheduler* do not set
> ExecutionTypeRequest which results in null pointer exceptions. This JIRA
> proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5442) TestYarnClient fails in trunk
[ https://issues.apache.org/jira/browse/YARN-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396894#comment-15396894 ] Rohith Sharma K S commented on YARN-5442: - its dup of YARN-5389 > TestYarnClient fails in trunk > - > > Key: YARN-5442 > URL: https://issues.apache.org/jira/browse/YARN-5442 > Project: Hadoop YARN > Issue Type: Test >Reporter: Xuan Gong > > testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) > Time elapsed: 2.218 sec <<< FAILURE! > java.lang.AssertionError: Exhausted attempts in checking if node capacity was > added to the plan > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584) > testUpdateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) > Time elapsed: 2.181 sec <<< FAILURE! > java.lang.AssertionError: Exhausted attempts in checking if node capacity was > added to the plan > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testUpdateReservation(TestYarnClient.java:1300) > testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) > Time elapsed: 2.257 sec <<< FAILURE! > java.lang.AssertionError: Exhausted attempts in checking if node capacity was > added to the plan > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494) > testCreateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) > Time elapsed: 2.29 sec <<< FAILURE! > java.lang.AssertionError: Exhausted attempts in checking if node capacity was > added to the plan > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) > at > org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testCreateReservation(TestYarnClient.java:1257) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396877#comment-15396877 ] Ying Zhang commented on YARN-5287: -- [~Naganarasimha] and [~rohithsharma], let me know your comments:-) > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4754) Too many connection opened to TimelineServer while publishing entities
[
https://issues.apache.org/jira/browse/YARN-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396875#comment-15396875
]
Ying Zhang commented on YARN-4754:
--
Hi [~varun_saxena], [~rohithsharma], if I understand this correctly, is the
following change enough for this?
{code:title=TimelineWriter.java|borderStyle=solid}
// Moving this call "resp.getEntity(String.class)" out of the if block of " if
(LOG.isDebugEnabled())"
private ClientResponse doPosting(final Object obj, final String path)
throws IOException, YarnException {
... ...
if (resp == null ||
resp.getStatusInfo().getStatusCode()
!= ClientResponse.Status.OK.getStatusCode()) {
... ...
if (resp != null) {
msg += " HTTP error code: " + resp.getStatus();
String output = resp.getEntity(String.class);
if (LOG.isDebugEnabled()) {
LOG.debug("HTTP error code: " + resp.getStatus()
+ " Server response : \n" + output);
}
... ...
}
{code}
> Too many connection opened to TimelineServer while publishing entities
> --
>
> Key: YARN-4754
> URL: https://issues.apache.org/jira/browse/YARN-4754
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
> Attachments: ConnectionLeak.rar
>
>
> It is observed that there are too many connections are kept opened to
> TimelineServer while publishing entities via SystemMetricsPublisher. This
> cause sometimes resource shortage for other process or RM itself
> {noformat}
> tcp0 0 10.18.99.110:3999 10.18.214.60:59265
> ESTABLISHED 115302/java
> tcp0 0 10.18.99.110:25001 :::*LISTEN
> 115302/java
> tcp0 0 10.18.99.110:25002 :::*LISTEN
> 115302/java
> tcp0 0 10.18.99.110:25003 :::*LISTEN
> 115302/java
> tcp0 0 10.18.99.110:25004 :::*LISTEN
> 115302/java
> tcp0 0 10.18.99.110:25005 :::*LISTEN
> 115302/java
> tcp1 0 10.18.99.110:48866 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:48137 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:47553 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:48424 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:48139 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:48096 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:47558 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> tcp1 0 10.18.99.110:49270 10.18.99.110:8188
> CLOSE_WAIT 115302/java
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-4280: -- Attachment: YARN-4280.011.patch Sigh. Fixing the extra check-style issue. Test failures are tracked through YARN-5441 and are unrelated. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396814#comment-15396814
]
Vinod Kumar Vavilapalli commented on YARN-3662:
---
Apologies for chiming in very late. I have more suggestions, please cherry-pick
things that you agree with are easy and feel free to defer the harder ones to
later JIRAs
- Drop *Federation* everywhere in the records? Instead, we can simply say
SubClusterInfoProto, SubClusterIdProto etc.
- None of these classes should be marked {{@Public}}.
- GetFederationSubClusterRequest -> GetSubClusterInfoRequest, similarly
response? Similarly the GetSubClustersRequest / response objects? Also, should
they be returning the info objects?
- HeartbeatFederationSubClusterRequest -> SubClusterHeartbeatRequest.
Similarly the response? Trying to follow NodeHeartbeat* records.
- Enums under FederationSubClusterState can afford not having the SC_ prefix -
the enum class is enough to distinguish them.. Again trying to follow other
enums in YARN.
> Federation Membership State Store internal APIs
> ---
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch,
> YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Zhang updated YARN-5287: - Attachment: YARN-5287.004.patch > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Zhang updated YARN-5287: - Attachment: (was: YARN-5287.004.patch) > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[
https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5221:
--
Attachment: YARN-5221.008.patch
Rebasing patch against trunk.
[~leftnoteasy], regarding your comment, The diagnostic is not dropped. Instead
of throwing an Exception, I am actually returning an {{UpdateError}} back to
the AM.
> Expose UpdateResourceRequest API to allow AM to request for change in
> container properties
> --
>
> Key: YARN-5221
> URL: https://issues.apache.org/jira/browse/YARN-5221
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5221.001.patch, YARN-5221.002.patch,
> YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch,
> YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch
>
>
> YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease
> of Container Resources after initial allocation.
> YARN-5085 proposes to allow an AM to request for a change of Container
> ExecutionType.
> This JIRA proposes to unify both of the above into an Update Container API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[
https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396743#comment-15396743
]
Hadoop QA commented on YARN-4280:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 289 unchanged - 4 fixed = 291 total (was 293)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 30s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820596/YARN-4280.010.patch |
| JIRA Issue | YARN-4280 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9abc3f7185b9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b43de80 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12532/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12532/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
[jira] [Commented] (YARN-3426) Add jdiff support to YARN
[ https://issues.apache.org/jira/browse/YARN-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396738#comment-15396738 ] Wangda Tan commented on YARN-3426: -- Back ported doclet changes to branch-2.7 and branch-2.7.3 > Add jdiff support to YARN > - > > Key: YARN-3426 > URL: https://issues.apache.org/jira/browse/YARN-3426 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu >Priority: Blocker > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-3426-040615-1.patch, YARN-3426-040615.patch, > YARN-3426-040715.patch, YARN-3426-040815.patch, YARN-3426-05-12-2016.txt, > YARN-3426-06-09-2016.txt, YARN-3426-branch-2.005.patch, > YARN-3426-branch-2.8.005.patch > > > Maybe we'd like to extend our current jdiff tool for hadoop-common and hdfs > to YARN as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396735#comment-15396735 ] Karthik Kambatla commented on YARN-3662: Quickly skimmed through. It looks good. > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396696#comment-15396696 ] Wangda Tan commented on YARN-5221: -- [~asuresh], could you update the patch? Since 2.8 release is closing.. > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5441) Fixing minor Scheduler test case failures
[
https://issues.apache.org/jira/browse/YARN-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396685#comment-15396685
]
Arun Suresh commented on YARN-5441:
---
Thanks for raising this [~subru]. +1
Please also verify this fixes the tests [~kshukla] mentioned in YARN-5351
> Fixing minor Scheduler test case failures
> -
>
> Key: YARN-5441
> URL: https://issues.apache.org/jira/browse/YARN-5441
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-5441-v1.patch
>
>
> YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}}
> comparator but the ResourceRequest object created via utility methods in few
> tests like *TestFifoScheduler* and *TestCapacityScheduler* do not set
> ExecutionTypeRequest which results in null pointer exceptions. This JIRA
> proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396675#comment-15396675 ] Li Lu commented on YARN-5440: - Thanks [~xgong]! LGTM. +1 for commit. > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > be used or not. But the AHSClientService is enabled by default when we start > the TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to rely on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396656#comment-15396656 ] Xuan Gong commented on YARN-5440: - Thanks for the review. [~gtCarrera9] Created https://issues.apache.org/jira/browse/YARN-5442 to track the testcase failures. bq. if the user turn off timeline service but enable AHS in the config. So, the configuration: yarn.timeline-service.generic-application-history.enabled is deprecated configuration which means we could remove this configuration without any impact (If i understand the deprecate correctly). In current code base, this configuration controls whether we could get the full information (include current and history) for the apps/attempts/containers by using YarnClient. But the pre-requirement is that we have to run the TimelineServer. If we turn off the timeline server but enable AHS, it would give the connection exception because it tries to connect the ATS. > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > be used or not. But the AHSClientService is enabled by default when we start > the TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to rely on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-4280: -- Attachment: YARN-4280.010.patch Thanks a lot [~jlowe] for the review. I have updated the patch to address the comments. This patch removes the change to ResourceLimits. The if block in ParentQueue was incorrect and has been removed now to cover cases where all child assignments are NULL_ASSIGNMENT(s) with the first(or any one but the last) assignments being a QUEUE_LIMIT skipped case. Will wait for precommit and update other version patches soon. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5442) TestYarnClient fails in trunk
Xuan Gong created YARN-5442: --- Summary: TestYarnClient fails in trunk Key: YARN-5442 URL: https://issues.apache.org/jira/browse/YARN-5442 Project: Hadoop YARN Issue Type: Test Reporter: Xuan Gong testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.218 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584) testUpdateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.181 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testUpdateReservation(TestYarnClient.java:1300) testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.257 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494) testCreateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.29 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testCreateReservation(TestYarnClient.java:1257) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3426) Add jdiff support to YARN
[ https://issues.apache.org/jira/browse/YARN-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396628#comment-15396628 ] Wangda Tan commented on YARN-3426: -- This JIRA has few changes to doclet that we need to backport to branch-2.7 and branch-2.7.3, otherwise generated jdiff is wrong. > Add jdiff support to YARN > - > > Key: YARN-3426 > URL: https://issues.apache.org/jira/browse/YARN-3426 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu >Priority: Blocker > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-3426-040615-1.patch, YARN-3426-040615.patch, > YARN-3426-040715.patch, YARN-3426-040815.patch, YARN-3426-05-12-2016.txt, > YARN-3426-06-09-2016.txt, YARN-3426-branch-2.005.patch, > YARN-3426-branch-2.8.005.patch > > > Maybe we'd like to extend our current jdiff tool for hadoop-common and hdfs > to YARN as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[
https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396608#comment-15396608
]
Hadoop QA commented on YARN-5436:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 14s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820582/YARN-5436.4.patch |
| JIRA Issue | YARN-5436 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1241e8c12390 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eb7ff0c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12531/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12531/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Race in AsyncDispatcher can cause random test failures in Tez(probably YARN
> also )
> --
>
> Key: YARN-5436
> URL: https://issues.apache.org/jira/browse/YARN-5436
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch,
> YARN-5436.4.patch
>
>
> In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also
> exists in
[jira] [Commented] (YARN-5351) ResourceRequest should take ExecutionType into account during comparison
[
https://issues.apache.org/jira/browse/YARN-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396603#comment-15396603
]
Kuhu Shukla commented on YARN-5351:
---
Great! Thanks [~subru], [~kkaranasos].
> ResourceRequest should take ExecutionType into account during comparison
>
>
> Key: YARN-5351
> URL: https://issues.apache.org/jira/browse/YARN-5351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0
>
> Attachments: YARN-5351.001.patch, YARN-5351.002.patch
>
>
> {{ExecutionTypeRequest}} should be taken into account in the {{compareTo}}
> method of {{ResourceRequest}}.
> Otherwise, in the {{ask}} list of the {{AMRMClientImpl}} we may incorrectly
> add pending container requests in the presence of both GUARANTEED and
> OPPORTUNISTIC containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5441) Fixing minor Scheduler test case failures
[
https://issues.apache.org/jira/browse/YARN-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396602#comment-15396602
]
Hadoop QA commented on YARN-5441:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820574/YARN-5441-v1.patch |
| JIRA Issue | YARN-5441 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 92d00e1bde61 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eb7ff0c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12530/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12530/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fixing minor Scheduler test case failures
> -
>
>
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396600#comment-15396600 ] Junping Du commented on YARN-5440: -- bq. if the user turn off timeline service but enable AHS in the config. In my understanding, if we enable AHS config but turn of timeline service, we will still create AHSClient to get finished application container information. The only different w/o the patch here is: this new added feature (log CLI enhancement) in 2.9 don't have to work by explicitly set a deprecated configuration of AHS. Instead, it also work with enabling ATS only. [~xgong], can you confirm this? > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > be used or not. But the AHSClientService is enabled by default when we start > the TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to rely on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5351) ResourceRequest should take ExecutionType into account during comparison
[
https://issues.apache.org/jira/browse/YARN-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396596#comment-15396596
]
Konstantinos Karanasos commented on YARN-5351:
--
Thank you for the comment, [~kshukla].
Jenkins did not ran those test cases, so I missed them.
Problem was that those test cases are using {{BuilderUtils}} and the
{{ExecutionTypeRequest}} was not properly initialed.
[~subru] opened YARN-5441 for this issue, and there is already a patch
available that fixes the issue.
> ResourceRequest should take ExecutionType into account during comparison
>
>
> Key: YARN-5351
> URL: https://issues.apache.org/jira/browse/YARN-5351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0
>
> Attachments: YARN-5351.001.patch, YARN-5351.002.patch
>
>
> {{ExecutionTypeRequest}} should be taken into account in the {{compareTo}}
> method of {{ResourceRequest}}.
> Otherwise, in the {{ask}} list of the {{AMRMClientImpl}} we may incorrectly
> add pending container requests in the presence of both GUARANTEED and
> OPPORTUNISTIC containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5351) ResourceRequest should take ExecutionType into account during comparison
[
https://issues.apache.org/jira/browse/YARN-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396592#comment-15396592
]
Subru Krishnan commented on YARN-5351:
--
I hit this too, have created YARN-5441 with the fix. Thanks.
> ResourceRequest should take ExecutionType into account during comparison
>
>
> Key: YARN-5351
> URL: https://issues.apache.org/jira/browse/YARN-5351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0
>
> Attachments: YARN-5351.001.patch, YARN-5351.002.patch
>
>
> {{ExecutionTypeRequest}} should be taken into account in the {{compareTo}}
> method of {{ResourceRequest}}.
> Otherwise, in the {{ask}} list of the {{AMRMClientImpl}} we may incorrectly
> add pending container requests in the presence of both GUARANTEED and
> OPPORTUNISTIC containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-5440: - Description: In YarnClient, depends on whether we enable yarn.timeline-service.generic-application-history.enabled, the AHSClient will be used or not. But the AHSClientService is enabled by default when we start the TimelineServer which means we are able to get history information for applications/applicationAttempts/containers by using ahsClient when the TimelineServer is running. So, we do not have to rely on this deprecated configuration to get history information by using YarnClient. was: In YarnClient, depends on whether we enable yarn.timeline-service.generic-application-history.enabled, the AHSClient will not be used. But the AHSClientService is enabled by default when we start the TimelineServer which means we are able to get history information for applications/applicationAttempts/containers by using ahsClient when the TimelineServer is running. So, we do not have to reply on this deprecated configuration to get history information by using YarnClient. > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > be used or not. But the AHSClientService is enabled by default when we start > the TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to rely on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396576#comment-15396576 ] Li Lu commented on YARN-5436: - Fix LGTM, +1. I'll wait for 24 hrs for more comments. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch, > YARN-5436.4.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[
https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396570#comment-15396570
]
Hadoop QA commented on YARN-5436:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 36s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 50s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820572/YARN-5436.3.patch |
| JIRA Issue | YARN-5436 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f1b6e06f22a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eb7ff0c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12529/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12529/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Race in AsyncDispatcher can cause random test failures in Tez(probably YARN
> also )
> --
>
> Key: YARN-5436
> URL: https://issues.apache.org/jira/browse/YARN-5436
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch,
> YARN-5436.4.patch
>
>
> In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also
> exists in
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.4.patch Uploaded the patch that doesn't use java 8 feature for branch-2 sake. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch, > YARN-5436.4.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5351) ResourceRequest should take ExecutionType into account during comparison
[
https://issues.apache.org/jira/browse/YARN-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396559#comment-15396559
]
Kuhu Shukla commented on YARN-5351:
---
I think this fix may be breaking the following tests in TestCapacityScheduler
with an NullPointerException.
testMoveAppViolateQueueState
testCapacityScheduler
testMoveAppSuccess
testResourceUpdateDecommissioningNode
testMoveAppForMoveToQueueWithFreeCap
testMoveAppQueueMetricsCheck
CC: [~kkaranasos]. Request for comments.
> ResourceRequest should take ExecutionType into account during comparison
>
>
> Key: YARN-5351
> URL: https://issues.apache.org/jira/browse/YARN-5351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0
>
> Attachments: YARN-5351.001.patch, YARN-5351.002.patch
>
>
> {{ExecutionTypeRequest}} should be taken into account in the {{compareTo}}
> method of {{ResourceRequest}}.
> Otherwise, in the {{ask}} list of the {{AMRMClientImpl}} we may incorrectly
> add pending container requests in the presence of both GUARANTEED and
> OPPORTUNISTIC containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396533#comment-15396533 ] Wangda Tan commented on YARN-3662: -- Latest patch LGTM as well, thanks [~subru]. > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5441) Fixing minor Scheduler test case failures
[
https://issues.apache.org/jira/browse/YARN-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-5441:
-
Attachment: YARN-5441-v1.patch
Attaching a patch that sets import {{ExecutionTypeRequest}} whenever
{{ResourceRequest}} is created through _BuilderUtils_.
> Fixing minor Scheduler test case failures
> -
>
> Key: YARN-5441
> URL: https://issues.apache.org/jira/browse/YARN-5441
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-5441-v1.patch
>
>
> YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}}
> comparator but the ResourceRequest object created via utility methods in few
> tests like *TestFifoScheduler* and *TestCapacityScheduler* do not set
> ExecutionTypeRequest which results in null pointer exceptions. This JIRA
> proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.3.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: (was: YARN-5436.3.patch) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396519#comment-15396519 ] Subru Krishnan commented on YARN-3662: -- Thanks [~vvasudev] for the sign-off. You are right, this is for branch YARN-2915. I'll commit it, just waiting to hear from [~leftnoteasy] as he had some feedback which I have also addressed. > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396514#comment-15396514 ] Zhiyuan Yang edited comment on YARN-5436 at 7/27/16 10:23 PM: -- Thanks [~gtCarrera9] for reviewing the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. was (Author: aplusplus): Thanks [~gtCarrera9] for review the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.3.patch Thanks [~gtCarrera9] for review the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5441) Fixing minor Scheduler test case failures
Subru Krishnan created YARN-5441:
Summary: Fixing minor Scheduler test case failures
Key: YARN-5441
URL: https://issues.apache.org/jira/browse/YARN-5441
Project: Hadoop YARN
Issue Type: Bug
Reporter: Subru Krishnan
Assignee: Subru Krishnan
YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}} comparator
but the ResourceRequest object created via utility methods in few tests like
*TestFifoScheduler* and *TestCapacityScheduler* do not set
ExecutionTypeRequest which results in null pointer exceptions. This JIRA
proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[
https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396512#comment-15396512
]
Li Lu commented on YARN-5436:
-
Thanks for the work [~aplusplus]! The {{FIXME}} part in AsyncDispatcher appears
to be confusing: There is no data race (per Java memory model's definition)
with the volatile variable {{drained}}. Maybe you'd like to rephrase a little
bit to express the potential nondeterminism?
Other changes in {{DrainedDispatcher}} appears to be fine to me.
> Race in AsyncDispatcher can cause random test failures in Tez(probably YARN
> also )
> --
>
> Key: YARN-5436
> URL: https://issues.apache.org/jira/browse/YARN-5436
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: YARN-5436.1.patch, YARN-5436.2.patch
>
>
> In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also
> exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never
> documented...). In YARN-2991, another DrainDispatcher bug was fixed by
> letting DrainDispatcher reuse some AsyncDispatcher method because
> AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and
> now similar race reappears in Tez unit tests (probably also YARN unit tests
> also).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396476#comment-15396476 ] Li Lu commented on YARN-5440: - I cannot reproduce the UT failure locally either. [~xgong] could you please open a separate JIRA to trace the UT failure? Patch generally LGTM. One question is that what will happen if the user turn off timeline service but enable AHS in the config? > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > not be used. But the AHSClientService is enabled by default when we start the > TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to reply on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[
https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396457#comment-15396457
]
Hadoop QA commented on YARN-5436:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 26s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820553/YARN-5436.2.patch |
| JIRA Issue | YARN-5436 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b5dd8fb851ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12528/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12528/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Race in AsyncDispatcher can cause random test failures in Tez(probably YARN
> also )
> --
>
> Key: YARN-5436
> URL: https://issues.apache.org/jira/browse/YARN-5436
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: YARN-5436.1.patch, YARN-5436.2.patch
>
>
> In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also
> exists in AsyncDispatcher (this was found and ignored in
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396435#comment-15396435 ] Xuan Gong commented on YARN-5440: - The testcase failures are not related > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > not be used. But the AHSClientService is enabled by default when we start the > TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to reply on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1079) Fix progress bar for long-lived services in YARN
[ https://issues.apache.org/jira/browse/YARN-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396406#comment-15396406 ] Chris Riccomini commented on YARN-1079: --- Sounds good to me. > Fix progress bar for long-lived services in YARN > > > Key: YARN-1079 > URL: https://issues.apache.org/jira/browse/YARN-1079 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Chris Riccomini > > YARN currently shows a progress bar for jobs in its web UI. This is > non-sensical for long-lived services, which have no concept of progress. For > example, with Samza, we have stream processors which run for an indefinite > amount of time (sometimes "forever"). > YARN should support jobs without a concept of progress. Some discussion about > this is on YARN-896. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.2.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[
https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396395#comment-15396395
]
Hadoop QA commented on YARN-5440:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
The patch generated 0 new + 30 unchanged - 1 fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 23s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820549/YARN-5440.1.patch |
| JIRA Issue | YARN-5440 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f0fa7c32b47a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12527/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12527/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12527/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12527/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[
https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396390#comment-15396390
]
Hadoop QA commented on YARN-5436:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The
patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820544/YARN-5436.1.patch |
| JIRA Issue | YARN-5436 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2325af36e225 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/12526/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12526/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12526/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12526/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Race in AsyncDispatcher can cause random test failures in Tez(probably YARN
> also )
>
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396345#comment-15396345 ] Zhiyuan Yang commented on YARN-5436: Upload the patch that fixes problems only in DrainDispatcher and documents minor race condition in AsyncDispatcher. Please help review. [~jianhe], [~rohithsharma], [~varun_saxena]. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5440: Attachment: YARN-5440.1.patch > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5440.1.patch > > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > not be used. But the AHSClientService is enabled by default when we start the > TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to reply on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-5440: --- Assignee: Xuan Gong > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > not be used. But the AHSClientService is enabled by default when we start the > TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to reply on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
[ https://issues.apache.org/jira/browse/YARN-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5440: Affects Version/s: 2.9.0 > Use AHSClient in YarnClient when TimelineServer is running > -- > > Key: YARN-5440 > URL: https://issues.apache.org/jira/browse/YARN-5440 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > > In YarnClient, depends on whether we enable > yarn.timeline-service.generic-application-history.enabled, the AHSClient will > not be used. But the AHSClientService is enabled by default when we start the > TimelineServer which means we are able to get history information for > applications/applicationAttempts/containers by using ahsClient when the > TimelineServer is running. So, we do not have to reply on this deprecated > configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
Xuan Gong created YARN-5440: --- Summary: Use AHSClient in YarnClient when TimelineServer is running Key: YARN-5440 URL: https://issues.apache.org/jira/browse/YARN-5440 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong In YarnClient, depends on whether we enable yarn.timeline-service.generic-application-history.enabled, the AHSClient will not be used. But the AHSClientService is enabled by default when we start the TimelineServer which means we are able to get history information for applications/applicationAttempts/containers by using ahsClient when the TimelineServer is running. So, we do not have to reply on this deprecated configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396348#comment-15396348 ] Zhiyuan Yang commented on YARN-5436: Race in AsyncDispatcher has been found and ignored in YARN-3887. Leave it there for now. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.1.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5416) TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped
[ https://issues.apache.org/jira/browse/YARN-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396285#comment-15396285 ] Eric Badger edited comment on YARN-5416 at 7/27/16 8:30 PM: Thanks for the response, Jason. That's exactly what I was thinking. I believe that it would mitigate the error that [~mitdesai] posted in his stack trace on YARN-1468. I'm fine with leaving this open since we have a patch here, but we need to make sure that we address all failures across both Jiras. was (Author: ebadger): Thanks for the response, Jason. That's exactly what I was thinking. I believe that it would mitigate the error that [~mitdesai] posted in his stack trace on YARN-1468. And as such, I think we should close this as a dup and continue our discussion over there. > TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently > due to not wait SchedulerApplicationAttempt to be stopped > > > Key: YARN-5416 > URL: https://issues.apache.org/jira/browse/YARN-5416 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: YARN-5416.patch > > > The test failure stack is: > Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 385.338 sec > <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > testRMRestartWaitForPreviousAMToFinish[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart) > Time elapsed: 43.134 sec <<< FAILURE! > java.lang.AssertionError: AppAttempt state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:86) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:594) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:1008) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:530) > This is due to the same issue that partially fixed in YARN-4968 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5416) TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped
[ https://issues.apache.org/jira/browse/YARN-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396285#comment-15396285 ] Eric Badger commented on YARN-5416: --- Thanks for the response, Jason. That's exactly what I was thinking. I believe that it would mitigate the error that [~mitdesai] posted in his stack trace on YARN-1468. And as such, I think we should close this as a dup and continue our discussion over there. > TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently > due to not wait SchedulerApplicationAttempt to be stopped > > > Key: YARN-5416 > URL: https://issues.apache.org/jira/browse/YARN-5416 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: YARN-5416.patch > > > The test failure stack is: > Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 385.338 sec > <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > testRMRestartWaitForPreviousAMToFinish[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart) > Time elapsed: 43.134 sec <<< FAILURE! > java.lang.AssertionError: AppAttempt state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:86) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:594) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:1008) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:530) > This is due to the same issue that partially fixed in YARN-4968 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396282#comment-15396282
]
Hadoop QA commented on YARN-5203:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 122 unchanged - 7 fixed = 123 total (was 129)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 22s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820531/YARN-5203.v5.patch |
| JIRA Issue | YARN-5203 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux db6372f932da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12525/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12525/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
[jira] [Assigned] (YARN-4889) Changes in AMRMClient for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-4889: - Assignee: Arun Suresh (was: Subru Krishnan) > Changes in AMRMClient for identifying resource-requests explicitly > -- > > Key: YARN-4889 > URL: https://issues.apache.org/jira/browse/YARN-4889 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Arun Suresh > > YARN-4879 proposes the notion of identifying allocate requests explicitly.. > This JIRA is to track the changes in AMRMClient to keep it wire compatible > with the changes. Please refer to the design doc in the parent JIRA for > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4889) Changes in AMRMClient for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396268#comment-15396268 ] Arun Suresh commented on YARN-4889: --- Taking this over from [~subru] > Changes in AMRMClient for identifying resource-requests explicitly > -- > > Key: YARN-4889 > URL: https://issues.apache.org/jira/browse/YARN-4889 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > > YARN-4879 proposes the notion of identifying allocate requests explicitly.. > This JIRA is to track the changes in AMRMClient to keep it wire compatible > with the changes. Please refer to the design doc in the parent JIRA for > details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Description: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never documented...). In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). (was: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also).) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5416) TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped
[
https://issues.apache.org/jira/browse/YARN-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396249#comment-15396249
]
Jason Lowe commented on YARN-5416:
--
bq. I think we can close this as dup of that. What do you think?
I don't care much if we want to close this one for that one or vice-versa, just
that we shouldn't keep both open. Since this is the one that has a patch, I'll
go ahead and comment on the patch here as Eric has also done.
bq. seems only necessary to wait before launch another AM immediately
I agree with Eric that it looks like another place was missed in the test.
IIUC we launch AM1 then wait for it to enter the FAILED state then launch AM2.
This patch changes that to do a more thorough wait before trying to launch AM2.
However later in the same test we wait for the second AM to fail and launch a
third attempt, which looks like the same case we're trying to fix -- waiting
for a previous AM to fully stop before immediately launching another attempt:
{code}
rm2.waitForState(am2.getApplicationAttemptId(), RMAppAttemptState.FAILED);
launchAM(rmApp, rm2, nm1);
Assert.assertEquals(3, rmApp.getAppAttempts().size());
{code}
> TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently
> due to not wait SchedulerApplicationAttempt to be stopped
>
>
> Key: YARN-5416
> URL: https://issues.apache.org/jira/browse/YARN-5416
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Minor
> Attachments: YARN-5416.patch
>
>
> The test failure stack is:
> Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 385.338 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 43.134 sec <<< FAILURE!
> java.lang.AssertionError: AppAttempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:86)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:594)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:1008)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:530)
> This is due to the same issue that partially fixed in YARN-4968
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-1468) TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
[
https://issues.apache.org/jira/browse/YARN-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396233#comment-15396233
]
Eric Badger edited comment on YARN-1468 at 7/27/16 7:51 PM:
[~djp], you are correct. When I encountered the error in a test failure it was
in the place as I explained it above. However, that failure is a different one
than the stack trace that [~mitdesai] included. I thought that they were the
same based on the overall structure of the exception, but that is not the case.
I should've added my own stack trace to make that clear on this jira and to
myself. I think that both point to issues, however. I've seen this test fail in
a multitude of different ways.
was (Author: ebadger):
[~djp], you are correct. When I encountered the error in a test failure it was
in the place as I explained it above. However, that failure is a different one
than the stack trace that [~mitdesai] included. I should've included my own
stack trace to make that point clear. I think that both point to issues,
however. I've seen this test fail in a multitude of different ways.
> TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
>
>
> Key: YARN-1468
> URL: https://issues.apache.org/jira/browse/YARN-1468
> Project: Hadoop YARN
> Issue Type: Test
> Components: resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> Log is as following:
> {code}
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.968 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 44.197 sec <<< FAILURE!
> junit.framework.AssertionFailedError: AppAttempt state is not correct
> (timedout) expected: but was:
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:292)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:464)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1468) TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
[
https://issues.apache.org/jira/browse/YARN-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396233#comment-15396233
]
Eric Badger commented on YARN-1468:
---
[~djp], you are correct. When I encountered the error in a test failure it was
in the place as I explained it above. However, that failure is a different one
than the stack trace that [~mitdesai] included. I should've included my own
stack trace to make that point clear. I think that both point to issues,
however. I've seen this test fail in a multitude of different ways.
> TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
>
>
> Key: YARN-1468
> URL: https://issues.apache.org/jira/browse/YARN-1468
> Project: Hadoop YARN
> Issue Type: Test
> Components: resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> Log is as following:
> {code}
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.968 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 44.197 sec <<< FAILURE!
> junit.framework.AssertionFailedError: AppAttempt state is not correct
> (timedout) expected: but was:
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:292)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:464)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[
https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396200#comment-15396200
]
Jason Lowe commented on YARN-4280:
--
Thanks for updating the patch, Kuhu!
The copy constructor for ResourceLimits doesn't copy all of the fields,
specifically allowPreempt is missing. Intentional? If so, maybe an alternate
method that generates a new ResourceLimit should be used rather than what looks
like a copy constructor? Otherwise it's going to be confusing to others that
assume it really is a full-blown copy of all fields or whether or not the copy
constructor should be updated when a new field is added to the class.
Nit: Adding a "_SKIPPED" suffix to every enum is somewhat redundant when the
enum type is SkippedType. These could be simplified to NONE, QUEUE_LIMIT, and
OTHER.
Similar copy constructor comment for CSAssignment since increaseAllocation and
containersToKill are not copied.
The debug log statement in ParentQueue needs to be wrapped with a log debug
check otherwise we'll always do the expensive string construction even when we
won't log it.
Do we realy want the folllowing to execute when the assignments skipped type is
OTHER_SKIPPED or the assignment is not the null assigment? Seems like if there
is a real assignment of resources we should return it without overriding the
skip type. Or am I missing a scenario where we need to set QUEUE_LIMIT_SKIPPED
even when there's a real assignment in a sibling queue?
{code}
if(!skippedType.equals(assignment.getSkippedType())) {
//Make a copy of the assignment to avoid changing when
// returned assignment == NULL_ASSIGNMENT
CSAssignment csAssignment = new CSAssignment(assignment);
csAssignment.setSkippedType(skippedType);
return csAssignment;
}
{code}
Why does MockNodes have a new static method that creates a Resource? Seems to
have nothing to do with the rest of the class and creates a maintenance burden
when the Resource class gets updated in the future. Anyone who needs a Resource
should be calling Resource.newInstance.
> CapacityScheduler reservations may not prevent indefinite postponement on a
> busy cluster
>
>
> Key: YARN-4280
> URL: https://issues.apache.org/jira/browse/YARN-4280
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.6.1, 2.8.0, 2.7.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: YARN-4280-branch-2.009.patch,
> YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch,
> YARN-4280-branch-2.8.003.patch, YARN-4280.001.patch, YARN-4280.002.patch,
> YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch,
> YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch,
> YARN-4280.008_.patch, YARN-4280.009.patch
>
>
> Consider the following scenario:
> There are 2 queues A(25% of the total capacity) and B(75%), both can run at
> total cluster capacity. There are 2 applications, appX that runs on Queue A,
> always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2
> GB containers.
> The user limit is high enough for the application to reach 100% of the
> cluster resource.
> appX is running at total cluster capacity, full with 1G containers releasing
> only one container at a time. appY comes in with a request of 2GB container
> but only 1 GB is free. Ideally, since appY is in the underserved queue, it
> has higher priority and should reserve for its 2 GB request. Since this
> request puts the alloc+reserve above total capacity of the cluster,
> reservation is not made. appX comes in with a 1GB request and since 1GB is
> still available, the request is allocated.
> This can continue indefinitely causing priority inversion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ellen Hui updated YARN-5203:
Attachment: YARN-5203.v5.patch
Add missing license to ExecutionTypeRequestInfo
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug, incompatible
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch, YARN-5203.v3.patch, YARN-5203.v4.patch, YARN-5203.v5.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5434) Add -client|server argument for graceful decom
[
https://issues.apache.org/jira/browse/YARN-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396197#comment-15396197
]
Hadoop QA commented on YARN-5434:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 2 new + 118 unchanged - 10 fixed = 120 total (was 128) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 21s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820525/YARN-5343.003.patch |
| JIRA Issue | YARN-5434 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 92320712cc88 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12524/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12524/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12524/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12524/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12524/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[ https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396195#comment-15396195 ] Jason Lowe commented on YARN-5438: -- Ah, thanks Rohith. My bad, I missed that it was creating the filesystem in a way that essentially avoids the cache. +1 lgtm. Will commit this tomorrow if there are no objections. > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > > > Key: YARN-5438 > URL: https://issues.apache.org/jira/browse/YARN-5438 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.0, 2.7.3 >Reporter: Karam Singh >Assignee: Rohith Sharma K S > Attachments: YARN-5438.0.patch > > > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl, > FileSystem.newInstance is invoked and is not closed. Causing over time > Filesystem instances getting accumulated in long runninh Client (like > Hiveserver2), finally causing them to OOM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[
https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396150#comment-15396150
]
Rohith Sharma K S commented on YARN-5438:
-
bq. Since the filesystem cache will implicitly link what looks like two
separate creations of a filesystem to a single instance, closing one will break
any subsequent use of the other.
If the user creates file system object using api {{FileSystem#newInstance}}
with in the JVM then always new *FS* object is given. For every *newInstance*
api call, object created using the combination of URI, Conf and *UniqueKey*. If
FS object is created using {{FS#get}} then this api search from cache. This API
always creates object with combination of URI and CONF only. So mainly it
matters how the FS object is being created.
Basically closing one instance which is created using
{{FileSystem#newInstance}} should not affect other FS object which is created
using {{FS#get}}. And also note that if two FS objects are created using
{{FS#get}} then closing one will definitely affect other FS object.
> TimelineClientImpl leaking FileSystem Instances causing Long running services
> like HiverServer2 daemon going OOM
>
>
> Key: YARN-5438
> URL: https://issues.apache.org/jira/browse/YARN-5438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Karam Singh
>Assignee: Rohith Sharma K S
> Attachments: YARN-5438.0.patch
>
>
> TimelineClientImpl leaking FileSystem Instances causing Long running services
> like HiverServer2 daemon going OOM
> In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl,
> FileSystem.newInstance is invoked and is not closed. Causing over time
> Filesystem instances getting accumulated in long runninh Client (like
> Hiveserver2), finally causing them to OOM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5434) Add -client|server argument for graceful decom
[
https://issues.apache.org/jira/browse/YARN-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-5434:
Attachment: YARN-5343.003.patch
The 003 patch updates the description to talk about blocking vs non-blocking
and the infinite timeout.
> Add -client|server argument for graceful decom
> --
>
> Key: YARN-5434
> URL: https://issues.apache.org/jira/browse/YARN-5434
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: YARN-5343.001.patch, YARN-5343.002.patch,
> YARN-5343.003.patch
>
>
> We should add {{-client|server}} argument to allow the user to specify if
> they want to use the client-side graceful decom tracking, or the server-side
> tracking (YARN-4676).
> Even though the server-side tracking won't go into 2.8, we should add the
> arguments to 2.8 for compatibility between 2.8 and 2.9, when it's added. In
> 2.8, using {{-server}} would just throw an Exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5434) Add -client|server argument for graceful decom
[
https://issues.apache.org/jira/browse/YARN-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396123#comment-15396123
]
Junping Du commented on YARN-5434:
--
Thanks [~rkanter] for working out a patch here. The 002 patch looks good in
overall. A minor comments, I think we would have a non-blocking call for server
side tracking and a blocking call for client side tracking (unless not
specified timeout value which is -1 by default). Isn't it? If so, better to
mention it (blocking/non-blocking behavior) in rmadmin CLI help messages?
> Add -client|server argument for graceful decom
> --
>
> Key: YARN-5434
> URL: https://issues.apache.org/jira/browse/YARN-5434
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: YARN-5343.001.patch, YARN-5343.002.patch
>
>
> We should add {{-client|server}} argument to allow the user to specify if
> they want to use the client-side graceful decom tracking, or the server-side
> tracking (YARN-4676).
> Even though the server-side tracking won't go into 2.8, we should add the
> arguments to 2.8 for compatibility between 2.8 and 2.9, when it's added. In
> 2.8, using {{-server}} would just throw an Exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Description: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). (was: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher extend AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also).) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher reuse some > AsyncDispatcher method because AsyncDispatcher doesn't have such issue. > However, this shadows YARN-2264, and now similar race reappears in Tez unit > tests (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Comment: was deleted (was: Data race can cause RM stop without handling last enqueued event.) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[ https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396044#comment-15396044 ] Jason Lowe commented on YARN-5438: -- Thanks for the patch, Rohith! This probably works for the HiveServer2 case iff the server never tries to use the filesystem after the timeline client is closed. However the timeline client is not just used by HS2, and I think this patch will be problematic for any code that could still use the filesystem after the timeline client is closed. Since the filesystem cache will implicitly link what looks like two separate creations of a filesystem to a single instance, closing one will break any subsequent use of the other. This makes me think HS2 is missing a closeAllforUGI call in it somewhere to make sure when it's done for a certain user it cleans up all the filesystems associated with that user. It also makes me wonder why we haven't implemented a reference-counting cache for the filesystem by now. > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > > > Key: YARN-5438 > URL: https://issues.apache.org/jira/browse/YARN-5438 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.0, 2.7.3 >Reporter: Karam Singh >Assignee: Rohith Sharma K S > Attachments: YARN-5438.0.patch > > > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl, > FileSystem.newInstance is invoked and is not closed. Causing over time > Filesystem instances getting accumulated in long runninh Client (like > Hiveserver2), finally causing them to OOM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396054#comment-15396054 ] Zhiyuan Yang commented on YARN-5436: Data race can cause RM stop without handling last enqueued event. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396052#comment-15396052 ] Zhiyuan Yang commented on YARN-5436: Data race can cause RM stop without handling last enqueued event. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[
https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396018#comment-15396018
]
Hadoop QA commented on YARN-5438:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 59s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820503/YARN-5438.0.patch |
| JIRA Issue | YARN-5438 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 872a7339fda9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12523/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12523/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TimelineClientImpl leaking FileSystem Instances causing Long running services
> like HiverServer2 daemon going OOM
>
>
> Key: YARN-5438
> URL: https://issues.apache.org/jira/browse/YARN-5438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.8.0, 2.7.3
>
[jira] [Comment Edited] (YARN-5137) Make DiskChecker pluggable in NodeManager
[
https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396008#comment-15396008
]
Yufei Gu edited comment on YARN-5137 at 7/27/16 5:19 PM:
-
Hi [~vvasudev], thanks for the review.
Since {{conf}} in class {{TestLinuxContainerExecutorWithMocks}} is changed to
{{YarnConfiguration}} instead of {{Configuration}}, it adds one item(local
dir) into {{result}}. So it is 19 instead of 18 now.
I changed {{conf}} to an instance {{YarnConfiguration}} since the class
{{TestLinuxContainerExecutorWithMocks}} has an unstable assumption that it
won't load any yarn configuration files, e.g., yarn-default.xml and
yarn-site.xml.
was (Author: yufeigu):
Hi [~vvasudev], thanks for the review.
Since {{conf}} in class {{TestLinuxContainerExecutorWithMocks}} is changed to
{{YarnConfiguration}} instead of {{Configuration}}, it adds one item(local
dir) into {{result}}. I changed {{conf}} to an instance {{YarnConfiguration}}
since the class {{TestLinuxContainerExecutorWithMocks}} has an unstable
assumption that it won't load any yarn configuration files, e.g.,
yarn-default.xml and yarn-site.xml.
> Make DiskChecker pluggable in NodeManager
> -
>
> Key: YARN-5137
> URL: https://issues.apache.org/jira/browse/YARN-5137
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Ray Chiang
>Assignee: Yufei Gu
> Labels: supportability
> Attachments: YARN-5137.001.patch, YARN-5137.002.patch,
> YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch
>
>
> It would be nice to have the option for a DiskChecker that has more
> sophisticated checking capabilities. In order to do this, we would first
> need DiskChecker to be pluggable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5137) Make DiskChecker pluggable in NodeManager
[
https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396008#comment-15396008
]
Yufei Gu commented on YARN-5137:
Hi [~vvasudev], thanks for the review.
Since {{conf}} in class {{TestLinuxContainerExecutorWithMocks}} is changed to
{{YarnConfiguration}} instead of {{Configuration}}, it adds one item(local
dir) into {{result}}. I changed {{conf}} to an instance {{YarnConfiguration}}
since the class {{TestLinuxContainerExecutorWithMocks}} has an unstable
assumption that it won't load any yarn configuration files, e.g.,
yarn-default.xml and yarn-site.xml.
> Make DiskChecker pluggable in NodeManager
> -
>
> Key: YARN-5137
> URL: https://issues.apache.org/jira/browse/YARN-5137
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Ray Chiang
>Assignee: Yufei Gu
> Labels: supportability
> Attachments: YARN-5137.001.patch, YARN-5137.002.patch,
> YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch
>
>
> It would be nice to have the option for a DiskChecker that has more
> sophisticated checking capabilities. In order to do this, we would first
> need DiskChecker to be pluggable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters
[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395992#comment-15395992 ] Basha Shaik commented on YARN-4327: --- I am encountering the same issue. When I use yarn.timeline-service.http-authentication.type= Kerberos, I am able to execute the job with execution engine Tez. But I can't see the Tez view. If I change yarn.timeline-service.http-authentication.type=simple, then I can see Tez, but can't execute the job. > RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters > -- > > Key: YARN-4327 > URL: https://issues.apache.org/jira/browse/YARN-4327 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver >Affects Versions: 2.7.1 > Environment: hadoop 2.7.1hdfs,yarn, mrhistoryserver, ATS all use > kerberos security. > conf like this: > > hadoop.security.authorization > true > Is service-level authorization enabled? > > > hadoop.security.authentication > kerberos > Possible values are simple (no authentication), and kerberos > > >Reporter: zhangshilong > > bin hadoop 2.7.1 > ATS conf like this: > > yarn.timeline-service.http-authentication.type > simple > > > yarn.timeline-service.http-authentication.kerberos.principal > HTTP/_h...@xxx.com > > > yarn.timeline-service.http-authentication.kerberos.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.principal > xxx/_h...@xxx.com > > > yarn.timeline-service.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.best-effort > true > > > yarn.timeline-service.enabled > true > > > I'd like to allow everyone to access ATS from HTTP as RM,HDFS. > client can submit job to RM and add TIMELINE_DELEGATION_TOKEN to AM > Context, but RM can not renew TIMELINE_DELEGATION_TOKEN and make application > to failure. > RM logs: > 2015-11-03 11:58:38,191 WARN > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: > Unable to add the application to the delegation token renewer. > java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, > Service: 10.12.38.4:8188, Ident: (owner=yarn-test, renewer=yarn-test, > realUser=, issueDate=1446523118046, maxDate=1447127918046, sequenceNumber=9, > masterKeyId=2) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:439) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:78) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:847) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:828) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: HTTP status [500], message [Null user] > at > org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:287) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:212) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:396) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:378) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:451) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:183) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:466) > at >
[jira] [Updated] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[ https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-5438: Attachment: YARN-5438.0.patch Updated patch for closing the FileSystem while stopping TimelineClient > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > > > Key: YARN-5438 > URL: https://issues.apache.org/jira/browse/YARN-5438 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.0, 2.7.3 >Reporter: Karam Singh >Assignee: Rohith Sharma K S > Attachments: YARN-5438.0.patch > > > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl, > FileSystem.newInstance is invoked and is not closed. Causing over time > Filesystem instances getting accumulated in long runninh Client (like > Hiveserver2), finally causing them to OOM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5439) ATS out log file keeps on getting filled WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" exceptio
[ https://issues.apache.org/jira/browse/YARN-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karam Singh updated YARN-5439: -- Description: While running various different type of using tez found that ATS out log file keeps on getting filled with following type of excpetions: com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve SEVERE: null java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) at java.lang.Class.newInstance(Class.java:436) at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120) at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.RestCsrfPreventionFilter$ServletFilterHttpInteraction.proceed(RestCsrfPreventionFilter.java:269) at org.apache.hadoop.security.http.RestCsrfPreventionFilter.handleHttpInteraction(RestCsrfPreventionFilter.java:197) at org.apache.hadoop.security.http.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:209) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:614) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:294) at
[jira] [Created] (YARN-5439) ATS out log file keeps on getting filled WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" exceptio
Karam Singh created YARN-5439: - Summary: ATS out log file keeps on getting filled WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" exceptions Key: YARN-5439 URL: https://issues.apache.org/jira/browse/YARN-5439 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.8.0, 2.7.3 Reporter: Karam Singh While running various different type of using tez found that ATS out log file keeps on getting filled with following type of excpetions: Jun 13, 2016 1:43:07 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve SEVERE: null java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) at java.lang.Class.newInstance(Class.java:436) at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120) at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.security.http.RestCsrfPreventionFilter$ServletFilterHttpInteraction.proceed(RestCsrfPreventionFilter.java:269) at org.apache.hadoop.security.http.RestCsrfPreventionFilter.handleHttpInteraction(RestCsrfPreventionFilter.java:197) at org.apache.hadoop.security.http.RestCsrfPreventionFilter.doFilter(RestCsrfPreventionFilter.java:209) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at
[jira] [Assigned] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
[ https://issues.apache.org/jira/browse/YARN-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-5438: --- Assignee: Rohith Sharma K S > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > > > Key: YARN-5438 > URL: https://issues.apache.org/jira/browse/YARN-5438 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.0, 2.7.3 >Reporter: Karam Singh >Assignee: Rohith Sharma K S > > TimelineClientImpl leaking FileSystem Instances causing Long running services > like HiverServer2 daemon going OOM > In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl, > FileSystem.newInstance is invoked and is not closed. Causing over time > Filesystem instances getting accumulated in long runninh Client (like > Hiveserver2), finally causing them to OOM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1468) TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
[
https://issues.apache.org/jira/browse/YARN-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395946#comment-15395946
]
Junping Du commented on YARN-1468:
--
Hi [~ebadger], that's indeed a nice catch! I think we should fix this test
issue. However, this seems not to be the same issue as exception stack so far.
Do I miss something?
> TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
>
>
> Key: YARN-1468
> URL: https://issues.apache.org/jira/browse/YARN-1468
> Project: Hadoop YARN
> Issue Type: Test
> Components: resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> Log is as following:
> {code}
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.968 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 44.197 sec <<< FAILURE!
> junit.framework.AssertionFailedError: AppAttempt state is not correct
> (timedout) expected: but was:
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:292)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:464)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395935#comment-15395935 ] Shane Kumpf commented on YARN-5428: --- new patch has been uploaded. The failures are the same as before (method line length and unrelated nodemanager test failure). > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5438) TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM
Karam Singh created YARN-5438: - Summary: TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM Key: YARN-5438 URL: https://issues.apache.org/jira/browse/YARN-5438 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.8.0, 2.7.3 Reporter: Karam Singh TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM In org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl, FileSystem.newInstance is invoked and is not closed. Causing over time Filesystem instances getting accumulated in long runninh Client (like Hiveserver2), finally causing them to OOM -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5416) TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped
[ https://issues.apache.org/jira/browse/YARN-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395930#comment-15395930 ] Junping Du commented on YARN-5416: -- bq. This looks like an exact dup of YARN-1468 which you also filed. Are they actually different? Oh. no. YARN-1468 is a very old jira and out of my radar for some reason (I didn't notice recent comments from Eric there). I think we can close this as dup of that. What do you think? bq. Junping Du, is there any reason why we would only add the waitSchedulerApplicationAttemptStopped call for the first app attempt, but not for the subsequent ones? Hi Eric, this is just following the pattern we applied in YARN-4968 which seems only necessary to wait before launch another AM immediately - that is exactly where the exception happens. Do you think there are other places we should add? > TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently > due to not wait SchedulerApplicationAttempt to be stopped > > > Key: YARN-5416 > URL: https://issues.apache.org/jira/browse/YARN-5416 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: YARN-5416.patch > > > The test failure stack is: > Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > Tests run: 54, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 385.338 sec > <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart > testRMRestartWaitForPreviousAMToFinish[0](org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart) > Time elapsed: 43.134 sec <<< FAILURE! > java.lang.AssertionError: AppAttempt state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:86) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:594) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:1008) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:530) > This is due to the same issue that partially fixed in YARN-4968 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5431) TimeLineReader daemon start should allow to pass its own reader opts
[
https://issues.apache.org/jira/browse/YARN-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395931#comment-15395931
]
Varun Saxena commented on YARN-5431:
LGTM.
[~aw], you have any further comments ?
> TimeLineReader daemon start should allow to pass its own reader opts
>
>
> Key: YARN-5431
> URL: https://issues.apache.org/jira/browse/YARN-5431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts, timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5431.0.patch, YARN-5431.1.patch, YARN-5431.2.patch
>
>
> In yarn script , timelinereader doesn't allow to pass reader_opts.
> {code}
> timelinereader)
> HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
> HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[
https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395917#comment-15395917
]
Hadoop QA commented on YARN-5428:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 2
new + 237 unchanged - 1 fixed = 239 total (was 238) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 6s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.nodemanager.TestDirectoryCollection |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12820484/YARN-5428.003.patch |
| JIRA Issue | YARN-5428 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 19789f0fc0cf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 54fe17a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12522/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12522/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit test logs |
[jira] [Commented] (YARN-5342) Improve non-exclusive node partition resource allocation in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395909#comment-15395909
]
Hadoop QA commented on YARN-5342:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
3s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 13 unchanged - 1 fixed = 13 total (was 14) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 44s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_101. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 46s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_101 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5af2af1 |
| JIRA Patch URL |
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395858#comment-15395858 ] Sunil G commented on YARN-4945: --- Extremely sorry for the delay. bq. With the proposed design, is it possible for an app to be preempted that is below its user limit? Yes, we should consider this case and should not allow preemption. However there could be bounder line scenarios where high priority apps has more demand and we might preempt apps (few containers) which may just over user-limit. Such deadzones will also be considered to avoid over kills. We can see how existing tuning configs can be made use of intra-queue scenarios too > [Umbrella] Capacity Scheduler Preemption Within a queue > --- > > Key: YARN-4945 > URL: https://issues.apache.org/jira/browse/YARN-4945 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan > > This is umbrella ticket to track efforts of preemption within a queue to > support features like: > YARN-2009. YARN-2113. YARN-4781. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-5428: -- Attachment: YARN-5428.003.patch > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4948) Support node labels store in zookeeper
[ https://issues.apache.org/jira/browse/YARN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395812#comment-15395812 ] jialei weng commented on YARN-4948: --- HI, [~Naganarasimha], Do you have any update for this? or need I to add some document? > Support node labels store in zookeeper > -- > > Key: YARN-4948 > URL: https://issues.apache.org/jira/browse/YARN-4948 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: jialei weng >Assignee: jialei weng > Attachments: YARN-4948.001.patch, YARN-4948.002.patch, > YARN-4948.003.patch, YARN-4948.006.patch, YARN-4948.007.patch > > > Support node labels store in zookeeper. The main scenario for this is to give > a way to decouple yarn with HDFS. Since nodelabel is a very important data > for yarn, if hdfs down, yarn will fail to start up,too. So it is meaningful > for make yarn much independence when user serve both yarn and HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5432) Lock already held by another process while LevelDB cache store creation for dag
[ https://issues.apache.org/jira/browse/YARN-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395762#comment-15395762 ] Junping Du commented on YARN-5432: -- bq. So, instead of using refcounts and handle corner cases like the one reported here, another solution is to insist on the app cache size. +1. KISS (keep it simple & stupid) rules still works here. 003 patch LGTM. Will commit it later today if not further comments from others. > Lock already held by another process while LevelDB cache store creation for > dag > --- > > Key: YARN-5432 > URL: https://issues.apache.org/jira/browse/YARN-5432 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.0, 2.7.3 >Reporter: Karam Singh >Assignee: Li Lu > Attachments: YARN-5432-trunk.001.patch, YARN-5432-trunk.002.patch, > YARN-5432-trunk.003.patch > > > While running ATS stress tests, 15 concurrent ATS reads (python thread > which gives ws/v1/time/TEZ_DAG_ID, > ws/v1/time/TEZ_VERTEX_DI?primaryFilter=TEZ_DAG_ID: etc) calls. > Note: Summary store for ATSv1.5 is RLD, but as we for each dag/application > ATS also creates leveldb cache when vertex/task/taskattempts information is > queried from ATS. > > Getting following type of excpetion very frequently in ATS logs :- > 2016-07-23 00:01:56,089 [1517798697@qtp-1198158701-850] INFO > org.apache.hadoop.service.AbstractService: Service > LeveldbCache.timelineEntityGroupId_1469090881194_4832_application_1469090881194_4832 > failed in state INITED; cause: > org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock > /grid/4/yarn_ats/atsv15_rld/timelineEntityGroupId_1469090881194_4832_application_1469090881194_4832-timeline-cache.ldb/LOCK: > already held by process > org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock > /grid/4/yarn_ats/atsv15_rld/timelineEntityGroupId_1469090881194_4832_application_1469090881194_4832-timeline-cache.ldb/LOCK: > already held by process > at > org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) > at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) > at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) > at > org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.serviceInit(LevelDBCacheTimelineStore.java:108) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:113) > at > org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:1021) > at > org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:936) > at > org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:989) > at > org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:1041) > at > org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:168) > at > org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:138) > at > org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:117) > at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at >
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395734#comment-15395734 ] Shane Kumpf commented on YARN-5428: --- While it would be a good idea to pass this to all docker commands, it is not necessary for the use case in mind. The types of configuration typically stored in the client config are formatting rules, http headers, credentials, and shortcuts. The main one we are concerned with is credentials, which are only needed by a docker run/pull, and properly handled by container executor, as the --config option is added to the docker command file. Hopefully we can address a broader subset of docker client commands once container executor is refactored. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395711#comment-15395711 ] Varun Vasudev commented on YARN-5428: - [~shaneku...@gmail.com] - sorry I forgot to ask - does the client configuration directory have to be passed for commands like docker pull, docker inspect, etc? We run these commands in the container-executor. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5137) Make DiskChecker pluggable in NodeManager
[
https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395707#comment-15395707
]
Varun Vasudev commented on YARN-5137:
-
Can you explain why this was needed -
{code}
- Assert.assertEquals(result.size(), 18);
+ Assert.assertEquals(result.size(), 19);
{code}
The other change looks fine to me.
> Make DiskChecker pluggable in NodeManager
> -
>
> Key: YARN-5137
> URL: https://issues.apache.org/jira/browse/YARN-5137
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Ray Chiang
>Assignee: Yufei Gu
> Labels: supportability
> Attachments: YARN-5137.001.patch, YARN-5137.002.patch,
> YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch
>
>
> It would be nice to have the option for a DiskChecker that has more
> sophisticated checking capabilities. In order to do this, we would first
> need DiskChecker to be pluggable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395639#comment-15395639 ] Shane Kumpf commented on YARN-5428: --- Thanks for the review [~vvasudev]! I'll get a new patch uploaded shortly with that fix. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[
https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395628#comment-15395628
]
Varun Vasudev commented on YARN-5428:
-
Thanks for the patch [~shaneku...@gmail.com]. Some feedback -
1)
{code}
+String clientConfigDir = conf.get(
+YarnConfiguration.NM_DOCKER_CLIENT_CONFIG_DIRECTORY);
+if(clientConfigDir != null) {
+ runCommand.setClientConfigDir(clientConfigDir);
+}
+
{code}
{code}
+
+ String clientConfigDir = conf.get(
+ YarnConfiguration.NM_DOCKER_CLIENT_CONFIG_DIRECTORY);
+ if(clientConfigDir != null) {
+stopCommand.setClientConfigDir(clientConfigDir);
+ }
+
{code}
Can we just move the null check into setClientConfigDir itself?
Rest of the patch looks good to me.
> Allow for specifying the docker client configuration directory
> --
>
> Key: YARN-5428
> URL: https://issues.apache.org/jira/browse/YARN-5428
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-5428.001.patch, YARN-5428.002.patch
>
>
> The docker client allows for specifying a configuration directory that
> contains the docker client's configuration. It is common to store "docker
> login" credentials in this config, to avoid the need to docker login on each
> cluster member.
> By default the docker client config is $HOME/.docker/config.json on Linux.
> However, this does not work with the current container executor user
> switching and it may also be desirable to centralize this configuration
> beyond the single user's home directory.
> Note that the command line arg is for the configuration directory NOT the
> configuration file.
> This change will be needed to allow YARN to automatically pull images at
> localization time or within container executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5342) Improve non-exclusive node partition resource allocation in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5342: -- Attachment: YARN-5342-branch-2.8.001.patch Thanks [~leftnoteasy] for the review and commit and thanks [~Naganarasimha Garla] for the review. Also attaching branch-2.8 patch. > Improve non-exclusive node partition resource allocation in Capacity Scheduler > -- > > Key: YARN-5342 > URL: https://issues.apache.org/jira/browse/YARN-5342 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: YARN-5342-branch-2.8.001.patch, YARN-5342.1.patch, > YARN-5342.2.patch, YARN-5342.3.patch, YARN-5342.4.patch > > > In the previous implementation, one non-exclusive container allocation is > possible when the missed-opportunity >= #cluster-nodes. And > missed-opportunity will be reset when container allocated to any node. > This will slow down the frequency of container allocation on non-exclusive > node partition: *When a non-exclusive partition=x has idle resource, we can > only allocate one container for this app in every > X=nodemanagers.heartbeat-interval secs for the whole cluster.* > In this JIRA, I propose a fix to reset missed-opporunity only if we have >0 > pending resource for the non-exclusive partition OR we get allocation from > the default partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
