[
https://issues.apache.org/jira/browse/TEZ-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892017#comment-16892017
]
TezQA commented on TEZ-4080:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-0.9.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
29s{color} | {color:red} root in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
22s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} branch-0.9.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} tez-api in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} tez-dag in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-dag in the patch failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color}
| {color:red} tez-api in the patch failed with JDK v1.8.0_212. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color}
| {color:red} tez-dag in the patch failed with JDK v1.8.0_212. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color}
| {color:red} tez-api in the patch failed with JDK v9-internal. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} tez-dag in the patch failed with JDK v9-internal. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} tez-api: The patch generated 32 new + 730
unchanged - 38 fixed = 762 total (was 768) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} tez-dag: The patch generated 3 new + 844
unchanged - 2 fixed = 847 total (was 846) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed with JDK v1.8.0_212.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red} tez-dag in the patch failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 14s{color}
| {color:red} tez-api in the patch failed with JDK v9-internal. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s{color}
| {color:red} tez-dag in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.0 Server=19.03.0 Image:yetus/tez:date2019-07-24 |
| JIRA Issue | TEZ-4080 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975658/TEZ-4080.branch-0.9.1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 40b56829644b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-0.9.1 / 23b58b2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 9-internal |
| Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_212
/usr/lib/jvm/java-9-openjdk-amd64:9-internal |
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-compile-tez-api-jdk1.8.0_212.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-compile-tez-dag-jdk1.8.0_212.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-compile-tez-api-jdk9-internal.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-compile-tez-dag-jdk9-internal.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-findbugs-tez-api.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-findbugs-tez-dag.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-javadoc-tez-api-jdk1.8.0_212.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-javadoc-tez-dag-jdk1.8.0_212.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-javadoc-tez-api-jdk9-internal.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/branch-javadoc-tez-dag-jdk9-internal.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-mvninstall-tez-api.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-mvninstall-tez-dag.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-api-jdk1.8.0_212.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-dag-jdk1.8.0_212.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-api-jdk1.8.0_212.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-dag-jdk1.8.0_212.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-api-jdk9-internal.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-dag-jdk9-internal.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-api-jdk9-internal.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-compile-tez-dag-jdk9-internal.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/diff-checkstyle-tez-api.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/diff-checkstyle-tez-dag.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-findbugs-tez-api.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-findbugs-tez-dag.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-javadoc-tez-api-jdk1.8.0_212.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-javadoc-tez-dag-jdk1.8.0_212.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-javadoc-tez-api-jdk9-internal.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-javadoc-tez-dag-jdk9-internal.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-unit-tez-api-jdk9-internal.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/artifact/out/patch-unit-tez-dag-jdk9-internal.txt
|
| JDK v9-internal Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/testReport/ |
| Max. process+thread count | 101 (vs. ulimit of 10000) |
| modules | C: tez-api tez-dag U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/151/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TezClient should close FileSystem objects to prevent leak
> ---------------------------------------------------------
>
> Key: TEZ-4080
> URL: https://issues.apache.org/jira/browse/TEZ-4080
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Zoltan Haindrich
> Priority: Major
> Attachments: TEZ-4080.branch-0.9.1.patch
>
>
> When opening/closing a lot of tez clients; some FileSystem object references
> are retained even after the client is closed - due to the fact the
> FileSystem has a "cache" which collects all open FileSystem objects - to be
> able to close all of them from a single shutdownhook.
> Not closing these FileSystem objects causes them to pile up in the "cache"
> which has hard references to them
> In a simple hive test which was run with 150M of memory; these "lost"
> filesystem objects could result in an OOM after ~170 sessions.
> A sample creation stack trace of a FileSystem object:
> {code}
> at
> org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem.<init>(ProxyLocalFileSystem.java:49)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3353)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:502)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2663)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
