[
https://issues.apache.org/jira/browse/TEZ-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926789#comment-16926789
]
TezQA commented on TEZ-4080:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 48m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-0.9.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
27s{color} | {color:red} root in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} branch-0.9.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} tez-api in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in branch-0.9.1 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-api in branch-0.9.1 failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} tez-dag in branch-0.9.1 failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color}
| {color:red} tez-api in the patch failed with JDK v1.8.0_222. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} tez-dag in the patch failed with JDK v1.8.0_222. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} tez-api in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
13s{color} | {color:red} tez-dag in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} tez-api in the patch failed with JDK v9-internal. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s{color}
| {color:red} tez-dag in the patch failed with JDK v9-internal. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange} tez-api: The patch generated 1 new + 340
unchanged - 1 fixed = 341 total (was 341) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} tez-dag: The patch generated 1 new + 741
unchanged - 0 fixed = 742 total (was 741) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} tez-dag in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} tez-api in the patch failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed with JDK v1.8.0_222.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-api in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} tez-dag in the patch failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 14s{color}
| {color:red} tez-api in the patch failed with JDK v9-internal. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s{color}
| {color:red} tez-dag in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/tez:date2019-09-10 |
| JIRA Issue | TEZ-4080 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979980/TEZ-4080.branch-0.9.1.02.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux a724ff454800 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-0.9.1 / 23b58b2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 9-internal |
| Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_222
/usr/lib/jvm/java-9-openjdk-amd64:9-internal |
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-compile-tez-api-jdk1.8.0_222.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-compile-tez-dag-jdk1.8.0_222.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-compile-tez-api-jdk9-internal.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-compile-tez-dag-jdk9-internal.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-findbugs-tez-api.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-findbugs-tez-dag.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-javadoc-tez-api-jdk1.8.0_222.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-javadoc-tez-dag-jdk1.8.0_222.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-javadoc-tez-api-jdk9-internal.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/branch-javadoc-tez-dag-jdk9-internal.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-mvninstall-tez-api.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-mvninstall-tez-dag.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-api-jdk1.8.0_222.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-dag-jdk1.8.0_222.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-api-jdk1.8.0_222.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-dag-jdk1.8.0_222.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-api-jdk9-internal.txt
|
| compile |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-dag-jdk9-internal.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-api-jdk9-internal.txt
|
| javac |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-compile-tez-dag-jdk9-internal.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/diff-checkstyle-tez-api.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/diff-checkstyle-tez-dag.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-findbugs-tez-api.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-findbugs-tez-dag.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-javadoc-tez-api-jdk1.8.0_222.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-javadoc-tez-dag-jdk1.8.0_222.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-javadoc-tez-api-jdk9-internal.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-javadoc-tez-dag-jdk9-internal.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-unit-tez-api-jdk9-internal.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/artifact/out/patch-unit-tez-dag-jdk9-internal.txt
|
| JDK v9-internal Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/testReport/ |
| Max. process+thread count | 89 (vs. ulimit of 5500) |
| modules | C: tez-api tez-dag U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/161/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> TezClient should close FileSystem objects to prevent leak
> ---------------------------------------------------------
>
> Key: TEZ-4080
> URL: https://issues.apache.org/jira/browse/TEZ-4080
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Zoltan Haindrich
> Priority: Major
> Attachments: TEZ-4080.branch-0.9.1.02.patch,
> TEZ-4080.branch-0.9.1.patch
>
>
> When opening/closing a lot of tez clients; some FileSystem object references
> are retained even after the client is closed - due to the fact the
> FileSystem has a "cache" which collects all open FileSystem objects - to be
> able to close all of them from a single shutdownhook.
> Not closing these FileSystem objects causes them to pile up in the "cache"
> which has hard references to them
> In a simple hive test which was run with 150M of memory; these "lost"
> filesystem objects could result in an OOM after ~170 sessions.
> A sample creation stack trace of a FileSystem object:
> {code}
> at
> org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem.<init>(ProxyLocalFileSystem.java:49)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3353)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:502)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2663)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
