[
https://issues.apache.org/jira/browse/TEZ-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023817#comment-17023817
]
TezQA commented on TEZ-4070:
----------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
41s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/258/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4070 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991849/TEZ-4070.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 1c1b646a6484 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / a3b2dbb |
| Default Java | 1.8.0_232 |
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/258/testReport/ |
| Max. process+thread count | 213 (vs. ulimit of 5500) |
| modules | C: tez-api U: tez-api |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/258/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> SSLFactory not closed in DAGClientTimelineImpl caused native memory issues
> --------------------------------------------------------------------------
>
> Key: TEZ-4070
> URL: https://issues.apache.org/jira/browse/TEZ-4070
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Xun REN
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4070.01.patch, TEZ-4070.02.patch
>
>
> Hi,
> Recently, we're facing native memory issues on Redhat servers. It crashed
> completely our servers.
> *Context:*
> - HDP-2.6.5
> - Redhat 7.4
> *Problem:*
> After upgrading from HDP-2.6.2 to HDP-2.6.5, after several days running, our
> HiveServer2 can eat up to more than 100GB memory. However, we have configured
> Xmx20G and MaxMetaspace to 10GB.
> After searching, we have found the similar issue here:
> https://issues.apache.org/jira/browse/YARN-5309
> This is fixed in the hadoop-common module. Our version includes already this
> issue, however, we still have the problem.
> After searching, I have found that in the class
> org.apache.tez.dag.api.client.TimelineReaderFactory of Tez, if HTTPS is used
> for YARN, it will create SSLFactory which is not destroyed after utilization.
> TimelineReaderFactory is referenced in the class DAGClientTimelineImpl.
> If ATS is used and DAG is completed, the method switchToTimelineClient in the
> class DAGClientImpl will be called. It will close the previous HTTPClient,
> but not the SSLFactory inside. And the SSLFactory will create a thread for
> each connection. Finally, we will get thousands of threads consuming a lot
> native memories.
> {code:java}
> private void switchToTimelineClient() throws IOException, TezException {
> realClient.close();
> realClient = new DAGClientTimelineImpl(appId, dagId, conf, frameworkClient,
> (int) (2 * PRINT_STATUS_INTERVAL_MILLIS));
> if (LOG.isDebugEnabled()) {
> LOG.debug("dag completed switching to DAGClientTimelineImpl");
> }
> }{code}
> I have checked on another environment which is still on HDP-2.6.2, we also
> have a lot of running threads holding by SSLFactory. That means the problem
> is zoomed in the version HDP-2.6.5
>
> *How to reproduce the problem:*
> 1. Use Tez as Hive execution engine
> 2. Launch a Beeline session for Hive
> 3. Do a select with a simple where clause on a table
> 4. Repeat steps 2-3 in order to open different connections (it is the case
> for a shared cluster with multiple clients).
> Finally, you can check in the thread dump file, that a lot of threads are
> named "Truststore reloader thread". And the native memory usage is very high
> by doing the command "top" or "ps".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
