[
https://issues.apache.org/jira/browse/KUDU-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613066#comment-17613066
]
ASF subversion and git services commented on KUDU-3404:
-------------------------------------------------------
Commit eacfee6fe2a9d323985225af0174ea6bb161f090 in kudu's branch
refs/heads/master from Marton Greber
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=eacfee6fe ]
KUDU-3404 disable TLS in glog
Glog 0.4.0 introduced support for using thread local storage for its
buffer. This feature is controlled by the WITH_TLS CMake variable, and
it defaults to ON. [1] When Kudu upgraded to glog 0.6.0 as part of the
Apple Silicon (M Series) fixes [2], it increased the thread local
storage usage by >30000 bytes.
This is a problem for Impala, because Impala starts a JVM. There are
certain JVM threads (like the "reaper thread") that have very small
stacks (e.g. 32KB) and with glibc the TLS space is allocated at the
expense of stack space. [3] 30k of TLS usage leaves very little for the
reaper thread. There are a series of bugs where the Java reaper thread
hits a StackOverflowException because of high TLS usage. [4] This can
cause various symptoms including hangs.
To resolve Impala's problem, it would be useful to build
libkudu_client.so with glog's WITH_TLS=OFF.
[1] https://github.com/google/glog/commit/
2df0ca34aa3000dadf76633ca700abf0bf50756d
[2] https://github.com/apache/kudu/commit/
543e128d473f8f7836e605bba8cd6512fa918550
[3] https://sourceware.org/bugzilla/show_bug.cgi?id=11787
[4] JDK bugs: JDK-8217475, JDK-8225035
Change-Id: I6b2151f7355ebb6a9ea120e5aa675c81eb3842ab
Reviewed-on: http://gerrit.cloudera.org:8080/19089
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor <[email protected]>
> glog 0.6.0 increases the TLS usage of libkudu_client.so substantially
> ---------------------------------------------------------------------
>
> Key: KUDU-3404
> URL: https://issues.apache.org/jira/browse/KUDU-3404
> Project: Kudu
> Issue Type: Bug
> Components: client
> Affects Versions: 1.17.0
> Reporter: Joe McDonnell
> Assignee: Marton Greber
> Priority: Critical
> Attachments: 0001-Add-WITH_TLS-OFF-to-glog-build-definition.patch
>
>
> Glog 0.4 introduced supported for using thread local storage for its buffer.
> This feature is controlled by the WITH_TLS CMake variable, and it defaults to
> ON. See
> [https://github.com/google/glog/commit/2df0ca34aa3000dadf76633ca700abf0bf50756d]
> . When Kudu upgraded to glog 0.6.0 as part of the M1 fixes in "[KUDU-3374
> Add support for M1 and macOS
> Monterey|https://github.com/apache/kudu/commit/543e128d473f8f7836e605bba8cd6512fa918550]";,
> it increased the thread local storage usage by >30000 bytes.
> {noformat}
> # Older libkudu_client.so has 0x100 = 256 bytes of TLS:
> $ readelf -l libkudu_client.so | grep "TLS" -A1
> TLS 0x00000000007d14c0 0x00000000007d24c0 0x00000000007d24c0
> 0x0000000000000080 0x0000000000000100 R 0x40
> # Newer libkudu_client.so has 0x77b9 = 30649 bytes of TLS:
> $ readelf -l libkudu_client.so.0 | grep TLS -A1
> TLS 0x0000000000751280 0x0000000000752280 0x0000000000752280
> 0x0000000000000080 0x00000000000077b9 R 40{noformat}
> This is a problem for Impala, because Impala starts a JVM. There are certain
> JVM threads (like the "reaper thread") that have very small stacks (e.g.
> 32KB) and with glibc the TLS space is allocated at the expense of stack
> space. 30k of TLS usage leaves very little for the reaper thread. There are a
> series of bugs where the Java reaper thread hits a StackOverflowException
> because of high TLS usage. This can cause various symptoms including hangs.
> GLIBC message thread: [https://sourceware.org/bugzilla/show_bug.cgi?id=11787]
> JDK bugs:
> [JDK-8217475|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8217475],
> [JDK-8225035|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8225035]
> To resolve Impala's problem, it would be useful to build libkudu_client.so
> with glog's WITH_TLS=OFF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
