[jira] [Commented] (IMPALA-12564) TSAN test run fails to start Hive Server on Ubuntu 22.04
[
https://issues.apache.org/jira/browse/IMPALA-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835885#comment-17835885
]
ASF subversion and git services commented on IMPALA-12564:
--
Commit 408c119f7da022e2f6de2df2b5af4546eefdb5bc in impala's branch
refs/heads/master from Laszlo Gaal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=408c119f7 ]
IMPALA-12564: Prevent Hive loading libfesupport.so in the minicluster during
TSAN runs
During TSAN runs all Impala binaries (including libfesupport.so) are
built with TSAN options, which include a reference to the external
symbol __tsan_init. This causes a problem for libfesupport.so when it is
loaded into Hive during minicluster startup, because the Java VM running
Hive's code cannot supply this symbol (the stock JVM is obviously not
built with TSAN).
Unfortunately this symbol resolution failure causes Hive's JVM simply to
abort on Red Hat 8 (or later) and on Ubuntu 20.04 (or later).
On earlier versions of the same platforms the JVM turned the same
failure into an UnsatisfiedLinkError exception, which is actually
handled by Hive.
This patch prevents libfesupport.so from being loaded into Hive for TSAN
runs so that the minicluster can actually be started. This is achieved
by not adding the directory containing libfesupport.so to
JAVA_LIBRARY_PATH, preventing the JVM from finding it.
Change-Id: Ie030d9876c297d6e9dae80eba37e525ee2bccb20
Reviewed-on: http://gerrit.cloudera.org:8080/21191
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> TSAN test run fails to start Hive Server on Ubuntu 22.04
>
>
> Key: IMPALA-12564
> URL: https://issues.apache.org/jira/browse/IMPALA-12564
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Laszlo Gaal
>Priority: Major
>
> Hive Server startup hangs during ARM TSAN test runs. Logs were not
> particularly illuminating.
> hive-server2.out shows
> {code}
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java: symbol lookup error:
> /home/michael/Impala/be/build/debug/service/libfesupport.so: undefined
> symbol: __tsan_init
> {code}
> Can work around it locally by commenting out
> https://github.com/apache/impala/blob/4.3.0/testdata/bin/run-hive-server.sh#L140-L146.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12564) TSAN test run fails to start Hive Server on Ubuntu 22.04
[
https://issues.apache.org/jira/browse/IMPALA-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829995#comment-17829995
]
Laszlo Gaal commented on IMPALA-12564:
--
https://gerrit.cloudera.org/c/21191
> TSAN test run fails to start Hive Server on Ubuntu 22.04
>
>
> Key: IMPALA-12564
> URL: https://issues.apache.org/jira/browse/IMPALA-12564
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Laszlo Gaal
>Priority: Major
>
> Hive Server startup hangs during ARM TSAN test runs. Logs were not
> particularly illuminating.
> hive-server2.out shows
> {code}
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java: symbol lookup error:
> /home/michael/Impala/be/build/debug/service/libfesupport.so: undefined
> symbol: __tsan_init
> {code}
> Can work around it locally by commenting out
> https://github.com/apache/impala/blob/4.3.0/testdata/bin/run-hive-server.sh#L140-L146.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12564) TSAN test run fails to start Hive Server on Ubuntu 22.04
[
https://issues.apache.org/jira/browse/IMPALA-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829302#comment-17829302
]
Laszlo Gaal commented on IMPALA-12564:
--
Since TSAN runs conclude successfully even in the absence of a loaded
libfesupport.so, the suggested workaround is to suppress the attempt to load it
into Hive during a TSAN run.
A test run of TSAN-specific tests with the removal hardcoded was successful, so
I'll submit a patch for this workaround.
> TSAN test run fails to start Hive Server on Ubuntu 22.04
>
>
> Key: IMPALA-12564
> URL: https://issues.apache.org/jira/browse/IMPALA-12564
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
>
> Hive Server startup hangs during ARM TSAN test runs. Logs were not
> particularly illuminating.
> hive-server2.out shows
> {code}
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java: symbol lookup error:
> /home/michael/Impala/be/build/debug/service/libfesupport.so: undefined
> symbol: __tsan_init
> {code}
> Can work around it locally by commenting out
> https://github.com/apache/impala/blob/4.3.0/testdata/bin/run-hive-server.sh#L140-L146.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12564) TSAN test run fails to start Hive Server on Ubuntu 22.04
[
https://issues.apache.org/jira/browse/IMPALA-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829300#comment-17829300
]
Laszlo Gaal commented on IMPALA-12564:
--
This is actually interesting.
I started chasing the symbol load failure by inserting
{{LD_DEBUG=libs,files,symbols}} into {{testdata/bin/run-hive-server.sh}}, the
script that launches Hive for the minicluster. The dynamic linker's debug
output confirmed that the above symbol, {{__tsan_init}} fails to get resolved
when libfesupport.so is loaded by the JVM as Hive tries to load it in
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/FeSupport.java#L549-L555,
via
https://github.infra.cloudera.com/CDH/hive/blob/cdpd-master/ql/src/java/org/apache/hadoop/hive/ql/engine/EngineLoader.java
What is interesting is that the symbol resolution failure, and the subsequent
failure to load libfesupport.so into Hive's JVM is actually present on all the
platforms, even on the ones that seem to survive the load attempt (Centos 7 and
Ubuntu 18.04). The difference between the platforms is in the JVM's subsequent
reaction:
* on older platforms (Centos 7, Ubuntu 18.04) the JVM swallows the load failure
and turns it into a Java exception that can be caught in FeSupport.java. This
allows execution to continue normally.
* on newer platforms the JVM just emits the message about the symbol resolution
failure and terminates, causing the minicluster's helper script that waits for
the Hive startup to time out eventually.
This doesn't even seem to be a difference between the JVM/JDK versions native
to the platform: I tried importing an older platform's JDK8 version to a newer
OS (downloading it as a tarball and setting it up in its own directory tree,
then pointing the PATH at this tree as well as setting JAVA_HOME for Impala),
but the failure remained.
> TSAN test run fails to start Hive Server on Ubuntu 22.04
>
>
> Key: IMPALA-12564
> URL: https://issues.apache.org/jira/browse/IMPALA-12564
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
>
> Hive Server startup hangs during ARM TSAN test runs. Logs were not
> particularly illuminating.
> hive-server2.out shows
> {code}
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java: symbol lookup error:
> /home/michael/Impala/be/build/debug/service/libfesupport.so: undefined
> symbol: __tsan_init
> {code}
> Can work around it locally by commenting out
> https://github.com/apache/impala/blob/4.3.0/testdata/bin/run-hive-server.sh#L140-L146.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12564) TSAN test run fails to start Hive Server on Ubuntu 22.04
[
https://issues.apache.org/jira/browse/IMPALA-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824358#comment-17824358
]
Laszlo Gaal commented on IMPALA-12564:
--
I don't think this is ARM-related: I saw exactly the same failure on RHEL 8.6
in an Intel-based run.
To me it looks more like a difference between older and newer platforms: the
fact that existing code works on Ubuntu 20.04 and on Centos 7.9 works, while
the same code fails on Red Hat 8.6 and Ubuntu 22.04 suggest a platform
difference caused by major version upgrades.
> TSAN test run fails to start Hive Server on Ubuntu 22.04
>
>
> Key: IMPALA-12564
> URL: https://issues.apache.org/jira/browse/IMPALA-12564
> Project: IMPALA
> Issue Type: Bug
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
>
> Hive Server startup hangs during ARM TSAN test runs. Logs were not
> particularly illuminating.
> hive-server2.out shows
> {code}
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java: symbol lookup error:
> /home/michael/Impala/be/build/debug/service/libfesupport.so: undefined
> symbol: __tsan_init
> {code}
> Can work around it locally by commenting out
> https://github.com/apache/impala/blob/4.3.0/testdata/bin/run-hive-server.sh#L140-L146.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
