[
https://issues.apache.org/jira/browse/HDFS-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835112#comment-16835112
]
Todd Lipcon commented on HDFS-12628:
------------------------------------
[~joemcdonnell] I tried running your test program against Hadoop 3.2 and Java
1.8.0_141 as follows:
{code}
g++ jni-util-test2.cc -Wall -I$HADOOP_HOME/include -I$JAVA_HOME/include
-I$JAVA_HOME/include/linux $JAVA_HOME/jre/lib/amd64/server/libjvm.so -pthread
$HADOOP_HOME/lib/native/libhdfs.so
CLASSPATH=$(hadoop classpath --glob) ./a.out
{code}
It doesn't seem to crash any more. Do you think we've fixed this somewhere
along the line? Or was the crash in your test program sporadic?
> libhdfs crashes on thread exit for JNI+libhdfs applications
> -----------------------------------------------------------
>
> Key: HDFS-12628
> URL: https://issues.apache.org/jira/browse/HDFS-12628
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: native
> Affects Versions: 3.0.0-alpha3
> Reporter: Joe McDonnell
> Priority: Critical
> Attachments: jni-util-test2.cc
>
>
> Impala uses libhdfs to access HDFS while also using JNI to run other Java
> code. Impala currently relies on HDFS's getJNIEnv to get a JNIEnv to interact
> with the process JVM (which is created by HDFS code). It uses this JNIEnv
> even for code that is not related to HDFS.
> In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to
> HDFS-7879. In HDFS-8474, the proposed solution was for Impala to write its
> own equivalent (tracked by IMPALA-2029). After implementing an equivalent of
> getJNIEnv (heavily based on HDFS code, but with distinct names), we are
> seeing crashes in hdfsThreadDestructor() in threads that use both HDFS and
> JNI codepaths. The crash shows up under concurrency and does not reproduce in
> serial execution.
> I have distilled it down to a simple testcase that reproduces the issue. It
> creates a JVM in the main thread (which Impala does at startup), then spawns
> multiple threads that do basic HDFS and JNI work. I have removed all but the
> essential steps.
> This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was
> merged). Note that exposing getJNIEnv should unblock Impala development if a
> fix is not forthcoming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
