Re: expr-test stuck in getJNIEnv

[Sailesh Mukil]

The upstream bug is posted below. I've requested a backport to the HDFS
version Impala depends on, but it hasn't happened yet. I'll make another
push for it.

https://issues.apache.org/jira/browse/HDFS-11851

On Sun, Sep 10, 2017 at 4:01 PM, Jim Apple  wrote:

> It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry!
>
> On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson  wrote:
> > I've seen this deadlock before although not in expr-test. I can't
> remember
> > exactly how I cleared it but I believe it was either:
> >
> > 1. make fe && . bin/set-classpath.sh
> > 2. bin/create-test-configuration.sh
> >
> > Sailesh knows about the upstream HDFS bug which I think has been fixed
> but
> > not incorporated into Impala's dependencies.
> >
> > On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:
> >
> >> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full
> stack
> >> trace:
> >>
> >> #0  __lll_lock_wait () at
> >> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> >> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
> >> ) at ../nptl/pthread_mutex_lock.c:80
> >> #2  0x02cf79f6 in mutexLock (m=) at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/os/posix/mutexes.c:28
> >> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
> >> stackTrace=0x0) at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:581
> >> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
> >> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
> >> "loadFileSystems", ap=0x7fff9da0)
> >> at
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/exception.c:183
> >> #5  0x02cf81dd in printExceptionAndFree (env=,
> >> exc=, noPrintFlags=, fmt= >> out>)
> >> at
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/exception.c:213
> >> #6  0x02cf0faf in getGlobalJNIEnv () at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:463
> >> #7  getJNIEnv () at
> >>
> >> /data/2/jenkins/workspace/impala-hadoop-dependency/
> hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/
> libhdfs/jni_helper.c:528
> >> #8  0x01a0116a in impala::JniUtil::Init () at
> >> be/src/util/jni-util.cc:105
> >> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
> >> argv=0x7fffa628, init_jvm=true,
> >> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
> >> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
> >> be/src/exprs/expr-test.cc:7420
> >>
> >> I've tried git fetch, bin/clean.sh, running with the minicluster on,
> >> running with the minicluster off, running with the impala cluster on,
> >> running with it off, running in release mode, debug mode, in gdb, and
> >> out of gdb.
> >>
> >> Has anyone else seen this and escaped from its clutches?
> >>
>

Re: expr-test stuck in getJNIEnv

[Jim Apple]

It was . bin/set-classpath.sh. Forgot about that one. Thanks, Henry!

On Sun, Sep 10, 2017 at 3:31 PM, Henry Robinson  wrote:
> I've seen this deadlock before although not in expr-test. I can't remember
> exactly how I cleared it but I believe it was either:
>
> 1. make fe && . bin/set-classpath.sh
> 2. bin/create-test-configuration.sh
>
> Sailesh knows about the upstream HDFS bug which I think has been fixed but
> not incorporated into Impala's dependencies.
>
> On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:
>
>> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack
>> trace:
>>
>> #0  __lll_lock_wait () at
>> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
>> ) at ../nptl/pthread_mutex_lock.c:80
>> #2  0x02cf79f6 in mutexLock (m=) at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
>> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
>> stackTrace=0x0) at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
>> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
>> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
>> "loadFileSystems", ap=0x7fff9da0)
>> at
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
>> #5  0x02cf81dd in printExceptionAndFree (env=,
>> exc=, noPrintFlags=, fmt=> out>)
>> at
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
>> #6  0x02cf0faf in getGlobalJNIEnv () at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
>> #7  getJNIEnv () at
>>
>> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
>> #8  0x01a0116a in impala::JniUtil::Init () at
>> be/src/util/jni-util.cc:105
>> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
>> argv=0x7fffa628, init_jvm=true,
>> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
>> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
>> be/src/exprs/expr-test.cc:7420
>>
>> I've tried git fetch, bin/clean.sh, running with the minicluster on,
>> running with the minicluster off, running with the impala cluster on,
>> running with it off, running in release mode, debug mode, in gdb, and
>> out of gdb.
>>
>> Has anyone else seen this and escaped from its clutches?
>>

Re: expr-test stuck in getJNIEnv

[Henry Robinson]

I've seen this deadlock before although not in expr-test. I can't remember
exactly how I cleared it but I believe it was either:

1. make fe && . bin/set-classpath.sh
2. bin/create-test-configuration.sh

Sailesh knows about the upstream HDFS bug which I think has been fixed but
not incorporated into Impala's dependencies.

On Sun, Sep 10, 2017 at 1:42 PM Jim Apple  wrote:

> When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack
> trace:
>
> #0  __lll_lock_wait () at
> ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
> ) at ../nptl/pthread_mutex_lock.c:80
> #2  0x02cf79f6 in mutexLock (m=) at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
> #3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
> stackTrace=0x0) at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
> #4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
> exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
> "loadFileSystems", ap=0x7fff9da0)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
> #5  0x02cf81dd in printExceptionAndFree (env=,
> exc=, noPrintFlags=, fmt= out>)
> at
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
> #6  0x02cf0faf in getGlobalJNIEnv () at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
> #7  getJNIEnv () at
>
> /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
> #8  0x01a0116a in impala::JniUtil::Init () at
> be/src/util/jni-util.cc:105
> #9  0x014fd881 in impala::InitCommonRuntime (argc=1,
> argv=0x7fffa628, init_jvm=true,
> test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
> #10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
> be/src/exprs/expr-test.cc:7420
>
> I've tried git fetch, bin/clean.sh, running with the minicluster on,
> running with the minicluster off, running with the impala cluster on,
> running with it off, running in release mode, debug mode, in gdb, and
> out of gdb.
>
> Has anyone else seen this and escaped from its clutches?
>

expr-test stuck in getJNIEnv

[Jim Apple]

When I run expr-test, it gets stuck in getJNIEnv(). Here's the full stack trace:

#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x74885dbd in __GI___pthread_mutex_lock (mutex=0x45fe5a0
) at ../nptl/pthread_mutex_lock.c:80
#2  0x02cf79f6 in mutexLock (m=) at
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
#3  0x02cf01b7 in setTLSExceptionStrings (rootCause=0x0,
stackTrace=0x0) at
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
#4  0x02cf7f77 in printExceptionAndFreeV (env=0x4f221e8,
exc=0x4eb6a00, noPrintFlags=, fmt=0x33d7994
"loadFileSystems", ap=0x7fff9da0)
at 
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
#5  0x02cf81dd in printExceptionAndFree (env=,
exc=, noPrintFlags=, fmt=)
at 
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
#6  0x02cf0faf in getGlobalJNIEnv () at
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
#7  getJNIEnv () at
/data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
#8  0x01a0116a in impala::JniUtil::Init () at
be/src/util/jni-util.cc:105
#9  0x014fd881 in impala::InitCommonRuntime (argc=1,
argv=0x7fffa628, init_jvm=true,
test_mode=impala::TestInfo::BE_TEST) at be/src/common/init.cc:236
#10 0x0143da3f in main (argc=1, argv=0x7fffa628) at
be/src/exprs/expr-test.cc:7420

I've tried git fetch, bin/clean.sh, running with the minicluster on,
running with the minicluster off, running with the impala cluster on,
running with it off, running in release mode, debug mode, in gdb, and
out of gdb.

Has anyone else seen this and escaped from its clutches?