[ https://issues.apache.org/jira/browse/IMPALA-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-6073. ----------------------------------- Resolution: Cannot Reproduce > No response from expr-codegen-test/expr-test > -------------------------------------------- > > Key: IMPALA-6073 > URL: https://issues.apache.org/jira/browse/IMPALA-6073 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 2.9.0 > Environment: ubuntu 14.04.5 LTS > Reporter: Jin Chul Kim > Assignee: Philip Martin > Priority: Major > > I checked out HEAD of Impala repo. There is no any code modification on my > local repo. > After full build, ./build/debug/exprs/expr-test and > ./build/debug/exprs/expr-codegen-test seem to be hung. The other gtests in be > are working appropriately. Some threads are waiting and the other threads are > on sleep. > I guess this issue is similar to the deadlock: > https://issues.apache.org/jira/browse/HDFS-11851 > Please let me know if you have any workaround. > Here are stack traces on expr-codegen-test using GDB: > {code:java} > jinchulkim@ubuntu:~/workspace/Impala/be$ gdb > ./build/debug/exprs/expr-codegen-test > ... > (gdb) info thread > Id Target Id Frame > 16 Thread 0x7fffe20ae700 (LWP 23374) "expr-codegen-te" > pthread_cond_timedwait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > 15 Thread 0x7fffe21af700 (LWP 23373) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 14 Thread 0x7fffe22b0700 (LWP 23372) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 13 Thread 0x7fffe23b1700 (LWP 23371) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 12 Thread 0x7fffe24b2700 (LWP 23370) "expr-codegen-te" sem_wait () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 > 11 Thread 0x7fffe2c95700 (LWP 23369) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 10 Thread 0x7fffe2d96700 (LWP 23368) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 9 Thread 0x7fffe2e97700 (LWP 23367) "expr-codegen-te" > pthread_cond_timedwait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > 8 Thread 0x7fffe915a700 (LWP 23366) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 7 Thread 0x7fffe925b700 (LWP 23365) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 6 Thread 0x7fffe935c700 (LWP 23364) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 5 Thread 0x7fffe945d700 (LWP 23363) "expr-codegen-te" > pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > 4 Thread 0x7fffee536700 (LWP 23362) "expr-codegen-te" 0x00007ffff037bb9d > in nanosleep () at ../sysdeps/unix/syscall-template.S:81 > 3 Thread 0x7fffeed37700 (LWP 23361) "expr-codegen-te" 0x00007ffff037bb9d > in nanosleep () at ../sysdeps/unix/syscall-template.S:81 > 2 Thread 0x7fffef538700 (LWP 23356) "expr-codegen-te" 0x00007ffff0067dfd > in nanosleep () at ../sysdeps/unix/syscall-template.S:81 > * 1 Thread 0x7fffef5408c0 (LWP 23150) "expr-codegen-te" __lll_lock_wait () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > (gdb) where > #0 __lll_lock_wait () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > #1 0x00007ffff0376649 in _L_lock_909 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #2 0x00007ffff0376470 in __GI___pthread_mutex_lock (mutex=0x7ffff4080600 > <jvmMutex>) at ../nptl/pthread_mutex_lock.c:79 > #3 0x00007ffff3e7b666 in mutexLock (m=<optimized out>) at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28 > #4 0x00007ffff3e73a11 in setTLSExceptionStrings (rootCause=0x0, > stackTrace=0x0) at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581 > #5 0x00007ffff3e73393 in printExceptionAndFreeV (env=0x307c1d8, > exc=0x301adc0, noPrintFlags=<optimized out>, fmt=0x7ffff3e7bf6e > "loadFileSystems", ap=0x7fffffffaab0) at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183 > #6 0x00007ffff3e735ef in printExceptionAndFree (env=<optimized out>, > exc=<optimized out>, noPrintFlags=<optimized out>, fmt=<optimized out>) at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213 > #7 0x00007ffff3e74880 in getGlobalJNIEnv () at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463 > #8 getJNIEnv () at > /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528 > #9 0x00007ffff7845a1c in impala::JniUtil::Init () at > /home/jinchulkim/workspace/Impala/be/src/util/jni-util.cc:105 > #10 0x00007ffff7e3c221 in impala::InitCommonRuntime (argc=1, > argv=0x7fffffffb1e8, init_jvm=true, test_mode=impala::TestInfo::BE_TEST) at > /home/jinchulkim/workspace/Impala/be/src/common/init.cc:236 > #11 0x00000000008333fc in main (argc=1, argv=0x7fffffffb1e8) at > /home/jinchulkim/workspace/Impala/be/src/exprs/expr-codegen-test.cc:361 > (gdb) thr 2 > [Switching to thread 2 (Thread 0x7fffef538700 (LWP 23356))] > #0 0x00007ffff0067dfd in nanosleep () at > ../sysdeps/unix/syscall-template.S:81 > 81 ../sysdeps/unix/syscall-template.S: No such file or directory. > (gdb) where > #0 0x00007ffff0067dfd in nanosleep () at > ../sysdeps/unix/syscall-template.S:81 > #1 0x00007ffff0067c94 in __sleep (seconds=0) at > ../sysdeps/unix/sysv/linux/sleep.c:137 > #2 0x00007ffff7e3b304 in LogMaintenanceThread () at > /home/jinchulkim/workspace/Impala/be/src/common/init.cc:113 > #3 0x00007ffff56e4a8e in > boost::detail::function::void_function_invoker0<void (*)(), void>::invoke > (function_ptr=...) at > /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:112 > #4 0x00007ffff7b22f22 in boost::function0<void>::operator() > (this=0x7fffef537ca0) at > /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767 > #5 0x00007ffff78c0ba3 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function<void ()>, impala::Promise<long>*) > (name=..., category=..., functor=..., thread_started=0x7fffffffaa60) at > /home/jinchulkim/workspace/Impala/be/src/util/thread.cc:352 > #6 0x00007ffff78ca32c in boost::_bi::list4<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string > const&, std::string const&, boost::function<void ()>, > impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void > (*&)(std::string const&, std::string const&, boost::function<void ()>, > impala::Promise<long>*), boost::_bi::list0&, int) (this=0x2fb47c0, > f=@0x2fb47b8: 0x7ffff78c0884 <impala::Thread::SuperviseThread(std::string > const&, std::string const&, boost::function<void ()>, > impala::Promise<long>*)>, a=...) > at > /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457 > #7 0x00007ffff78ca26f in boost::_bi::bind_t<void, void (*)(std::string > const&, std::string const&, boost::function<void ()>, > impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x2fb47b8) > at > /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20 > #8 0x00007ffff78ca232 in boost::detail::thread_data<boost::_bi::bind_t<void, > void (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::Promise<long>*> > > >::run() > (this=0x2fb4600) at > /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116 > #9 0x000000000090c53a in thread_proxy () > #10 0x00007ffff0374184 in start_thread (arg=0x7fffef538700) at > pthread_create.c:312 > #11 0x00007ffff00a137d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > (gdb) > {code} > Here are the relevant environment variables: > {code:java} > jinchulkim@ubuntu:~/workspace/Impala/be$ echo $JAVA_HOME > /usr/lib/jvm/java-7-openjdk-amd64 > jinchulkim@ubuntu:~/workspace/Impala/be$ echo $BOOST_LIBRARYDIR > /usr/lib/x86_64-linux-gnu > jinchulkim@ubuntu:~/workspace/Impala/be$ echo $LD_LIBRARY_PATH > :/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org