The first crash is a symptom of some classes being missing from the classpath. If you look at the code where it crashed, it's loading a bunch of HDFS classes - https://github.com/apache/hadoop/blob/18c57cf0464f4d1fa95899d75b2f59cae33c7c33/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c#L69
You need a lot of things on the classpath, you really need something automated to set it up correctly. In the dev environment we generate a file that contains the classpath https://github.com/apache/impala/blob/master/bin/set-classpath.sh#L45 On Thu, Jun 4, 2020 at 3:20 AM Hashan Gayasri <hashan.gaya...@gmail.com> wrote: > Hi all, > > I've been trying to get Impala (v3.3.0) that I compiled running. Upon > startup, Catalogd (Impalad binary) seems to crash in native code. > In a dynamically-linked, debug build, the stack trace was as follows. > > [1] > ... > #4 0x00007ffff1fd0a05 in JVM_handle_linux_signal () from > /home/hashan/jdk1.8.0_191/jre/lib/amd64/server/libjvm.so > #5 0x00007ffff1fc3cd8 in signalHandler(int, siginfo*, void*) () from > /home/hashan/jdk1.8.0_191/jre/lib/amd64/server/libjvm.so > #6 <signal handler called> > #7 initCachedClass (cachedJclass=<optimized out>, > className=<optimized out>, env=0x0) > at > /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:54 > #8 initCachedClasses (env=0x0) at > > /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jclasses.c:117 > #9 0x00007ffff26a3f62 in getJNIEnv () at > > /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:555 > #10 0x00007ffff26aa3b1 in hdfsBuilderConnect (bld=0x389b2c0) > at > /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:697 > #11 0x00007ffff77540e3 in impala::JniUtil::InitLibhdfs () at > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/util/jni-util.cc:215 > #12 0x00007ffff7753660 in impala::JniUtil::Init () at > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/util/jni-util.cc:132 > #13 0x00007ffff7e84146 in impala::InitCommonRuntime (argc=1, > argv=0x7fffffff65c8, init_jvm=true, > test_mode=impala::TestInfo::NON_TEST) at > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/common/init.cc:364 > #14 0x00007ffff3c31bdc in CatalogdMain (argc=1, argv=0x7fffffff65c8) > at > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/catalog/catalogd-main.cc:62 > #15 0x00000000008c60ef in main (argc=1, argv=0x7fffffff65c8) at > > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/service/daemon-main.cc:41 > ... > > The loaded hdfs native library was: > > impala-3.3.0/toolchain/cdh_components-1173663/hadoop-3.0.0-cdh6.x-SNAPSHOT/lib/native/libhdfs.so.0.0.0 > > > After getting the same results when using the native hdfs library > (libhdfs.so.0.0.0) shipped with the rpm package > "hadoop-libhdfs-3.0.0+cdh6.3.0-1279813.el7.x86_64", > I tried using the libhdfs.so.0.0.0 library compiled from hadoop v3.1.3 > github sources. This seemed to pass the previous stage. > > [2] > This time the /tmp/catalogd.ERROR file contained: > > E0604 09:50:31.550122 69397 catalog.cc:91] NoClassDefFoundError: > org/apache/hadoop/hive/metastore/api/Database > CAUSED BY: ClassNotFoundException: > org.apache.hadoop.hive.metastore.api.Database > . Impalad exiting. > loadFileSystems error: > ClassNotFoundException: > org.apache.hadoop.fs.FileSystemjava.lang.NoClassDefFoundError: > org/apache/hadoop/fs/FileSystem > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.fs.FileSystem > ... > hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, > kerbTicketCachePath=(NULL), userName=(NULL)) error: > ClassNotFoundException: > org.apache.hadoop.conf.Configurationjava.lang.NoClassDefFoundError: > org/apache/hadoop/conf/Configuration > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.conf.Configuration > ... > > [3] > After adding the following jar files to the CLASSPATH, > * hive-3.1.2/lib/hive-metastore-3.1.2.jar > * hive-3.1.2/lib/hive-standalone-metastore-3.1.2.jar > * hadoop-3.1.3/share/hadoop/client/hadoop-client-runtime-3.1.3.jar > > /tmp/catalogd.ERROR file contained: > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.fs.FileSystem > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.conf.Configuration > > In both latter cases using the newly compiled libhdfs.so.0.0.0 > library, although the /tmp/catalogd.ERROR is different, the crash > would occur at: > catalog-server.cc:252 > (gdb) bt > #0 impala::CatalogServer::Start (this=0x7fffffff6030) at > > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/catalog/catalog-server.cc:252 > #1 0x00007ffff3c3235e in CatalogdMain (argc=1, argv=0x7fffffff65c8) > at > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/catalog/catalogd-main.cc:87 > #2 0x00000000008c60ef in main (argc=1, argv=0x7fffffff65c8) at > > /home/hashan/BUILD/impala-3.3.0/impala-3.3.0/be/src/service/daemon-main.cc:41 > (gdb) l > 252 catalog_.reset(new Catalog()); > (gdb) p catalog_ > $2 = {px = 0x0} > > > 1) Does anyone have some idea why the first issue arises when using > the native hdfs library built as a part of the toolchain? > > 2) Does anyone know if the issue in 2nd and 3rd runs (using locally > built libhdfs) is actually related to missing JAR files and if so, > which JAR files are missing from the classpath? > > I'm sorry for the length of this mail. Any help in resolving these > issues would be greatly appreciated. > > Thanks in advance. > > Regards, > Hashan Gayasri >
