Hi community, I was trying to load 500GB of TPCH data into kudu table using the following query:
*insert into lineitem select * from PARQUETIMPALA500.LINEITEM* While executing the query for around 17 hrs it got cancelled as the impalad process of that machine got aborted. Here are the logs of the impalad process. *impalad.ERROR* Log file created at: 2018/05/06 13:40:34 Running on machine: slave2 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this file. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176 0x7fd9e9fc3929 tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176 0x7fd9e9fc3929 F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0) *** Check failure stack trace: *** @ 0x3fc0c0d google::LogMessage::Fail() @ 0x3fc24b2 google::LogMessage::SendToLog() @ 0x3fc05e7 google::LogMessage::Flush() @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal() @ 0x1bbcb31 impala::PrintErrorMap() @ 0x1bbcd07 impala::PrintErrorMapToString() @ 0x2decbd7 impala::Coordinator::GetErrorLog() @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery() @ 0x1b29264 impala::ImpalaServer::CloseOperation() @ 0x2c5ce86 apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation() @ 0x2c56b8c apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @ 0x2c2fcb1 impala::ImpalaHiveServer2ServiceProcessor::dispatchCall() @ 0x16fdb20 apache::thrift::TDispatchProcessor::process() @ 0x18ea6b3 apache::thrift::server::TAcceptQueueServer::Task::run() @ 0x18e2181 impala::ThriftThread::RunRunnable() @ 0x18e3885 boost::_mfi::mf2<>::operator()() @ 0x18e371b boost::_bi::list3<>::operator()<>() @ 0x18e3467 boost::_bi::bind_t<>::operator()() @ 0x18e337a boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x192761c boost::function0<>::operator()() @ 0x1c3ebf7 impala::Thread::SuperviseThread() @ 0x1c470cd boost::_bi::list5<>::operator()<>() @ 0x1c46ff1 boost::_bi::bind_t<>::operator()() @ 0x1c46fb4 boost::detail::thread_data<>::run() @ 0x2eedb4a thread_proxy @ 0x7fda1dbb16ba start_thread @ 0x7fda1d8e741d clone Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp *impalad.FATAL* Log file created at: 2018/05/07 09:46:12 Running on machine: slave2 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0) *Impalad.INFO* edentials={real_user=root}} blocked reactor thread for 34288.6us I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5), user_credentials={real_user=root}} blocked reactor thread for 35859.8us I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5), user_credentials={real_user=root}} blocked reactor thread for 40664.9us I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5), user_credentials={real_user=root}} blocked reactor thread for 49514.6us I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control resources for query_id=3e4a4c646800e1d9:c859bb7f00000000 F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0) Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp *Note*: We are executing the queries on 8 node cluster with the following configuration Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each, Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz -- Regards, Geetika Gupta
