Philip Zeyliger has posted comments on this change.
Change subject: IMPALA-5926: Avoid printing expensive stack when closing a
session
......................................................................
Patch Set 3:
I took the liberty of trawling through some large impalad log files to see
where else this sort of thing could be happening. What you're seeing below is
the stack traces produced by status.cc from about 1GB of log files of impala on
a cluster at work. I've removed the timestamps, removed the hostnames (outside
of the script below), and the query ids.
The punch line is that "Not implemented for this format" sure seems to come up
a lot and may benefit from similar treatment.
I'm not proud of it, but here's the "query":
find . -type f -name '*IMPALAD*.gz' | xargs gzcat | awk '/^I/ { if(x) { print
x; } x = "" } /status.cc/ { x=" "; } { if(x) { x=x $0 } }' | sed -e
's/0x[0-9a-fx]* //g' | sed -e 's/[0-9a-f]\{16\}:[0-9a-f]*/QUERYID/g' | tr -s
'\t' ' ' | tr '[0-9]' 'N' | sort | uniq -c | sort -n | tee output.txt
300090 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
70111 INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
52119 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::ConnectionEnd() @
impala::ThriftServer::ThriftServerEventProcessor::deleteContext() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
50797 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
43115 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::TopNNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
21064 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::CloseSession() @
apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @
apache::thrift::TDispatchProcessor::process() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
16750 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::ConnectionEnd() @
impala::ThriftServer::ThriftServerEventProcessor::deleteContext() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
10784 INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::TopNNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
9112 INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::ConnectionEnd() @
impala::ThriftServer::ThriftServerEventProcessor::deleteContext() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
7855 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Not implemented for this
format. @ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::TopNNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
5700 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::CloseSession() @
apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @
apache::thrift::TDispatchProcessor::process() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
4761 INNNN NN:NN:NN.NNNNNN NNNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::CloseSession() @
apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @
apache::thrift::TDispatchProcessor::process() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
4080 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN RPC recv timed out: Client
hostname:NNNNN timed-out during recv call. @ impala::Status::Status() @
impala::DataStreamSender::Channel::DoTransmitDataRpc() @
impala::DataStreamSender::Channel::TransmitDataHelper() @
impala::DataStreamSender::Channel::TransmitData() @
impala::ThreadPool<>::WorkerThread() @ impala::Thread::SuperviseThread() @
boost::detail::thread_data<>::run() @ (unknown) @ start_thread @ clone
3976 INNNN NN:NN:NN.NNNNNN NNN status.cc:NNNN Not implemented for this format.
@ impala::Status::Status() @ impala::HdfsScanNodeBase::Codegen() @
impala::ExecNode::Codegen() @ impala::PartitionedAggregationNode::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
3421 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Timestamp not yet supported
for codegen. @ impala::Status::Status() @
impala::HdfsScanner::CodegenWriteCompleteTuple() @
impala::HdfsTextScanner::Codegen() @ impala::HdfsScanNodeBase::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
3227 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN ReportExecStatus(): Received
report for unknown query ID (probably closed or cancelled): QUERYID @
impala::Status::Status() @ impala::ImpalaServer::ReportExecStatus() @
impala::ImpalaInternalServiceProcessor::process_ReportExecStatus() @
impala::ImpalaInternalServiceProcessor::dispatchCall() @
apache::thrift::TDispatchProcessor::process() @
apache::thrift::server::TAcceptQueueServer::Task::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
2335 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN Invalid session id: QUERYID @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::CloseSession() @
apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall() @
apache::thrift::TDispatchProcessor::process() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
2016 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN Timestamp not yet supported
for codegen. @ impala::Status::Status() @
impala::HdfsScanner::CodegenWriteCompleteTuple() @
impala::HdfsTextScanner::Codegen() @ impala::HdfsScanNodeBase::Codegen() @
impala::FragmentInstanceState::Prepare() @
impala::FragmentInstanceState::Exec() @ impala::QueryState::ExecFInstance() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
1404 INNNN NN:NN:NN.NNNNNN NNN status.cc:NNNN Session closed @
impala::Status::Status() @ impala::ImpalaServer::CloseSessionInternal() @
impala::ImpalaServer::ConnectionEnd() @
impala::ThriftServer::ThriftServerEventProcessor::deleteContext() @
apache::thrift::server::TThreadPoolServer::Task::run() @
apache::thrift::concurrency::ThreadManager::Worker::run() @
impala::ThriftThread::RunRunnable() @
boost::detail::function::void_function_obj_invokerN<>::invoke() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
1384 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNNN couldn't deserialize thrift
msg:No more data to read. @ impala::Status::Status() @
impala::DeserializeThriftMsg<>() @
impala::BaseScalarColumnReader::ReadPageHeader() @
impala::BaseScalarColumnReader::ReadDataPage() @
impala::BaseScalarColumnReader::NextPage() @
impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch() @
impala::HdfsParquetScanner::AssembleRows() @
impala::HdfsParquetScanner::GetNextInternal() @
impala::HdfsParquetScanner::ProcessSplit() @
impala::HdfsScanNode::ProcessSplit() @ impala::HdfsScanNode::ScannerThread() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
1358 INNNN NN:NN:NN.NNNNNN NNNNN status.cc:NNN RPC recv timed out: Client
hostname:NNNNN timed-out during recv call. @ impala::Status::Status() @
impala::ClientConnection<>::DoRpc<>() @
impala::DataStreamSender::Channel::DoTransmitDataRpc() @
impala::DataStreamSender::Channel::TransmitDataHelper() @
impala::DataStreamSender::Channel::TransmitData() @
impala::ThreadPool<>::WorkerThread() @ impala::Thread::SuperviseThread() @
boost::detail::thread_data<>::run() @ (unknown) @ start_thread @ clone
1031 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNN RPC recv timed out: Client
hostname:NNNNN timed-out during recv call. @ impala::Status::Status() @
impala::DataStreamSender::Channel::DoTransmitDataRpc() @
impala::DataStreamSender::Channel::TransmitDataHelper() @
impala::DataStreamSender::Channel::TransmitData() @
impala::ThreadPool<>::WorkerThread() @ impala::Thread::SuperviseThread() @
boost::detail::thread_data<>::run() @ (unknown) @ start_thread @ clone
1008 INNNN NN:NN:NN.NNNNNN NNNNNN status.cc:NNNN couldn't deserialize thrift
msg:No more data to read. @ impala::Status::Status() @
impala::DeserializeThriftMsg<>() @
impala::BaseScalarColumnReader::ReadPageHeader() @
impala::BaseScalarColumnReader::ReadDataPage() @
impala::BaseScalarColumnReader::NextPage() @
impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch() @
impala::HdfsParquetScanner::AssembleRows() @
impala::HdfsParquetScanner::GetNextInternal() @
impala::HdfsParquetScanner::ProcessSplit() @
impala::HdfsScanNode::ProcessSplit() @ impala::HdfsScanNode::ScannerThread() @
impala::Thread::SuperviseThread() @ boost::detail::thread_data<>::run() @
(unknown) @ start_thread @ clone
--
To view, visit http://gerrit.cloudera.org:8080/8060
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9d0ca485dd17bca758d916040745288c1a20c69f
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Mostafa Mokhtar <[email protected]>
Gerrit-Reviewer: Alex Behm <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-HasComments: No
