[
https://issues.apache.org/jira/browse/IMPALA-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5831.
-----------------------------------
Resolution: Won't Fix
This should not happen with the local catalog as far as we know.
> Create table operation stuck on statestore update
> -------------------------------------------------
>
> Key: IMPALA-5831
> URL: https://issues.apache.org/jira/browse/IMPALA-5831
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.10.0
> Reporter: Mostafa Mokhtar
> Priority: Minor
> Labels: statestore, stuck, topic
>
> Noticed a create table statement being stuck for a couple of days.
> {code}
> User Default Db Statement Query Type Start Time
> Duration Scan Progress State Last Event # rows fetched
> Resource Pool Details Action
> [email protected] tpcds_1000_parquet CREATE TABLE store_sales_insert LIKE
> store_sales DDL 2017-08-19 16:39:36.417982000 90h26m N/A
> CREATED Planning finished 0 Details Cancel
> {code}
> Coordinator log
> {code}
> I0819 16:39:36.418120 178699 Frontend.java:888] Compiling query: CREATE TABLE
> store_sales_insert LIKE store_sales
> I0819 16:39:36.421865 178699 Frontend.java:927] Compiled query.
> I0819 16:39:36.529786 178699 impala-server.cc:1475] Waiting for catalog
> version: 282 current version: 281
> I0819 16:39:40.296090 178699 impala-server.cc:1489] Waiting for min
> subscriber topic version: 314 current version: 0
> I0820 00:04:10.183965 177679 authentication.cc:498] Registering
> impala/[email protected], keytab file
> /var/run/cloudera-scm-agent/process/3262-impala-IMPALAD/impala.keytab
> I0821 00:02:15.233804 177679 authentication.cc:498] Registering
> impala/[email protected], keytab file
> /var/run/cloudera-scm-agent/process/3262-impala-IMPALAD/impala.keytab
> I0822 00:00:22.234457 177679 authentication.cc:498] Registering
> impala/[email protected], keytab file
> /var/run/cloudera-scm-agent/process/3262-impala-IMPALAD/impala.keytab
> I0822 23:58:11.268100 177679 authentication.cc:498] Registering
> impala/[email protected], keytab file
> /var/run/cloudera-scm-agent/process/3262-impala-IMPALAD/impala.keytab
> {code}
> Catalog log
> {code}
> I0823 08:55:40.240483 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:05:40.350579 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:15:40.460618 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:25:40.558987 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:35:40.669566 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:45:40.780369 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 09:55:40.891625 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 10:05:40.990028 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 10:15:41.098749 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 10:25:41.205286 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 10:35:41.312054 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> I0823 10:45:41.808549 102562 catalog-server.cc:237] Catalog Version: 282 Last
> Catalog Version: 282
> {code}
> Statestore log
> {code}
> I0823 11:05:31.107852 102250 statestore.cc:476] Received request for
> different delta base of topic: catalog-update from:
> [email protected]:22000 subscriber from_version: 0
> I0823 11:05:33.108608 102241 statestore.cc:549] Preparing initial
> catalog-update topic update for [email protected]:22000. Size = 7.83 MB
> I0823 11:05:33.502251 102241 statestore.cc:476] Received request for
> different delta base of topic: catalog-update from:
> [email protected]:22000 subscriber from_version: 0
> I0823 11:05:35.503106 102248 statestore.cc:549] Preparing initial
> catalog-update topic update for [email protected]:22000. Size = 7.83 MB
> I0823 11:05:35.903831 102248 statestore.cc:476] Received request for
> different delta base of topic: catalog-update from:
> [email protected]:22000 subscriber from_version: 0
> I0823 11:05:37.904654 102254 statestore.cc:549] Preparing initial
> catalog-update topic update for [email protected]:22000. Size = 7.83 MB
> {code}
> On vd1305 the StatestoreSubscriber-1 thread appears to be stuck in the stack
> below:
> {code}
> Thread 1 (process 96924):
> #0 0x00000035f960e82d in read () from /lib64/libpthread.so.0
> #1 0x00000035fc2dea71 in ?? () from /usr/lib64/libcrypto.so.10
> #2 0x00000035fc2dcdc9 in BIO_read () from /usr/lib64/libcrypto.so.10
> #3 0x00000035ff62c1df in ssl3_read_n () from /usr/lib64/libssl.so.10
> #4 0x00000035ff62c8dd in ssl3_read_bytes () from /usr/lib64/libssl.so.10
> #5 0x00000035ff6281a0 in ?? () from /usr/lib64/libssl.so.10
> #6 0x0000000001b432e2 in
> apache::thrift::transport::TSSLSocket::read(unsigned char*, unsigned int) ()
> #7 0x0000000001b3fe13 in unsigned int
> apache::thrift::transport::readAll<apache::thrift::transport::TSocket>(apache::thrift::transport::TSocket&,
> unsigned char*, unsigned int) ()
> #8 0x0000000000b38bea in
> apache::thrift::transport::TSaslTransport::readLength() ()
> #9 0x0000000000b38e76 in
> apache::thrift::transport::TSaslTransport::read(unsigned char*, unsigned int)
> ()
> #10 0x0000000001b47092 in
> apache::thrift::transport::TBufferedTransport::peek() ()
> #11 0x00000000009d7cf2 in
> apache::thrift::server::TAcceptQueueServer::Task::run() ()
> #12 0x00000000009d21f9 in
> impala::ThriftThread::RunRunnable(boost::shared_ptr<apache::thrift::concurrency::Runnable>,
> impala::Promise<unsigned long>*) ()
> #13 0x00000000009d2c52 in
> boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,
> boost::_mfi::mf2<void, impala::ThriftThread,
> boost::shared_ptr<apache::thrift::concurrency::Runnable>,
> impala::Promise<unsigned long>*>,
> boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>,
> boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >,
> boost::_bi::value<impala::Promise<unsigned long>*> > >,
> void>::invoke(boost::detail::function::function_buffer&) ()
> #14 0x0000000000bd0952 in
> impala::Thread::SuperviseThread(std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >
> const&, boost::function<void ()()>, impala::Promise<long>*) ()
> #15 0x0000000000bd12a4 in boost::detail::thread_data<boost::_bi::bind_t<void,
> void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&, boost::function<void ()()>,
> impala::Promise<long>*),
> boost::_bi::list4<boost::_bi::value<std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > >,
> boost::_bi::value<std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > >, boost::_bi::value<boost::function<void ()()> >,
> boost::_bi::value<impala::Promise<long>*> > > >::run() ()
> #16 0x0000000000e591ca in thread_proxy ()
> #17 0x00000035f9607aa1 in start_thread () from /lib64/libpthread.so.0
> #18 0x00000035f92e893d in clone () from /lib64/libc.so.6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
