On Sat, Jun 13, 2009 at 9:50 AM, Mateusz Berezecki<[email protected]> wrote:
>
> Hi list
>
> I've managed to trigger one of the errors that were mentioned. It
> seems like all of them get triggered on filesystem write operations
> for some reason.
Can you tell us what's in the thrift broker log?
> Here's the relevant part of the stack traces from all threads
>
> Thread 1 (Thread 0xb7f6a700 (LWP 27256)):
> #0 0x00a4b402 in __kernel_vsyscall ()
> #1 0x0065e472 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib/tls/i686/nosegneg/libpthread.so.0
> ---Type <return> to continue, or q <return> to quit---
> #2 0x084c69c2 in
> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex>
>> (this=0x949fbe0, m...@0xbfde0930, wait_unt...@0xbfde08d0)
> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:110
> #3 0x084c6abc in
> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex>
>> (this=0x949fbe0, m...@0xbfde0930, wait_unt...@0xbfde0924)
> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:127
> #4 0x084c6b31 in
> Hypertable::TableMutatorCompletionCounter::wait_for_completion
> (this=0x949fbc8, tim...@0xbfde0b88)
> at
> /home/mateusz/episcale-hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:70
> #5 0x084c0b04 in
> Hypertable::TableMutatorScatterBuffer::wait_for_completion
> (this=0x949fb78, tim...@0xbfde0b88)
> at
> /home/mateusz/episcale-hypertable/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:262
> #6 0x084ba444 in Hypertable::TableMutator::wait_for_previous_buffer
> (this=0x948a9f8, tim...@0xbfde0b88) at
> /home/mateusz/episcale-hypertable/src/cc/Hypertable/Lib/TableMutator.cc:283
> #7 0x084bc762 in Hypertable::TableMutator::flush (this=0x948a9f8) at
> /home/mateusz/episcale-hypertable/src/cc/Hypertable/Lib/TableMutator.cc:233
> #8 0x08410c9c in
> Hypertable::ThriftBroker::ServerHandler::flush_mutator
> (this=0x93b7c38, mutator=155757048, tok...@0xbfde2fdc)
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/ThriftBroker.cc:1395
> #9 0x084165da in
> Hypertable::ThriftBroker::ServerHandler::close_mutator
> (this=0x93b7c38, mutator=155757048, tok...@0xbfde2fdc, flush=true)
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/ThriftBroker.cc:1417
> #10 0x0843cb06 in
> Hypertable::ThriftGen::ClientServiceProcessor::process_close_mutator
> (this=0x93b78a8, seqid=0, iprot=0x93d27a8, oprot=0x941afe8)
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/gen-cpp/ClientService.cpp:7153
> #11 0x08431a5c in
> Hypertable::ThriftGen::ClientServiceProcessor::process_fn
> (this=0x93b78a8, iprot=0x93d27a8, oprot=0x941afe8, fna...@0xbfde3164,
> seqid=0)
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/gen-cpp/ClientService.cpp:6708
> #12 0x0845de17 in
> Hypertable::ThriftGen::HqlServiceProcessor::process_fn
> (this=0x93b78a8, iprot=0x93d27a8, oprot=0x941afe8, fna...@0xbfde3164,
> seqid=0)
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/gen-cpp/HqlService.cpp:582
> #13 0x0845e4e5 in Hypertable::ThriftGen::HqlServiceProcessor::process
> (this=0x93b78a8, piprot={px = 0xbfde31c0, pn = {pi_ = 0xbfde31b8}},
> poprot={px = 0xbfde31b8, pn = {pi_ = 0x1}})
> at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/gen-cpp/HqlService.cpp:575
> #14 0x00576551 in apache::thrift::server::TConnection::transition
> (this=0x93b9118) at src/server/TNonblockingServer.cpp:303
> #15 0x00576d89 in apache::thrift::server::TConnection::workSocket
> (this=0x93b9118) at src/server/TNonblockingServer.cpp:147
> #16 0x0057c5c5 in apache::thrift::server::TConnection::eventHandler
> (fd=13, v=0x93b9118) at src/server/TNonblockingServer.h:418
> #17 0x005875dc in event_base_loop () from /usr/lib/libevent-1.3e.so.1
> #18 0x0057555b in apache::thrift::server::TNonblockingServer::serve
> (this=0xbfde336c) at src/server/TNonblockingServer.cpp:747
> #19 0x0839e647 in main (argc=Cannot access memory at address 0x0
> ) at
> /home/mateusz/episcale-hypertable/src/cc/ThriftBroker/ThriftBroker.cc:1818
> (gdb) detach
>
>
> It gets triggered once, then any subsequent call afterwards works for
> some time. After some period of inactivity the problem re-appears.
You mean after the timeout. The writes will work again until later?
__Luke
>
> Mateusz
>
> On Wed, Jun 10, 2009 at 2:04 AM, Josh Adams<[email protected]> wrote:
>>
>> On Tue, Jun 9, 2009 at 4:51 PM, Luke<[email protected]> wrote:
>>> Is there any request timeout errors in the thrift broker log after the
>>> restart?
>>
>> There might be but I'm not sure because I think the log lines I sent
>> were from before I restarted.
>>
>>> You can do me a favor by restarting with --log-api true and see which
>>> queries are taking a long time, next time you need to do a restart.
>>
>> Will do, thanks!
>>
>> Josh
>>
>> >
>>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
