[
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463255#comment-16463255
]
Tim Armstrong commented on IMPALA-6970:
---------------------------------------
I think there's some accounting issue with the scanner thread reservations- the
reservations from the scanner threads doesn't appear to add up to the clients
reservation. Extracted from the core:
{noformat}
scanner_thread_reservation=25165824
scanner_thread_reservation=16777216
scanner_thread_reservation=25165824
scanner_thread_reservation=25165824
scanner_thread_reservation=8388608
scanner_thread_reservation=16777216
scanner_thread_reservation=25165824
scanner_thread_reservation=25165824
total = 167772160
(gdb) p buffer_pool_client_->impl_->reservation_
$47 = (impala::ReservationTracker) {
_vptr.ReservationTracker = 0x6194350 <vtable for
impala::ReservationTracker+16>,
increase_deny_probability_ = 0,
lock_ = {
l_ = {
static LINKER_INITIALIZED = base::LINKER_INITIALIZED,
lockword_ = 2
}
},
initialized_ = true,
dummy_profile_ = {
px = 0x0
},
counters_ = {
peak_reservation = 0xe6ca020,
peak_used_reservation = 0xd437b00,
reservation_limit = 0x0
},
parent_ = 0xc07e3f0,
mem_tracker_ = 0xa879a40,
reservation_limit_ = 9223372036854775807,
reservation_ = 142606336,
child_reservations_ = 0,
used_reservation_ = 136314880
{noformat}
I think there might be some race between increasing reservations and
ReturnReservationFromScannerThread(), not sure if that's the cause
> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> ---------------------------------------------------------
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.13.0
> Reporter: Sailesh Mukil
> Priority: Blocker
> Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way.
> Cannot tell if the root cause is the same as that though without further
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes
> <= unused_reservation() (8388608 vs. 6291456)
> *** Check failure stack trace: ***
> @ 0x4277c1d google::LogMessage::Fail()
> @ 0x42794c2 google::LogMessage::SendToLog()
> @ 0x42775f7 google::LogMessage::Flush()
> @ 0x427abbe google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ef1343 impala::ReservationTracker::AllocateFromLocked()
> @ 0x1ef111d impala::ReservationTracker::AllocateFrom()
> @ 0x1ee8c57
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @ 0x1ee5543 impala::BufferPool::AllocateBuffer()
> @ 0x2f50f68 impala::io::DiskIoMgr::AllocateBuffersForRange()
> @ 0x1f74762 impala::HdfsScanNodeBase::StartNextScanRange()
> @ 0x1f6b052 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a4ea
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c5cc
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd4748 boost::function0<>::operator()()
> @ 0x1ebf349 impala::Thread::SuperviseThread()
> @ 0x1ec74e5 boost::_bi::list5<>::operator()<>()
> @ 0x1ec7409 boost::_bi::bind_t<>::operator()()
> @ 0x1ec73cc boost::detail::thread_data<>::run()
> @ 0x31a1f0a thread_proxy
> @ 0x36d1607851 (unknown)
> @ 0x36d12e894d (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
