subject:"\[jira\] \[Commented\] \(IMPALA\-6970\) DiskMgr\:\:AllocateBuffersForRange crashes on failed DCHECK"

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-08 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467633#comment-16467633
 ] 

ASF subversion and git services commented on IMPALA-6970:
-

Commit 18455ec6f8dda2f1d47bf0bc2ed10eb847d27b9c in impala's branch 
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=18455ec ]

IMPALA-6970: race with decreasing scanner reservation

The usage pattern of DecreaseReservationTo() in
ReturnReservationFromScannerThread() can decrease the reservation by
more than 'bytes' if there is a concurrent IncreaseReservation() call.

The API is changed to fit the requirements of the scanner, which
does not want to release more than the memory reserved for that
scanner thread.

Testing:
Added some backend tests (including concurrency) for the new API.

Change-Id: I653225c981bf674d2b2b947f3a3cb4d8f382d36b
Reviewed-on: http://gerrit.cloudera.org:8080/10314
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 


> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-08 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467631#comment-16467631
 ] 

ASF subversion and git services commented on IMPALA-6970:
-

Commit 25fb6e30ef4c70c4097e70cfa47e90d78e2313a1 in impala's branch 
refs/heads/2.x from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=25fb6e3 ]

IMPALA-6970: race with decreasing scanner reservation

The usage pattern of DecreaseReservationTo() in
ReturnReservationFromScannerThread() can decrease the reservation by
more than 'bytes' if there is a concurrent IncreaseReservation() call.

The API is changed to fit the requirements of the scanner, which
does not want to release more than the memory reserved for that
scanner thread.

Testing:
Added some backend tests (including concurrency) for the new API.

Change-Id: I653225c981bf674d2b2b947f3a3cb4d8f382d36b
Reviewed-on: http://gerrit.cloudera.org:8080/10314
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 


> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-04 Thread Tim Armstrong (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464283#comment-16464283
 ] 

Tim Armstrong commented on IMPALA-6970:
---

I can't reproduce locally but there is definitely a race that can happen if a 
thread returns its reservation at the same time as another is increasing it.

> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-03 Thread Tim Armstrong (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463231#comment-16463231
 ] 

Tim Armstrong commented on IMPALA-6970:
---

So the failing test is

{noformat} 09:30:36 
query_test/test_cancellation.py::TestCancellationSerial::test_cancel_insert[table_format:
 seq/gzip/block | exec_option: {'batch_size': 0, 'num_nodes': 0, 
'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 
0} | query_type: SELECT | cancel_delay: 3 | action: None | query: compute stats 
lineitem | buffer_pool_limit: 0] FAILED
{noformat}

It might be happening on the local filesystem build only because there are 
multiple scan ranges scheduled on the same daemon.

> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Priority: Blocker
>  Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-03 Thread Tim Armstrong (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463230#comment-16463230
 ] 

Tim Armstrong commented on IMPALA-6970:
---

{noformat}
(gdb) p **scan_range
$29 = (impala::io::ScanRange) {
   = {
::Node> = {
  _vptr.Node = 0x61a6ef0 , 
  parent_queue = 0x0, 
  next = 0x0, 
  prev = 0x0
}, 
members of impala::io::RequestRange: 
fs_ = 0xbf86828, 
file_ = {
  static npos = , 
  _M_dataplus = {
 = {
  <__gnu_cxx::new_allocator> = {}, }, 
members of std::basic_string::_Alloc_hider: 
_M_p = 0xbef1f18 
"file:/tmp/test-warehouse/tpch.lineitem_seq_gzip/00_0"
  }
}, 
offset_ = 33554432, 
len_ = 9276980, 
disk_id_ = 1, 
request_type_ = impala::io::RequestType::READ
  }, 
{noformat}

> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Priority: Blocker
>  Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-03 Thread Tim Armstrong (JIRA)


[ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463216#comment-16463216
 ] 

Tim Armstrong commented on IMPALA-6970:
---

{noformat}
(gdb) p runtime_state_->query_state_->query_ctx_.client_request.stmt
$24 = {
  static npos = , 
  _M_dataplus = {
 = {
  <__gnu_cxx::new_allocator> = {}, }, 
members of std::basic_string::_Alloc_hider: 
_M_p = 0xaea0e58 "SELECT COUNT(*) FROM lineitem"
  }
}
{noformat}

> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Priority: Blocker
>  Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

6 matches

Site Navigation

Mail list logo

Footer information