[
https://issues.apache.org/jira/browse/KUDU-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-2561:
--------------------------------
Description:
A reference to {{Tablet::schema_lock_}} member is kept in
{{AlterSchemaTransactionState}} instance after call to
AlterSchemaTransactionState::AcquireSchemaLock(&schema_lock_). If the instance
of the {{Tablet}} is destroyed when there are some pending AlterSchema
transactions, there is use-after-free condition upon destroying
AlterSchemaTransactionState later on.
The issues is reproducible in about 1 of 100 ASAN runs of the
{{MasterStressTest.Test}} parameterized scenario ({{master-stress-test}}
binary) with {{--stress_cpu_threads=16}}. The full ASAN trace is attached.
{noformat}
=================================================================
==6745==ERROR: AddressSanitizer: heap-use-after-free on address 0x61700008c788
at pc 0x7f522c881ef4 bp 0x7f51fa39b970 sp 0x7f51fa39b968
READ of size 4 at 0x61700008c788 thread T56 (rpc worker-6837)
#0 0x7f522c881ef3 in base::subtle::NoBarrier_Load(int const volatile*)
/data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
#1 0x7f522a64c43c in kudu::rw_semaphore::unlock()
/data/8/aserbin/Projects/kudu/src/kudu/util/rw_semaphore.h:148:5
#2 0x7f522a6917e1 in std::unique_lock<kudu::rw_semaphore>::unlock()
../../../include/c++/4.9.2/mutex:530:17
#3 0x7f522a691729 in std::unique_lock<kudu::rw_semaphore>::~unique_lock()
../../../include/c++/4.9.2/mutex:439:4
#4 0x7f522a689437 in
kudu::tablet::AlterSchemaTransactionState::~AlterSchemaTransactionState()
/data/8/aserbin/Projects/kudu/src/kudu/tablet/transactions/alter_schema_transaction.h:49:3
...
0x61700008c788 is located 8 bytes inside of 704-byte region
[0x61700008c780,0x61700008ca40)
freed by thread T128 (tablet-delete [) here:
#0 0x532670 in operator delete(void*)
/data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:149
#1 0x5383a6 in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
../../../include/c++/4.9.2/bits/shared_ptr_base.h:149:6
#2 0x7f522a6cba1b in std::__shared_ptr<kudu::tablet::Tablet,
(__gnu_cxx::_Lock_policy)2>::reset()
../../../include/c++/4.9.2/bits/shared_ptr_base.h:1015:9
#3 0x7f522a6bc66f in kudu::tablet::TabletReplica::Stop()
/data/8/aserbin/Projects/kudu/src/kudu/tablet/tablet_replica.cc:300:13
#4 0x7f522c9aa94c in
kudu::tserver::TSTabletManager::DeleteTablet(std::string const&,
kudu::tablet::TabletDataState, boost::optional<long> const&,
kudu::tserver::TabletServerErrorPB_Code*)
/data/8/aserbin/Projects/kudu/src/kudu/tserver/ts_tablet_manager.cc:918:12
...
SUMMARY: AddressSanitizer: heap-use-after-free
/data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
in base::subtle::NoBarrier_Load(int const volatile*)
Shadow bytes around the buggy address:
0x0c2e800098a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098d0: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e800098e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c2e800098f0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009940: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
{noformat}
was:
A reference to {{Tablet::schema_lock_}} member is kept in
{{AlterSchemaTransactionState}} instance after call to
AlterSchemaTransactionState::AcquireSchemaLock(&schema_lock_). If the instance
of the {{Tablet}} is destroyed when there are some pending AlterSchema
transactions, there is use-after-free condition upon destroying
AlterSchemaTransactionState later on. The full ASAN trace is attached.
{noformat}
=================================================================
==6745==ERROR: AddressSanitizer: heap-use-after-free on address 0x61700008c788
at pc 0x7f522c881ef4 bp 0x7f51fa39b970 sp 0x7f51fa39b968
READ of size 4 at 0x61700008c788 thread T56 (rpc worker-6837)
#0 0x7f522c881ef3 in base::subtle::NoBarrier_Load(int const volatile*)
/data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
#1 0x7f522a64c43c in kudu::rw_semaphore::unlock()
/data/8/aserbin/Projects/kudu/src/kudu/util/rw_semaphore.h:148:5
#2 0x7f522a6917e1 in std::unique_lock<kudu::rw_semaphore>::unlock()
../../../include/c++/4.9.2/mutex:530:17
#3 0x7f522a691729 in std::unique_lock<kudu::rw_semaphore>::~unique_lock()
../../../include/c++/4.9.2/mutex:439:4
#4 0x7f522a689437 in
kudu::tablet::AlterSchemaTransactionState::~AlterSchemaTransactionState()
/data/8/aserbin/Projects/kudu/src/kudu/tablet/transactions/alter_schema_transaction.h:49:3
...
0x61700008c788 is located 8 bytes inside of 704-byte region
[0x61700008c780,0x61700008ca40)
freed by thread T128 (tablet-delete [) here:
#0 0x532670 in operator delete(void*)
/data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:149
#1 0x5383a6 in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
../../../include/c++/4.9.2/bits/shared_ptr_base.h:149:6
#2 0x7f522a6cba1b in std::__shared_ptr<kudu::tablet::Tablet,
(__gnu_cxx::_Lock_policy)2>::reset()
../../../include/c++/4.9.2/bits/shared_ptr_base.h:1015:9
#3 0x7f522a6bc66f in kudu::tablet::TabletReplica::Stop()
/data/8/aserbin/Projects/kudu/src/kudu/tablet/tablet_replica.cc:300:13
#4 0x7f522c9aa94c in
kudu::tserver::TSTabletManager::DeleteTablet(std::string const&,
kudu::tablet::TabletDataState, boost::optional<long> const&,
kudu::tserver::TabletServerErrorPB_Code*)
/data/8/aserbin/Projects/kudu/src/kudu/tserver/ts_tablet_manager.cc:918:12
...
SUMMARY: AddressSanitizer: heap-use-after-free
/data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
in base::subtle::NoBarrier_Load(int const volatile*)
Shadow bytes around the buggy address:
0x0c2e800098a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e800098d0: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2e800098e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c2e800098f0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2e80009940: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
{noformat}
> Heap use-after-free in AlterSchemaTransactionState upon concurrent
> TSTabletManager::DeleteTablet
> ------------------------------------------------------------------------------------------------
>
> Key: KUDU-2561
> URL: https://issues.apache.org/jira/browse/KUDU-2561
> Project: Kudu
> Issue Type: Bug
> Components: tserver
> Affects Versions: 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.9.0, 0.9.1, 1.0.0,
> 1.0.1, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1
> Reporter: Alexey Serbin
> Priority: Major
> Attachments: use-after-free-on-tablet-schema_lock_.txt
>
>
> A reference to {{Tablet::schema_lock_}} member is kept in
> {{AlterSchemaTransactionState}} instance after call to
> AlterSchemaTransactionState::AcquireSchemaLock(&schema_lock_). If the
> instance of the {{Tablet}} is destroyed when there are some pending
> AlterSchema transactions, there is use-after-free condition upon destroying
> AlterSchemaTransactionState later on.
> The issues is reproducible in about 1 of 100 ASAN runs of the
> {{MasterStressTest.Test}} parameterized scenario ({{master-stress-test}}
> binary) with {{--stress_cpu_threads=16}}. The full ASAN trace is attached.
> {noformat}
> =================================================================
> ==6745==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x61700008c788 at pc 0x7f522c881ef4 bp 0x7f51fa39b970 sp 0x7f51fa39b968
> READ of size 4 at 0x61700008c788 thread T56 (rpc worker-6837)
> #0 0x7f522c881ef3 in base::subtle::NoBarrier_Load(int const volatile*)
> /data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
> #1 0x7f522a64c43c in kudu::rw_semaphore::unlock()
> /data/8/aserbin/Projects/kudu/src/kudu/util/rw_semaphore.h:148:5
> #2 0x7f522a6917e1 in std::unique_lock<kudu::rw_semaphore>::unlock()
> ../../../include/c++/4.9.2/mutex:530:17
> #3 0x7f522a691729 in std::unique_lock<kudu::rw_semaphore>::~unique_lock()
> ../../../include/c++/4.9.2/mutex:439:4
> #4 0x7f522a689437 in
> kudu::tablet::AlterSchemaTransactionState::~AlterSchemaTransactionState()
> /data/8/aserbin/Projects/kudu/src/kudu/tablet/transactions/alter_schema_transaction.h:49:3
> ...
> 0x61700008c788 is located 8 bytes inside of 704-byte region
> [0x61700008c780,0x61700008ca40)
> freed by thread T128 (tablet-delete [) here:
> #0 0x532670 in operator delete(void*)
> /data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:149
> #1 0x5383a6 in
> std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
> ../../../include/c++/4.9.2/bits/shared_ptr_base.h:149:6
> #2 0x7f522a6cba1b in std::__shared_ptr<kudu::tablet::Tablet,
> (__gnu_cxx::_Lock_policy)2>::reset()
> ../../../include/c++/4.9.2/bits/shared_ptr_base.h:1015:9
> #3 0x7f522a6bc66f in kudu::tablet::TabletReplica::Stop()
> /data/8/aserbin/Projects/kudu/src/kudu/tablet/tablet_replica.cc:300:13
> #4 0x7f522c9aa94c in
> kudu::tserver::TSTabletManager::DeleteTablet(std::string const&,
> kudu::tablet::TabletDataState, boost::optional<long> const&,
> kudu::tserver::TabletServerErrorPB_Code*)
> /data/8/aserbin/Projects/kudu/src/kudu/tserver/ts_tablet_manager.cc:918:12
> ...
> SUMMARY: AddressSanitizer: heap-use-after-free
> /data/8/aserbin/Projects/kudu/src/kudu/gutil/atomicops-internals-x86.h:200:10
> in base::subtle::NoBarrier_Load(int const volatile*)
> Shadow bytes around the buggy address:
> 0x0c2e800098a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e800098b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e800098c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e800098d0: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c2e800098e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c2e800098f0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e80009900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e80009910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e80009920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e80009930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c2e80009940: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
