[
https://issues.apache.org/jira/browse/IMPALA-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655908#comment-16655908
]
ASF subversion and git services commented on IMPALA-7272:
---------------------------------------------------------
Commit 9f5c5e6df03824cba292fe5a619153462c11669c in impala's branch
refs/heads/master from [~twmarshall]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=9f5c5e6 ]
IMPALA-7272: Fix crash in StringMinMaxFilter
StringMinMaxFilter uses a MemPool to allocate space for StringBuffers.
Previously, the MemPool was created by RuntimeFilterBank and passed to
each StringMinMaxFilter. In queries with multiple StringMinMaxFilters
being generated by the same fragment instance, this leads to
overlapping use of the MemPool by different threads, which is
incorrect as MemPools are not thread-safe.
The solution is to have each StringMinMaxFilter create its own
MemPool.
This patch also documents MemPool as not thread-safe and introduces a
DFAKE_MUTEX to help enforce correct usage. Doing this requires
modifying our CMakeLists.txt to pass '-DNDEBUG' to clang only in
RELEASE builds, so that the DFAKE_MUTEX will be present in the
compiled IR for DEBUG builds.
Testing:
- I have been unable to repro the actual crash despite trying a large
variety of different things. However, with additional logging added
its clear that the MemPool is being used concurrently, which is
incorrect.
- Added an e2e test that covers the potential issue. It hits the
DFAKE_MUTEX with a sleep added to MemPool::Allocate.
- Ran a full exhaustive build in both DEBUG and RELEASE.
Change-Id: I751cad7e6b75c9d95d7ad029bbd1e52fe09e8a29
Reviewed-on: http://gerrit.cloudera.org:8080/11650
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> impalad crash when Fatigue test
> ---------------------------------
>
> Key: IMPALA-7272
> URL: https://issues.apache.org/jira/browse/IMPALA-7272
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0, Impala 2.12.0
> Environment: apache branch
> [329979d6fb0caa0dc449d7e0aa75460c30e868f0]
> centos 6.5
> ./buildall.sh -skiptests -noclean -asan
> Reporter: yyzzjj
> Assignee: Thomas Tauber-Marshall
> Priority: Blocker
> Labels: crash
> Attachments: e4386102-833c-40bb-4eec10b2-827c76be.dmp,
> impalad_node0.ERROR, impalad_node0.WARNING, testing_impala.sh
>
>
> (gdb) bt
> #0 0x0000003269832635 in raise () from /lib64/libc.so.6
> #1 0x0000003269833e15 in abort () from /lib64/libc.so.6
> #2 0x0000000004010f64 in google::DumpStackTraceAndExit() ()
> #3 0x00000000040079dd in google::LogMessage::Fail() ()
> #4 0x0000000004009282 in google::LogMessage::SendToLog() ()
> #5 0x00000000040073b7 in google::LogMessage::Flush() ()
> #6 0x000000000400a97e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x0000000001a2dfab in impala::MemPool::CheckIntegrity (this=0x5916e1f8,
> check_current_chunk_empty=true)
> at /export/ldb/online/kudu_rpc_branch/be/src/runtime/mem-pool.cc:258
> #8 0x0000000001a2cf56 in impala::MemPool::FindChunk (this=0x5916e1f8,
> min_size=10, check_limits=true) at
> /export/ldb/online/kudu_rpc_branch/be/src/runtime/mem-pool.cc:158
> #9 0x0000000001a3dd1b in impala::MemPool::Allocate<true> (alignment=8,
> size=10, this=0x5916e1f8) at
> /export/ldb/online/kudu_rpc_branch/be/src/runtime/mem-pool.h:273
> #10 impala::MemPool::TryAllocate (this=0x5916e1f8, size=10) at
> /export/ldb/online/kudu_rpc_branch/be/src/runtime/mem-pool.h:109
> #11 0x0000000001caefb8 in impala::StringBuffer::GrowBuffer
> (this=0x7f90d9489c28, new_size=10) at
> /export/ldb/online/kudu_rpc_branch/be/src/runtime/string-buffer.h:85
> #12 0x0000000001caee18 in impala::StringBuffer::Append (this=0x7f90d9489c28,
> str=0x7f92cda6e039 "1104700843don...@jd.com业务运营部\230\340\246͒\177",
> str_len=10)
> at /export/ldb/online/kudu_rpc_branch/be/src/runtime/string-buffer.h:53
> #13 0x0000000001cac864 in impala::StringMinMaxFilter::CopyToBuffer
> (this=0x7f90d9489c00, buffer=0x7f90d9489c28, value=0x7f90d9489c08, len=10)
> at /export/ldb/online/kudu_rpc_branch/be/src/util/min-max-filter.cc:304
> #14 0x0000000001cac2a9 in impala::StringMinMaxFilter::MaterializeValues
> (this=0x7f90d9489c00) at
> /export/ldb/online/kudu_rpc_branch/be/src/util/min-max-filter.cc:229
> #15 0x0000000002b9641a in impala::FilterContext::MaterializeValues
> (this=0x61cc0b70) at
> /export/ldb/online/kudu_rpc_branch/be/src/exec/filter-context.cc:97
> #16 0x00007f93fdb9440e in ?? ()
> #17 0x00007f90a97f5400 in ?? ()
> #18 0x2acd2bba01a2e0f7 in ?? ()
> #19 0x000000005916e140 in ?? ()
> #20 0x00007f930c34d740 in ?? ()
> #21 0x00007f90a97f5220 in ?? ()
> #22 0x66aa77bb66aa77bb in ?? ()
> #23 0x0000000061cc0b70 in ?? ()
> #24 0x0000000061cc0b70 in ?? ()
> #25 0x0000000061cc0b98 in ?? ()
> #26 0x0000000061cc0b70 in ?? ()
> #27 0x00007f90a97f5300 in ?? ()
> #28 0x0000000001ab84ed in
> impala::RuntimeFilterBank::AllocateScratchMinMaxFilter (this=<error reading
> variable: Cannot access memory at address 0xffffffffffffff4f>,
> filter_id=<error reading variable: Cannot access memory at address
> 0xffffffffffffff4b>,
> type=<error reading variable: Cannot access memory at address
> 0xffffffffffffff3f>) at
> /export/ldb/online/kudu_rpc_branch/be/src/runtime/runtime-filter-bank.cc:250
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
