[
https://issues.apache.org/jira/browse/IMPALA-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297771#comment-17297771
]
ASF subversion and git services commented on IMPALA-10529:
----------------------------------------------------------
Commit d89c04bf806682d3449c566ce979632bd2ac5b29 in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d89c04b ]
IMPALA-10529: Fix hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
For start option "scratch_dirs", it only considers local filesystem as
the default filesystem, regardless of the setting of DefaultFS(for a
remote scratch dir, it needs to explicitly set it with the remote fs
prefix). However, the function AssignQueue() would assign the queue
based on not only the path string but also the default filesystem
setting. For example, if scratch_dirs is set as "/tmp", the scratch dir
is supposed to be in the local filesystem, but the AssignQueue() would
consider it as "s3a://xxx/tmp" if a s3 path is set as the default fs.
To fix this, the solution is to add a bool variable to AssignQueue() to
decide whether or not to check the default fs setting when parsing the
file path. For all of the scratch dirs, AssignQueue() won't check the
default fs.
Tests:
Added a unit testcase: TmpFileMgrTest::TestSpillingWithRemoteDefaultFS.
Ran and Passed TmpFileMgrTest.
Change-Id: Ic07945abe65d90235aa8dea92dd3c3821a4f1f53
Reviewed-on: http://gerrit.cloudera.org:8080/17136
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
> -----------------------------------------------------
>
> Key: IMPALA-10529
> URL: https://issues.apache.org/jira/browse/IMPALA-10529
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Wenzhe Zhou
> Assignee: Yida Wu
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Hit DCHECK error in DiskIoMgr::AssignQueue when ran
> TestDiskSpillConfigurations::test_disk_spill_encryption_disabled and
> TestQueryRetries::test_query_retry_reaches_spool_limit.
> Stacks:
> Crash reason: SIGABRT
> Crash address: 0x7d1000061d5
> Process uptime: not available
> Thread 512 (crashed)
> 0 libc-2.17.so + 0x351f7
> rax = 0x0000000000000000 rdx = 0x0000000000000006
> rcx = 0xffffffffffffffff rbx = 0x0000000000000004
> rsi = 0x0000000000005a0b rdi = 0x00000000000061d5
> rbp = 0x00007f577787b8f0 rsp = 0x00007f577787b578
> r8 = 0x0000000000000000 r9 = 0x00007f577787b3f0
> r10 = 0x0000000000000008 r11 = 0x0000000000000202
> r12 = 0x00000000079b6d00 r13 = 0x0000000000000073
> r14 = 0x00000000079b6d04 r15 = 0x00000000079af320
> rip = 0x00007f586e7951f7
> Found by: given as instruction pointer in context
> 1 impalad!google::LogMessage::Flush() + 0x1eb
> rbp = 0x00007f577787b9d0 rsp = 0x00007f577787b900
> rip = 0x0000000005418d6b
> Found by: previous frame's frame pointer
> 2 impalad!google::LogMessageFatal::~LogMessageFatal() + 0x9
> rbx = 0x0000000000000001 rbp = 0x00007f577787ba60
> rsp = 0x00007f577787b9a0 r12 = 0x00000000079be238
> r13 = 0x0000000000000001 r14 = 0x000000001a6ccf90
> r15 = 0x000000001a7feab8 rip = 0x000000000541c969
> Found by: call frame info
> 3 impalad!impala::io::DiskIoMgr::AssignQueue(char const*, int, bool)
> [disk-io-mgr.cc : 825 + 0xf]
> rbx = 0x0000000000000001 rbp = 0x00007f577787ba60
> rsp = 0x00007f577787b9b0 r12 = 0x00000000079be238
> r13 = 0x0000000000000001 r14 = 0x000000001a6ccf90
> r15 = 0x000000001a7feab8 rip = 0x0000000002dda765
> Found by: call frame info
> 4 impalad!impala::TmpFile::AssignDiskQueue(bool) const [tmp-file-mgr.cc :
> 658 + 0x41]
> rbx = 0x0000000018bb95c0 rbp = 0x00007f577787baa0
> rsp = 0x00007f577787ba70 r12 = 0x00000000ffffffff
> r13 = 0x0000000000000001 r14 = 0x000000001a6ccf90
> r15 = 0x000000001a7feab8 rip = 0x000000000242dcd4
> Found by: call frame info
> 5 impalad!impala::TmpWriteHandle::Write(impala::io::RequestContext*,
> impala::MemRange, std::function<void (impala::Status const&)>,
> impala::BufferPoolClientCounters const*) [tmp-file-mgr.cc : 1519 + 0x26]
> rbx = 0x000000001a6ccf80 rbp = 0x00007f577787bc50
> rsp = 0x00007f577787bab0 r12 = 0x0000000020612f30
> r13 = 0x0000000020612f00 r14 = 0x000000001a6ccf90
> r15 = 0x000000001a7feab8 rip = 0x0000000002437565
> Found by: call frame info
> 6 impalad!impala::TmpFileGroup::Write(impala::MemRange, std::function<void
> (impala::Status const&)>, std::unique_ptr<impala::TmpWriteHandle,
> std::default_delete<impala::TmpWriteHandle> >*,
> impala::BufferPoolClientCounters const*) [tmp-file-mgr.cc : 1190 + 0x4d]
> rbx = 0x0000000020612f00 rbp = 0x00007f577787bdb0
> rsp = 0x00007f577787bc60 r12 = 0x0000000020e8f600
> r13 = 0x0000000020612f00 r14 = 0x00007f577787bd10
> r15 = 0x000000001a7feab8 rip = 0x000000000243333c
> Found by: call frame info
> 7 impalad!impala::BufferPool::Client::WriteDirtyPagesAsync(long)
> [buffer-pool.cc : 747 + 0x34]
> rbx = 0x0000000020e8f600 rbp = 0x00007f577787c010
> rsp = 0x00007f577787bdc0 r12 = 0x000000001e269b00
> r13 = 0x000000001a7fea80 r14 = 0x000000001e269ba0
> r15 = 0x000000001a7feab8 rip = 0x0000000002914b35
> Found by: call frame info
> 8
> impalad!impala::BufferPool::Client::MoveToDirtyUnpinned(impala::BufferPool::Page*)
> [buffer-pool.cc : 508 + 0x14]
> rbx = 0x0000000000000000 rbp = 0x00007f577787c0b0
> rsp = 0x00007f577787c020 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x00000000029123f2
> Found by: call frame info
> 9 impalad!impala::BufferPool::Unpin(impala::BufferPool::ClientHandle*,
> impala::BufferPool::PageHandle*) [buffer-pool.cc : 210 + 0x19]
> rbx = 0x0000000000000000 rbp = 0x00007f577787c190
> rsp = 0x00007f577787c0c0 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x000000000290ed91
> Found by: call frame info
> 10
> impalad!impala::BufferedTupleStream::UnpinPageIfNeeded(impala::BufferedTupleStream::Page*,
> bool) [buffered-tuple-stream.cc : 315 + 0x2b]
> rbx = 0x0000000000000000 rbp = 0x00007f577787c250
> rsp = 0x00007f577787c1a0 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x0000000002eb6305
> Found by: call frame info
> 11
> impalad!impala::BufferedTupleStream::UnpinStream(impala::BufferedTupleStream::UnpinMode)
> [buffered-tuple-stream.cc : 726 + 0x2b]
> rbx = 0x0000000000000000 rbp = 0x00007f577787c310
> rsp = 0x00007f577787c260 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x0000000002eb9994
> Found by: call frame info
> 12 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 227 + 0x2d]
> rbx = 0x0000000000000000 rbp = 0x00007f577787c540
> rsp = 0x00007f577787c320 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x0000000002b3c285
> Found by: call frame info
> 13 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 1022 + 0x3a]
> rbx = 0x0000000024ce1a70 rbp = 0x00007f577787c630
> rsp = 0x00007f577787c550 r12 = 0x000000001e90d600
> r13 = 0x0000000000000001 r14 = 0x0000000000000001
> r15 = 0x0000000016ac1538 rip = 0x0000000002b2b670
> Found by: call frame info
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
