[
https://issues.apache.org/jira/browse/IMPALA-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890753#comment-17890753
]
ASF subversion and git services commented on IMPALA-13286:
----------------------------------------------------------
Commit e4c05fce5998a1c2d10e111cf72fed00995ec85f in impala's branch
refs/heads/master from Xuebin Su
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e4c05fce5 ]
IMPALA-13286: Make stream row counters 64-bit
Previously, adding a large number of zero-sized rows to a
BufferedTupleStream could cause impalad to crash. This was because
the row counters for a page include
- Page::num_rows, and
- ReadIterator::read_page_rows_returned_
were 32-bit integers. When the row size is zero, all rows are
added to one single page. And if the number of rows in the whole
stream exceeds the limit, the two counters will overflow, making the
DCHECK failed.
This patch fixes the issue by making the row counters 64-bit integers
so that they will not easily overflow even when row size is zero.
DCHECKs are also added to make sure that overflow will not happen.
Testing:
- Added unit test StreamStateTest::TestAddAndGetZeroSizedRows to ensure
it is OK to add and get rows when the number of rows in the stream
exceeds INT_MAX or UINT_MAX.
- Ran the previously failed query manually and it completed with the
correct result after the patch.
Change-Id: I4d4cf8f424360717de0c4a5571a638a4c11b9606
Reviewed-on: http://gerrit.cloudera.org:8080/21741
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Zero-sized rows in BufferedTupleStream can cause impalad to crash
> -----------------------------------------------------------------
>
> Key: IMPALA-13286
> URL: https://issues.apache.org/jira/browse/IMPALA-13286
> Project: IMPALA
> Issue Type: Bug
> Reporter: Xuebin Su
> Assignee: Xuebin Su
> Priority: Major
> Fix For: Impala 4.5.0
>
>
> When running the following query on the DEBUG build,
> {code:java}
> with s as (
> select row_number() over (order by 1) as i
> from tl as a, tl as b
> )
> select count(i) from s; {code}
> where `tl` can be any table with millions of rows, impalad will crash with
> the following error:
> {code:java}
> F0808 10:35:59.101099 1111686 buffered-tuple-stream.h:552]
> c64e98ad45ff13ec:0eb3838c00000000] Check failed: read_page_->num_rows >=
> read_page_rows_returned_ (0 vs. 4294965248)
> *** Check failure stack trace: ***
> @ 0x3a2760d google::LogMessage::Fail()
> @ 0x3a29524 google::LogMessage::SendToLog()
> @ 0x3a26fec google::LogMessage::Flush()
> @ 0x3a29a49 google::LogMessageFatal::~LogMessageFatal()
> @ 0x235272a impala::BufferedTupleStream::GetNextInternal<>()
> @ 0x2352d8c impala::BufferedTupleStream::GetNextInternal<>()
> @ 0x23500a9 impala::BufferedTupleStream::GetNext()
> @ 0x1bf6c35 impala::AnalyticEvalNode::GetNextOutputBatch()
> @ 0x1bfa759 impala::AnalyticEvalNode::GetNext()
> @ 0x1c5637f impala::AggregationNode::Open()
> @ 0x14ea9f3 impala::FragmentInstanceState::Open()
> @ 0x14eec09 impala::FragmentInstanceState::Exec()
> @ 0x146b528 impala::QueryState::ExecFInstance()
> @ 0x146b919
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1a7d108 impala::Thread::SuperviseThread()
> @ 0x1a7df11 boost::detail::thread_data<>::run()
> @ 0x2549267 thread_proxy
> @ 0x7b53ad294ac3 (unknown)
> @ 0x7b53ad326850 (unknown)
> Minidump in thread [1111686]exec-finstance
> (finst:c64e98ad45ff13ec:0eb3838c00000000) running query
> c64e98ad45ff13ec:0eb3838c00000000, fragment instance
> c64e98ad45ff13ec:0eb3838c00000000 {code}
> This is because the sizes of rows added to `input_stream_` of the
> `AnalyticEvalNode` are all zero bytes. In that case, all rows will be added
> to a single page. And if the number of rows exceeds the limit of
> `Page::num_rows`, which is `INT_MAX`, `Page::num_rows` will overflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
