[
https://issues.apache.org/jira/browse/IMPALA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901669#comment-16901669
]
ASF subversion and git services commented on IMPALA-8833:
---------------------------------------------------------
Commit 33f1e86ce3dc5e278b804004671062f46d42d90e in impala's branch
refs/heads/master from Daniel Becker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=33f1e86 ]
IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
After raising the maximum bit width for bit packing to 64 bits,
DictDecoder accepted bit widths between 32 and 64, but internally it
uses 32 bit integers and unpacking ran into a DCHECK.
Adding a check to DictDecoder to catch if the bit width is higher than
32.
Testing:
Added a test that asserts that DictDecoder accepts bit widths 0-32
and rejects higher bit widths which could still be unpacked
otherwise.
Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4
Reviewed-on: http://gerrit.cloudera.org:8080/14019
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Tim Armstrong <[email protected]>
> Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in
> BatchedBitReader::UnpackBatch()
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-8833
> URL: https://issues.apache.org/jira/browse/IMPALA-8833
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Daniel Becker
> Priority: Blocker
> Labels: broken-build, crash, flaky
>
> {noformat}
> F0801 21:24:10.571285 15993 bit-stream-utils.inline.h:126]
> d04ba69d5da8ffd1:a9045b8200000001] Check failed: bit_width <= sizeof(T) * 8
> (40 vs. 32)
> *** Check failure stack trace: ***
> @ 0x52f63ac google::LogMessage::Fail()
> @ 0x52f7c51 google::LogMessage::SendToLog()
> @ 0x52f5d86 google::LogMessage::Flush()
> @ 0x52f934d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2b265b5 impala::BatchedBitReader::UnpackBatch<>()
> @ 0x2ae8623 impala::RleBatchDecoder<>::FillLiteralBuffer()
> @ 0x2b2cadb impala::RleBatchDecoder<>::DecodeLiteralValues<>()
> @ 0x2b27bfb impala::DictDecoder<>::DecodeNextValue()
> @ 0x2b16fed
> impala::ScalarColumnReader<>::ReadSlotsNoConversion()
> @ 0x2ac7252 impala::ScalarColumnReader<>::ReadSlots()
> @ 0x2a76cef
> impala::ScalarColumnReader<>::MaterializeValueBatchRepeatedDefLevel()
> @ 0x2a58faa impala::ScalarColumnReader<>::ReadValueBatch<>()
> @ 0x2a20e8e
> impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch()
> @ 0x29b189c impala::HdfsParquetScanner::AssembleRows()
> @ 0x29ac6de impala::HdfsParquetScanner::GetNextInternal()
> @ 0x29aa656 impala::HdfsParquetScanner::ProcessSplit()
> @ 0x249172d impala::HdfsScanNode::ProcessSplit()
> @ 0x2490902 impala::HdfsScanNode::ScannerThread()
> @ 0x248fc8b
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x2492253
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6915
> Log lines around the failure:
> {noformat}
> [gw5] PASSED
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q7[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q10a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw10] PASSED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw6] FAILED
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_decimal_ops[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> [gw2] FAILED
> query_test/test_scanners.py::TestParquet::test_type_widening[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action':
> 'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw4] FAILED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw12] FAILED
> query_test/test_scanners.py::TestParquet::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw8] ERROR
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw13] FAILED
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> [gw5] ERROR
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw11] FAILED
> query_test/test_scanners.py::TestParquet::test_corrupt_footer_len_decr[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> [gw10] ERROR
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: text/none]
> [gw9] FAILED
> query_test/test_insert_parquet.py::TestInsertParquetQueries::test_insert_parquet[compression_codec:
> snappy | protocol: beeswax | exec_option: {'sync_ddl': 1, 'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> table_format: parquet/none | file_size: 0-unique_database0]
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]',
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> {noformat}
> I'm guessing this is the scanner fuzzing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
