[
https://issues.apache.org/jira/browse/IMPALA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900972#comment-16900972
]
Daniel Becker commented on IMPALA-8833:
---------------------------------------
The problem seems to be here:
[https://github.com/apache/impala/blob/bbe064ec194aff4ecf1e794bd4071df4ea4be166/be/src/util/dict-encoding.h#L256]
The dict decoder checks the bit width. So far the maximum bit width for bit
packing was 32 and bit widths like 40 were caught here. Now that the maximum is
64, it is not caught but when the decoder tries to write it to a uint32_t it
DCHECKs.
> Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in
> BatchedBitReader::UnpackBatch()
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-8833
> URL: https://issues.apache.org/jira/browse/IMPALA-8833
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Daniel Becker
> Priority: Blocker
> Labels: broken-build, crash, flaky
>
> {noformat}
> F0801 21:24:10.571285 15993 bit-stream-utils.inline.h:126]
> d04ba69d5da8ffd1:a9045b8200000001] Check failed: bit_width <= sizeof(T) * 8
> (40 vs. 32)
> *** Check failure stack trace: ***
> @ 0x52f63ac google::LogMessage::Fail()
> @ 0x52f7c51 google::LogMessage::SendToLog()
> @ 0x52f5d86 google::LogMessage::Flush()
> @ 0x52f934d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2b265b5 impala::BatchedBitReader::UnpackBatch<>()
> @ 0x2ae8623 impala::RleBatchDecoder<>::FillLiteralBuffer()
> @ 0x2b2cadb impala::RleBatchDecoder<>::DecodeLiteralValues<>()
> @ 0x2b27bfb impala::DictDecoder<>::DecodeNextValue()
> @ 0x2b16fed
> impala::ScalarColumnReader<>::ReadSlotsNoConversion()
> @ 0x2ac7252 impala::ScalarColumnReader<>::ReadSlots()
> @ 0x2a76cef
> impala::ScalarColumnReader<>::MaterializeValueBatchRepeatedDefLevel()
> @ 0x2a58faa impala::ScalarColumnReader<>::ReadValueBatch<>()
> @ 0x2a20e8e
> impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch()
> @ 0x29b189c impala::HdfsParquetScanner::AssembleRows()
> @ 0x29ac6de impala::HdfsParquetScanner::GetNextInternal()
> @ 0x29aa656 impala::HdfsParquetScanner::ProcessSplit()
> @ 0x249172d impala::HdfsScanNode::ProcessSplit()
> @ 0x2490902 impala::HdfsScanNode::ScannerThread()
> @ 0x248fc8b
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x2492253
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6915
> Log lines around the failure:
> {noformat}
> [gw5] PASSED
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q7[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q10a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw10] PASSED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw6] FAILED
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_decimal_ops[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> [gw2] FAILED
> query_test/test_scanners.py::TestParquet::test_type_widening[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action':
> 'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw4] FAILED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw12] FAILED
> query_test/test_scanners.py::TestParquet::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw8] ERROR
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw13] FAILED
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> [gw5] ERROR
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw11] FAILED
> query_test/test_scanners.py::TestParquet::test_corrupt_footer_len_decr[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> [gw10] ERROR
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: text/none]
> [gw9] FAILED
> query_test/test_insert_parquet.py::TestInsertParquetQueries::test_insert_parquet[compression_codec:
> snappy | protocol: beeswax | exec_option: {'sync_ddl': 1, 'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> table_format: parquet/none | file_size: 0-unique_database0]
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]',
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> {noformat}
> I'm guessing this is the scanner fuzzing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
