[
https://issues.apache.org/jira/browse/IMPALA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903101#comment-16903101
]
ASF subversion and git services commented on IMPALA-8833:
---------------------------------------------------------
Commit ef796639451af63bc9f4dbdc2a79bfc2f5048a93 in impala's branch
refs/heads/master from Daniel Becker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ef79663 ]
IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4)
The actual DCHECK failure was fixed by the change for IMPALA-8833.
Added a DCHECK to RleBatchDecoder so that it does not accept bit widths
higher than the width of its type parameter.
Also preventing UnpackAndDecodeValues from using higher bit widths than
32, the width of the dictionary index type, with a static assert. This
also reduces compile time because the compiler does not have to generate
code for invalid bit widths.
Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16
Reviewed-on: http://gerrit.cloudera.org:8080/14029
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in
> BatchedBitReader::UnpackBatch()
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-8833
> URL: https://issues.apache.org/jira/browse/IMPALA-8833
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Daniel Becker
> Priority: Blocker
> Labels: broken-build, crash, flaky
> Fix For: Impala 3.3.0
>
>
> {noformat}
> F0801 21:24:10.571285 15993 bit-stream-utils.inline.h:126]
> d04ba69d5da8ffd1:a9045b8200000001] Check failed: bit_width <= sizeof(T) * 8
> (40 vs. 32)
> *** Check failure stack trace: ***
> @ 0x52f63ac google::LogMessage::Fail()
> @ 0x52f7c51 google::LogMessage::SendToLog()
> @ 0x52f5d86 google::LogMessage::Flush()
> @ 0x52f934d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2b265b5 impala::BatchedBitReader::UnpackBatch<>()
> @ 0x2ae8623 impala::RleBatchDecoder<>::FillLiteralBuffer()
> @ 0x2b2cadb impala::RleBatchDecoder<>::DecodeLiteralValues<>()
> @ 0x2b27bfb impala::DictDecoder<>::DecodeNextValue()
> @ 0x2b16fed
> impala::ScalarColumnReader<>::ReadSlotsNoConversion()
> @ 0x2ac7252 impala::ScalarColumnReader<>::ReadSlots()
> @ 0x2a76cef
> impala::ScalarColumnReader<>::MaterializeValueBatchRepeatedDefLevel()
> @ 0x2a58faa impala::ScalarColumnReader<>::ReadValueBatch<>()
> @ 0x2a20e8e
> impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch()
> @ 0x29b189c impala::HdfsParquetScanner::AssembleRows()
> @ 0x29ac6de impala::HdfsParquetScanner::GetNextInternal()
> @ 0x29aa656 impala::HdfsParquetScanner::ProcessSplit()
> @ 0x249172d impala::HdfsScanNode::ProcessSplit()
> @ 0x2490902 impala::HdfsScanNode::ScannerThread()
> @ 0x248fc8b
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x2492253
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6915
> Log lines around the failure:
> {noformat}
> [gw5] PASSED
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q7[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q10a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw10] PASSED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw6] FAILED
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_decimal_ops[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> [gw2] FAILED
> query_test/test_scanners.py::TestParquet::test_type_widening[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action':
> 'HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw4] FAILED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> [gw12] FAILED
> query_test/test_scanners.py::TestParquet::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw8] ERROR
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw13] FAILED
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action': None,
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> [gw5] ERROR
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:[email protected]',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw11] FAILED
> query_test/test_scanners.py::TestParquet::test_corrupt_footer_len_decr[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> [gw10] ERROR
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: parquet/none]
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]', 'abort_on_error': False,
> 'mem_limit': '512m', 'num_nodes': 0} | table_format: text/none]
> [gw9] FAILED
> query_test/test_insert_parquet.py::TestInsertParquetQueries::test_insert_parquet[compression_codec:
> snappy | protocol: beeswax | exec_option: {'sync_ddl': 1, 'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> table_format: parquet/none | file_size: 0-unique_database0]
> query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_naaj[protocol:
> beeswax | exec_option: {'debug_action':
> '-1:OPEN:[email protected]',
> 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
> query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]
> {noformat}
> I'm guessing this is the scanner fuzzing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
