[
https://issues.apache.org/jira/browse/IMPALA-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-11192.
-------------------------------------
Fix Version/s: Impala 4.1.0
Resolution: Fixed
> test_scanner_fuzz.py runs super slow on ORC format
> --------------------------------------------------
>
> Key: IMPALA-11192
> URL: https://issues.apache.org/jira/browse/IMPALA-11192
> Project: IMPALA
> Issue Type: Test
> Components: Infrastructure
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Fix For: Impala 4.1.0
>
>
> I recently need to iterate test_scanner_fuzz.py multiple times and find it
> takes more than 0.5h to run it once (only for ORC).
> {code:bash}
> $ time -p impala-py.test --skip_hbase --table_formats=orc/def/block
> tests/query_test/test_scanners_fuzz.py
> real 2155.47
> user 2779.64
> sys 193.76
> {code}
> Looking into a Jenkins job, it shows that ORC tests are much slower than
> other formats:
> ||Test name||Duration||Status||
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|31 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}2 min 1 sec{color}|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|35 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|48 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|40 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}2 min 55 sec{color}|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|22 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|29 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
> avro/snap/block]|32 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
> orc/def/block]|{color:#FF0000}3 min 25 sec{color}|Passed|
> |test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
> parquet/none]|29 sec|Passed|
> |test_fuzz_alltypes[... 'debug_action': None, \\| table_format: text/none]|20
> sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|20 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}1 min 35 sec{color}|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|22 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|18 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|20 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}1 min 16 sec{color}|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|17 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|16 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
> avro/snap/block]|19 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
> orc/def/block]|{color:#FF0000}1 min 4 sec{color}|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
> parquet/none]|22 sec|Passed|
> |test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
> text/none]|29 sec|Passed|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|4 sec|Skipped|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}1 min 35 sec{color}|Passed|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|18 sec|Passed|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|4.7 sec|Skipped|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|5.2 sec|Skipped|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}1 min 59 sec{color}|Passed|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|17 sec|Passed|
> |test_fuzz_nested_types[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|4.3 sec|Skipped|
> |test_fuzz_nested_types[... 'debug_action': None \\| table_format:
> avro/snap/block]|4.1 sec|Skipped|
> |test_fuzz_nested_types[... 'debug_action': None \\| table_format:
> orc/def/block]|{color:#FF0000}2 min 30 sec{color}|Passed|
> |test_fuzz_nested_types[... 'debug_action': None \\| table_format:
> parquet/none]|18 sec|Passed|
> |test_fuzz_nested_types[... 'debug_action': None \\| table_format:
> text/none]|3.5 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|4 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}17 min{color}|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|20 sec|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|3.1 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> avro/snap/block]|3.8 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> orc/def/block]|{color:#FF0000}9 min 51 sec{color}|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> parquet/none]|19 sec|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action':
> '-1:OPEN:[email protected]' \\| table_format:
> text/none]|4.4 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\|
> table_format: avro/snap/block]|3.9 sec|Skipped|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\|
> table_format: orc/def/block]|{color:#FF0000}9 min 27 sec{color}|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\|
> table_format: parquet/none]|19 sec|Passed|
> |test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\|
> table_format: text/none]|4.3 sec|Skipped|
> Tests on other formats take less than 1min for each. ORC tests usually takes
> several minutes.
> CC [~boroknagyz]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
