Quanlong Huang created IMPALA-11192:
---------------------------------------
Summary: test_scanner_fuzz.py runs super slow on ORC format
Key: IMPALA-11192
URL: https://issues.apache.org/jira/browse/IMPALA-11192
Project: IMPALA
Issue Type: Test
Components: Infrastructure
Reporter: Quanlong Huang
Assignee: Quanlong Huang
I recently need to iterate test_scanner_fuzz.py multiple times and find it
takes more than 0.5h to run it once (only for ORC).
{code:bash}
$ time -p impala-py.test --skip_hbase --table_formats=orc/def/block
tests/query_test/test_scanners_fuzz.py
real 2155.47
user 2779.64
sys 193.76
{code}
Looking into a Jenkins job, it shows that ORC tests are much slower than other
formats:
||Test name||Duration||Status||
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|31 sec|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}2 min 1 sec{color}|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|35 sec|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|48
sec|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|40 sec|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}2 min 55 sec{color}|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|22 sec|Passed|
|test_fuzz_alltypes[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|29
sec|Passed|
|test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
avro/snap/block]|32 sec|Passed|
|test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
orc/def/block]|{color:#FF0000}3 min 25 sec{color}|Passed|
|test_fuzz_alltypes[... 'debug_action': None, \\| table_format:
parquet/none]|29 sec|Passed|
|test_fuzz_alltypes[... 'debug_action': None, \\| table_format: text/none]|20
sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|20 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}1 min 35 sec{color}|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|22 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|18
sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|20 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}1 min 16 sec{color}|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|17 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|16
sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
avro/snap/block]|19 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
orc/def/block]|{color:#FF0000}1 min 4 sec{color}|Passed|
|test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format:
parquet/none]|22 sec|Passed|
|test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format: text/none]|29
sec|Passed|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|4 sec|Skipped|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}1 min 35 sec{color}|Passed|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|18 sec|Passed|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|4.7
sec|Skipped|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|5.2 sec|Skipped|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}1 min 59 sec{color}|Passed|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|17 sec|Passed|
|test_fuzz_nested_types[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|4.3
sec|Skipped|
|test_fuzz_nested_types[... 'debug_action': None \\| table_format:
avro/snap/block]|4.1 sec|Skipped|
|test_fuzz_nested_types[... 'debug_action': None \\| table_format:
orc/def/block]|{color:#FF0000}2 min 30 sec{color}|Passed|
|test_fuzz_nested_types[... 'debug_action': None \\| table_format:
parquet/none]|18 sec|Passed|
|test_fuzz_nested_types[... 'debug_action': None \\| table_format:
text/none]|3.5 sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|4 sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}17 min{color}|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|20 sec|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|3.1
sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
avro/snap/block]|3.8 sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
orc/def/block]|{color:#FF0000}9 min 51 sec{color}|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format:
parquet/none]|19 sec|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action':
'-1:OPEN:[email protected]' \\| table_format: text/none]|4.4
sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format:
avro/snap/block]|3.9 sec|Skipped|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format:
orc/def/block]|{color:#FF0000}9 min 27 sec{color}|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format:
parquet/none]|19 sec|Passed|
|test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format:
text/none]|4.3 sec|Skipped|
Tests on other formats take less than 1min for each. ORC tests usually takes
several minutes.
CC [~boroknagyz]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
