Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19172 )
Change subject: IMPALA-7098: Re-enable tests under EC ...................................................................... IMPALA-7098: Re-enable tests under EC Re-enables tests under erasure coding, or provides more specific exceptions. Erasure coding uses multiple data blocks to construct a block group. Our tests use RS-3-2-1024k, which includes 3 data blocks in a block group. Each of these blocks is sized according to `dfs.block.size`, so block groups by default hold up to 384MB of data. Impala schedules work to executors based on blocks reported by HDFS, which for EC actually represent block groups. So with default block size, a file in EC has 1/3rd the number of schedulable blocks. In the case of tpch.lineitem, this produces 2 parquet files instead of 3 and reduces the number of executors scheduled to read parquet lineitem as 1. lineitem.tbl is loaded via Hive. With EC it uses 2 block groups, without EC it uses 6 blocks. 2. parquet lineitem is created by select/insert from lineitem.tbl. Impala schedules reads to executors based on available blocks, so with EC this gets scheduled across 2 executors instead of 3 and each executor writes a separate parquet file. Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Reviewed-on: http://gerrit.cloudera.org:8080/19172 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Joe McDonnell <[email protected]> --- M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_runtime_profile.py M tests/metadata/test_explain.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_mt_dop.py M tests/query_test/test_nested_types.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/util/auto_scaler.py 13 files changed, 31 insertions(+), 37 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19172 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib452024993e35d5a8d2854c6b2085115b26e40df Gerrit-Change-Number: 19172 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
