Surya Hebbar has uploaded a new patch set (#10). (
http://gerrit.cloudera.org:8080/23154 )
Change subject: IMPALA-9846: Enable Aggregated Runtime Profile by Default
......................................................................
IMPALA-9846: Enable Aggregated Runtime Profile by Default
The traditional query profile generated bigger forests of child counters
and runtime profiles, from fragments and operators upto instance levels.
In the aggregated profile, the depth of the forests have been reduced
by transforming instance-level counters into operator level arrays
or maps.
With this change, aggregated runtime profile is enabled by having
'gen_aggregated_profile' set to 'true' by default. This serves as
replacement for 'gen_experimental_profile'.
The following tests are to be modified to support the usage of
aggregated runtime profile -
Backend -
No further changes are required for backend tests.
Frontend -
org.apache.impala.customcluster.ExternalFrontendTest
- testExternalFrontendBinary
- testExternalFrontendHttp
End-to-end Python tests -
(partial/all child tests of the following)
custom_cluster
- test_admission_controller.*
- test_ext_data_sources.*
- test_iceberg_strict_data.*
- test_mt_dop.*
- test_query_live.*
- test_query_log.*
- test_runtime_filter_aggregation.*
- test_tuple_cache.*
query_test
- test_acid.*
- test_aggregation.*
- test_analytic_tpcds.*
- test_codegen.*
- test_hash_join_timer.*
- test_iceberg.*
- test_insert.*
- test_join_queries.*
- test_kudu.*
- test_mem_usage_scaling.*
- test_mt_dop.*
- test_nested_types.*
- test_observability.*
- test_parquet_late_materialization.*
- test_partitioning.*
- test_queries.*
- test_result_spooling.*
- test_scanners.*
- test_runtime_filters.*
- test_tpch_queries.*
Also, in aggregated profile, instance-level time series counters
(i.e. MemoryUsage, ThreadUsage) are currently not being included,
as they are considered too large for longer queries.
Further sampling or aggregation(IMPALA-14256) is required to include
these counters.
Change-Id: If41d6322361fba82c946efd614cc7d28cb1c36e8
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/service/impala-server.cc
M be/src/service/query-state-record.cc
M be/src/util/runtime-profile-counters.h
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M testdata/impala-profiles/README
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.json
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.pretty.json
M testdata/impala-profiles/impala_profile_log_tpcds_compute_stats.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.pretty.json
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2.expected.json
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.pretty.json
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.txt
M testdata/workloads/functional-query/queries/QueryTest/binary-type.test
M
testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M testdata/workloads/functional-query/queries/QueryTest/exprs.test
M
testdata/workloads/functional-query/queries/QueryTest/full-acid-original-file.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-compute-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-scan-metrics-with-deletes.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/in_list_filters.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/insert_null.test
M
testdata/workloads/functional-query/queries/QueryTest/kudu_insert_mem_limit.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-array-materialization.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M
testdata/workloads/functional-query/queries/QueryTest/parquet-corrupt-rle-counts.test
M
testdata/workloads/functional-query/queries/QueryTest/processing-cost-admission-slots.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test
M
testdata/workloads/functional-query/queries/QueryTest/runtime_filters_mt_dop.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M
testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test
M testdata/workloads/tpcds-insert/queries/partitioned-insert.test
M testdata/workloads/tpch/queries/runtime-profile-aggregated.test
M testdata/workloads/tpch/queries/tpch-passthrough-aggregations.test
M testdata/workloads/tpch/queries/tpch-q1.test
M testdata/workloads/tpch/queries/tpch-q10.test
M testdata/workloads/tpch/queries/tpch-q11.test
M testdata/workloads/tpch/queries/tpch-q12.test
M testdata/workloads/tpch/queries/tpch-q13.test
M testdata/workloads/tpch/queries/tpch-q14.test
M testdata/workloads/tpch/queries/tpch-q15.test
M testdata/workloads/tpch/queries/tpch-q16.test
M testdata/workloads/tpch/queries/tpch-q17.test
M testdata/workloads/tpch/queries/tpch-q18.test
M testdata/workloads/tpch/queries/tpch-q19.test
M testdata/workloads/tpch/queries/tpch-q2.test
M testdata/workloads/tpch/queries/tpch-q20.test
M testdata/workloads/tpch/queries/tpch-q21.test
M testdata/workloads/tpch/queries/tpch-q22.test
M testdata/workloads/tpch/queries/tpch-q3.test
M testdata/workloads/tpch/queries/tpch-q4.test
M testdata/workloads/tpch/queries/tpch-q5.test
M testdata/workloads/tpch/queries/tpch-q6.test
M testdata/workloads/tpch/queries/tpch-q7.test
M testdata/workloads/tpch/queries/tpch-q8.test
M testdata/workloads/tpch/queries/tpch-q9.test
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_executor_groups.py
M tests/custom_cluster/test_mt_dop.py
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_query_log.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_runtime_profile.py
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_fetch.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_observability.py
M tests/query_test/test_parquet_bloom_filter.py
M tests/query_test/test_queries.py
M tests/query_test/test_result_spooling.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_sort.py
M tests/unittests/test_result_verifier.py
M tests/util/workload_management.py
87 files changed, 3,619 insertions(+), 3,287 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/23154/10
--
To view, visit http://gerrit.cloudera.org:8080/23154
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If41d6322361fba82c946efd614cc7d28cb1c36e8
Gerrit-Change-Number: 23154
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar <sheb...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
