Surya Hebbar has uploaded a new patch set (#34). (
http://gerrit.cloudera.org:8080/23154 )
Change subject: IMPALA-9846: Enable AGGREGATED PROFILE by Default
......................................................................
IMPALA-9846: Enable AGGREGATED PROFILE by Default
The traditional query profile generated bigger forests of runtime profiles
and child counters, from fragments and operators up to instance levels.
This structure of the runtime profile can potentially stress the memory
allocator and use up a lot more memory and cache than is really necessary.
To mitigate these issues, the aggregated profiles were introduced,
which are substantially denser and faster to process for higher
'mt_dop' values.
In the aggregated profile, the depth of the forests has been reduced
by transforming instance-level counters into operator-level arrays
or maps.
The aggregation is also done in a single step, merging the aggregated
thrift profiles from the executor directly into the final aggregated
profile, without converting it to an unaggregated profile first.
This representation helps produce a nice, high-level, readable text
profile by default with the option to produce more detailed profiles
and alternate views when required.
With this change, the aggregated runtime profile is enabled by default
with the 'aggregated_profile' flag set to 'true'. This sets the default
value for the 'AGGREGATED_PROFILE' query option.
By enabling the aggregated profile by default, more than 2700+ tests
are impacted, including the backend and end-to-end tests.
We now utilize a combined single advanced regex search on the entire
runtime profile at once, instead of the line-by-line approach.
The impacted tests can largely be categorized into either counter value
aggregation or row regex searches. Several other tests that rely on
instance-level details are executed with the traditional profile by
disabling the aggregated profile through the query option introduced in
IMPALA-15174.
For nearly all of these tests, the major requirement was the sum of
counter values across fragment instances. But only mean, min, and max
values were present in the text representation.
In order to present this information to the tests, the 'total' statistic
has been added to the text representation of averaged counters along with
the existing min/max/avg.
- BytesRead: total=243.99 KB (246834) mean=81.31 KB (85258) min=...
Notes:
1. 'total' statistic is not printed for duration or time metrics.
2. 'total' statistic is not printed for counters that have already gone
through some form of aggregation or sampling, such as
'PeakMemoryUsage'.
Currently, this is applicable for all 'HighWaterMark' counters.
(i.e., LargestPartitionPercent, MaxPartitionLevel, PeakMemoryUsage,
PeakUnpinnedBytes, and PeakReservation).
The same logic can be extended to other types of counters as required.
This has been achieved by adding an associated 'TCounterCategory::type'
property to the Counter and TAggCounter classes.
TCounterCategory Enum class has the following possible values:
1. RAW: For all counters not sampled or averaged.
2. AGGREGATED: For counter values that have been sampled or averaged.
The test outputs for impala-profile-tool have been updated.
With these changes, all existing, updated, and newly added tests
successfully pass for both the aggregated and the traditional profile.
Note: Instance-level time series counters are currently not present in
the aggregated profile, as they are considered too large for longer
queries (i.e., MemoryUsage, ThreadUsage, etc.). To include these
counters, further sampling or aggregation is required (See IMPALA-14256).
Change-Id: If41d6322361fba82c946efd614cc7d28cb1c36e8
---
M be/src/util/runtime-profile-counters.h
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M common/thrift/RuntimeProfile.thrift
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt
M
testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_extended.expected.txt
M
testdata/impala-profiles/legacy-profiles/impala_profile_log_tpcds_compute_stats_default.expected.txt
M
testdata/impala-profiles/legacy-profiles/impala_profile_log_tpcds_compute_stats_extended.expected.txt
M
testdata/workloads/functional-query/queries/QueryTest/analytic-fns-tpcds-partitioned-topn.test
M testdata/workloads/functional-query/queries/QueryTest/calcite.test
M
testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-planner.test
M
testdata/workloads/functional-query/queries/QueryTest/hdfs_parquet_scan_node_profile.test
M
testdata/workloads/functional-query/queries/QueryTest/hdfs_scanner_profile.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-compute-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-create-table-like-non-iceberg.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-like-pushdown.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-delete.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-merge.test
M
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/in_list_filters.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/insert_null.test
M testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test
M
testdata/workloads/functional-query/queries/QueryTest/kudu_insert_mem_limit.test
M
testdata/workloads/functional-query/queries/QueryTest/kudu_runtime_filter_with_timestamp_conversion.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-array-materialization.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M
testdata/workloads/functional-query/queries/QueryTest/processing-cost-admission-slots.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test
M
testdata/workloads/functional-query/queries/QueryTest/runtime_filters_mt_dop.test
M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test
M testdata/workloads/functional-query/queries/QueryTest/spilling.test
M
testdata/workloads/functional-query/queries/QueryTest/union-const-scalar-expr-codegen.test
M testdata/workloads/tpcds-insert/queries/partitioned-insert.test
M testdata/workloads/tpch/queries/runtime-profile-aggregated.test
M testdata/workloads/tpch/queries/tpch-passthrough-aggregations.test
M
testdata/workloads/tpch_nested/queries/QueryTest/nested-types-subplan-single-node.test
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_tuple_cache.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_fetch.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_kudu.py
M tests/query_test/test_observability.py
M tests/query_test/test_parquet_bloom_filter.py
M tests/query_test/test_queries.py
M tests/query_test/test_result_spooling.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_sort.py
M tests/unittests/test_result_verifier.py
M tests/util/workload_management.py
62 files changed, 3,500 insertions(+), 3,932 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/23154/34
--
To view, visit http://gerrit.cloudera.org:8080/23154
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If41d6322361fba82c946efd614cc7d28cb1c36e8
Gerrit-Change-Number: 23154
Gerrit-PatchSet: 34
Gerrit-Owner: Surya Hebbar <[email protected]>
Gerrit-Reviewer: Abhishek Rawat <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Surya Hebbar <[email protected]>
