Hello David Rorke, Surya Hebbar, Joe McDonnell, Michael Smith, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23950
to look at the new patch set (#4).
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB)
host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660
v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-hardware-info-test.cc
A be/src/runtime/backend-hardware-info.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 503 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/4
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: David Rorke <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Surya Hebbar <[email protected]>
