Mihaly Szjatinya has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
This continues the effort in IMPALA-6070 and consequent tickets.
Still mostly WIP to agree on basic concepts, some features not
finalized / for debugging purposes. Tested locally and on Jenkins,
mostly for individual test suites.
1. Container Infrastructure
- Cgroup v1/v2 Compatibility: Added smart detection for both cgroup
v1 and v2 systems
- Support for cgroup v2 unified hierarchy
(/sys/fs/cgroup/cgroup.controllers)
- Enhanced container startup logging and diagnostic information
- Fixed IMPALA_BUILD_THREADS detection for BE_TEST
- Docker attach readiness improvements with timing optimizations
- CPU workload integration hooks for performance testing
- Better error reporting and container lifecycle management
2. Improved Monitoring and Timeline UI
Timeline Improvements
- 6-graph comprehensive timeline: CPU/Memory/Disk usage for both
Container and Host
- Enhanced JSON data structure for rich metrics collection
- Timeline generation in case of job failure
- Collapsible test results per Test Suite
- Added min_memory_usage_gb and max_memory_usage_gb attributes to
ContainerMonitor
CPU Workload Testing System
- Build and test phase CPU load simulation for realistic metrics
- cpu_workloads.sh: Configurable CPU stress testing utilities for
performance benchmarking
3. Docker Registry Support
- Complete registry push/pull automation
- Human-readable tagging system (repo:tag format instead of
digest-only)
- Docker registry authentication with robot account support
- SHA256 digest handling for image validation and integrity
- Multi-format registry support (standard Docker + Harbor-style
registries)
4. Python 2.7 Compatibility
[Perhaps not needed in the future]
- Replace f-string syntax with .format() method calls for string
formatting
- Replace subprocess.DEVNULL with open(os.devnull, 'w') for
compatibility
- Remove Python 3+ only subprocess.TimeoutExpired exception handling
- Fix various import and syntax compatibility issues
- Ensure all Docker scripts work across Python 2.7+ environments
Dependencies and Integration
- Docker registry with authentication support
TODO:
- Test concurrent runs / tune for better performance
- Fix individual tests
- Auto-mode for most optimal test run
- Finalize test report in Timeline
- Incremental build support
Stats
For tests suites when run separately:
SUITE | Result
---------------------|--------------------
BE_TEST | ✅ PASS
| 44m
---------------------|--------------------
FE_TEST | ❌ 4 failures - jwtAuth related
| 1h 26m
---------------------|--------------------
EE_TEST_PARALLEL | ❌ 2 failures
| 1h
---------------------|--------------------
EE_TEST_SERIAL (2) | ✅ PASS
| 47m
---------------------|--------------------
CLUSTER_TEST (2) | ❌ 35 failures - ssl/ipv6 related
| 2h 27m
---------------------|--------------------
CLUSTER_TEST (1) | ✅ PASS
| 3h 9m
---------------------|--------------------
JDBC_TEST | ✅ PASS
build image - 1h5m
docker push - 36min (23GB)
docker pull - 5min
Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
---
A docker/cpu_workloads.sh
M docker/entrypoint.sh
M docker/monitor.py
M docker/test-with-docker.py
M docker/timeline.html.template
5 files changed, 1,339 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/23583/3
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Laszlo Gaal <[email protected]>
