Hello Aleksandr Efimov, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24822
to look at the new patch set (#3).
Change subject: IMPALA-14598: Store HBO cache in Redis/Valkey
......................................................................
IMPALA-14598: Store HBO cache in Redis/Valkey
This patch adds support for storing the HBO stats in Redis/Valkey, which
lets multiple impalad coordinators read and write the same history, and
lets the history survive coordinator restarts.
To support this a CacheBackend is implemented using JedisPool. HBO
values, i.e. List<TPlanNodeRun>, are serialized to Thrift binary as
THistoricalStatsValue.
The following flags are added:
- hbo_cache_backend: Select the HBO cache backend implementation.
Values: in_memory (the default per-coordinator Guava cache) or
redis (distributed cache backed by a Redis/Valkey server).
- hbo_cache_redis_host: Redis/Valkey server hostname or IP address.
Default: localhost
- hbo_cache_redis_port: Redis/Valkey server port. Default: 6379.
- hbo_cache_redis_password_cmd: A Unix command whose output returns the
password of the Redis/Valkey server.
Default: "" (no authentication)
- hbo_cache_redis_db: Redis/Valkey logical database index to use.
Default: 0. Redis supports 16 databases by default (0-15).
- hbo_cache_redis_timeout_ms: Timeout for Redis/Valkey operations in
milliseconds. Default: 2000.
- hbo_cache_redis_max_connections: Maximum number of connections in the
Redis/Valkey HBO cache client connection pool (JedisPool). The pool is
shared across all queries on a coordinator.
Default: 16
- hbo_cache_redis_ttl_seconds: Time-to-live in seconds for HBO cache
entries in Redis/Valkey. The TTL is refreshed on every write. 0
disables expiry.
Default: 4320000 (50 days).
Moved all hbo_* flags to backend-gflag-util.cc since they are only used
in Frontend.
To test this feature, instead of adding Redis/Valkey into the
native-toolchain, we use the official valkey/valkey docker image and
launch a Valkey container.
Testing
- Add FE tests that do not require a running Redis/Valkey server.
- Add e2e tests which run a Valkey container.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: I43a171bcd436f57bcff14ceaaaa98c0f7dcec769
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M bin/impala-config.sh
M common/thrift/BackendGflags.thrift
M common/thrift/HBO.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/main/java/org/apache/impala/service/CacheBackendFactory.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/main/java/org/apache/impala/service/HistoricalStatsValue.java
A fe/src/main/java/org/apache/impala/service/RedisCacheBackend.java
A fe/src/test/java/org/apache/impala/service/RedisCacheBackendTest.java
M java/pom.xml
A testdata/bin/kill-valkey.sh
A testdata/bin/run-valkey.sh
M tests/common/custom_cluster_test_suite.py
A tests/common/hbo_test_util.py
A tests/common/valkey_cluster.py
A tests/custom_cluster/test_hbo_redis.py
M tests/query_test/test_hbo.py
20 files changed, 1,317 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/24822/3
--
To view, visit http://gerrit.cloudera.org:8080/24822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I43a171bcd436f57bcff14ceaaaa98c0f7dcec769
Gerrit-Change-Number: 24822
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
