Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24824
Change subject: IMPALA-14598: Add HDFS backend for HBO cache ...................................................................... IMPALA-14598: Add HDFS backend for HBO cache Add an optional per-coordinator HBO backend that keeps normal cache access in memory and periodically persists the cache to HDFS. Snapshot restore is asynchronous and fail-open, so unavailable HDFS does not block coordinator startup. Snapshots use a versioned format with bounded fields and per-entry checksums. They are written through a temporary file, synced, and atomically renamed. Restored entries do not overwrite newer values written after startup. A final bounded checkpoint runs during graceful shutdown. Striped locks also prevent lost updates between threads in one coordinator. Each coordinator must use a unique stable snapshot path. This backend does not provide a shared cache between coordinators. Testing: - Full impalad build. - HdfsCacheBackendTest and RedisCacheBackendTest (13 tests). - MiniHDFS graceful-restart and SIGKILL recovery scenarios. - Isolated restart validation against HDFS. Assisted-by: gpt-5.6-sol (OpenAI Codex) Change-Id: Ifda33dcfed88bd17359cef84f2069c72ef7b0dd9 --- M be/src/common/global-flags.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CacheBackend.java M fe/src/main/java/org/apache/impala/service/CacheBackendFactory.java A fe/src/main/java/org/apache/impala/service/HdfsCacheBackend.java M fe/src/main/java/org/apache/impala/service/HistoricalStats.java M fe/src/main/java/org/apache/impala/service/HistoricalStatsValue.java M fe/src/main/java/org/apache/impala/service/InMemoryCacheBackend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/RedisCacheBackend.java A fe/src/test/java/org/apache/impala/service/HdfsCacheBackendTest.java M fe/src/test/java/org/apache/impala/service/RedisCacheBackendTest.java A tests/custom_cluster/test_hbo_hdfs.py 18 files changed, 1,077 insertions(+), 74 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/24824/1 -- To view, visit http://gerrit.cloudera.org:8080/24824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifda33dcfed88bd17359cef84f2069c72ef7b0dd9 Gerrit-Change-Number: 24824 Gerrit-PatchSet: 1 Gerrit-Owner: Aleksandr Efimov <[email protected]>
