This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 3c53ec749fc branch-4.1: [fix](docker)(case) Restore Kerberos/Paimon
external env and fix flaky file cache cases (#66314)
3c53ec749fc is described below
commit 3c53ec749fc02694c7f24d4ac5ef933af5d95df7
Author: Socrates <[email protected]>
AuthorDate: Fri Jul 31 16:12:36 2026 +0800
branch-4.1: [fix](docker)(case) Restore Kerberos/Paimon external env and
fix flaky file cache cases (#66314)
### What problem does this PR solve?
Issue Number: None
Related PR: #66254
Problem Summary:
Forward-port the external environment and regression-case fixes from
#66254 to branch-4.1. The branch inherited the same five defects:
1. The lightweight Kerberos environment creates principals with random
keys on every container start, so keytabs provisioned to Doris hosts no
longer match after the KDC is rebuilt.
2. A successful Hive baseline restore skips `init-hive-baseline.sh`,
while the incremental module list does not include the Paimon HMS table
definitions. The restored environment therefore misses the Paimon
databases.
3. `test_es_query` lives under `external_table_p2` but still switches to
its old P0 database name.
4. `test_file_cache_query_limit` sends BE brpc and HTTP requests to
`externalEnvIp`, which identifies the third-party Docker host rather
than the backend host in a multi-host deployment.
5. `test_file_cache_statistics` treats each queue's `max_size` as a hard
limit even though queues may borrow unused capacity from one another.
The actual hard bound is the total file-cache capacity.
This PR uses deterministic Kerberos keys, registers Paimon HMS as an
incremental Hive module, uses the suite-derived ES database, resolves
the single live backend before accessing BE endpoints, and validates
normal-queue usage against the sum of all queue capacities.
The patch was cherry-picked from commit
`954fde35c03257208b993421fb0d80d8b88769df` and applied cleanly to the
latest branch-4.1.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test
- `bash -n` for all changed shell scripts
-
`docker/thirdparties/test/run-thirdparties-docker-hive-bootstrap-groups-test.sh`
- Paimon HMS module selection and refresh harness
- Groovy CONVERSION-phase parse for all three changed regression suites
- Full external regression was not run because it requires a multi-host
Doris cluster and the third-party Docker stack
- Behavior changed:
- [x] No
- [ ] Yes
- Does this need documentation?
- [x] No
- [ ] Yes
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
docker/thirdparties/docker-compose/hive/README.md | 1 +
.../thirdparties/docker-compose/hive/README_ZH.md | 1 +
.../docker-compose/hive/scripts/hive-module-lib.sh | 26 +++++++++++++--
.../hive/scripts/init-hive-baseline.sh | 8 +++--
.../kerberos/entrypoint-hive-master.sh | 11 +++++--
.../cache/test_file_cache_query_limit.groovy | 21 ++++++++++--
.../cache/test_file_cache_statistics.groovy | 37 +++++++++++++++++-----
.../external_table_p2/es/test_es_query.groovy | 7 ++--
8 files changed, 92 insertions(+), 20 deletions(-)
diff --git a/docker/thirdparties/docker-compose/hive/README.md
b/docker/thirdparties/docker-compose/hive/README.md
index 443b1d6ada7..57c6d414b1d 100644
--- a/docker/thirdparties/docker-compose/hive/README.md
+++ b/docker/thirdparties/docker-compose/hive/README.md
@@ -60,6 +60,7 @@ Modules are refreshed incrementally: only modules whose
content SHA changed are
| `test` | `scripts/data/test/` | Lightweight smoke-test datasets |
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | ~77 HQL
files, executed in parallel via `xargs -P` |
| `view` | `scripts/create_view_scripts/create_view.hql` | View definitions |
+| `paimon_hms` |
`scripts/create_external_paimon_scripts/create_paimon_tables.hql` | Paimon
tables registered in HMS (`hdfs_db`, `ali_db`, …); only selected when
`enablePaimonHms=true` |
### Layer 3 — Version-Specific File Selection
diff --git a/docker/thirdparties/docker-compose/hive/README_ZH.md
b/docker/thirdparties/docker-compose/hive/README_ZH.md
index 2b1423e387b..8cc32162dc6 100644
--- a/docker/thirdparties/docker-compose/hive/README_ZH.md
+++ b/docker/thirdparties/docker-compose/hive/README_ZH.md
@@ -60,6 +60,7 @@ Hive 启动被拆分为三层互相独立的抽象：
| `test` | `scripts/data/test/` | 轻量级冒烟测试数据 |
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | 约 77 个
HQL 文件，通过 `xargs -P` 并行执行 |
| `view` | `scripts/create_view_scripts/create_view.hql` | View 定义 |
+| `paimon_hms` |
`scripts/create_external_paimon_scripts/create_paimon_tables.hql` | 注册到 HMS 的
Paimon 表（`hdfs_db`、`ali_db` 等），仅在 `enablePaimonHms=true` 时纳入 |
### Layer 3 — 按版本自动选文件
diff --git a/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
b/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
index 1cd00099c9b..beb26b1ff2e 100644
--- a/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
@@ -22,12 +22,19 @@ set -eo pipefail
. /mnt/scripts/hive-common-lib.sh
BOOTSTRAP_GROUPS="$(bootstrap_normalize_groups "${HIVE_BOOTSTRAP_GROUPS:-}")"
-DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf
regression test preinstalled_hql view)
+DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf
regression test preinstalled_hql view paimon_hms)
LAST_REFRESH_DETAIL=""
HIVE_HQL_PARALLEL="${HIVE_HQL_PARALLEL:-${LOAD_PARALLEL}}"
ensure_hive_state_layout
+# The Paimon HMS tables need the Paimon storage handler and the object storage
+# credentials that only the Hive3 stack is configured with, so the module opts
+# in via the same flag the settings env files carry.
+paimon_hms_enabled() {
+ [[ "${enablePaimonHms:-false}" == "true" ]]
+}
+
normalize_hive_modules() {
local raw_modules="${1:-}"
local cleaned_modules="${raw_modules// /}"
@@ -35,14 +42,19 @@ normalize_hive_modules() {
local normalized=()
if [[ -z "${cleaned_modules}" || "${cleaned_modules}" == "all" ]]; then
- printf '%s\n' "${DEFAULT_MODULES[@]}"
+ for module in "${DEFAULT_MODULES[@]}"; do
+ if [[ "${module}" == "paimon_hms" ]] && ! paimon_hms_enabled; then
+ continue
+ fi
+ echo "${module}"
+ done
return 0
fi
IFS=',' read -r -a normalized <<<"${cleaned_modules}"
for module in "${normalized[@]}"; do
case "${module}" in
-
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view)
+
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view|paimon_hms)
echo "${module}"
;;
*)
@@ -126,6 +138,9 @@ calc_module_sha() {
view)
files+=("/mnt/scripts/create_view_scripts/create_view.hql")
;;
+ paimon_hms)
+
files+=("/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql")
+ ;;
*)
echo "Unknown module for sha: ${module}" >&2
return 1
@@ -314,6 +329,11 @@ refresh_module() {
LAST_REFRESH_DETAIL="create_view.hql"
run_hive_hql /mnt/scripts/create_view_scripts/create_view.hql
"create_view.hql"
;;
+ paimon_hms)
+ LAST_REFRESH_DETAIL="create_paimon_tables.hql"
+ run_hive_hql
/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql \
+ "create_paimon_tables.hql"
+ ;;
*)
echo "Unknown module for refresh: ${module}" >&2
return 1
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
b/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
index 49424054fbd..02ce6c42f9c 100644
--- a/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
@@ -31,6 +31,10 @@ copy_to_hdfs_if_selected "paimon1"
copy_to_hdfs_if_selected "tvf_data"
copy_to_hdfs_if_selected "preinstalled_data"
-if [[ ${enablePaimonHms:-false} == "true" ]]; then
- run_hive_hql
/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql
"create_paimon_table.hql"
+# Go through the module framework rather than calling the HQL directly: this
+# script only runs on the full-init path, while a baseline restore reaches the
+# Paimon tables through refresh-hive-modules.sh. Recording the module state
+# here also keeps the follow-up refresh pass from running the same HQL twice.
+if paimon_hms_enabled; then
+ refresh_module paimon_hms
fi
diff --git
a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
index 5dd4adf46d4..6735ae7ee40 100644
--- a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
+++ b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -37,6 +37,7 @@ readonly HTTP_PRINCIPAL="HTTP/${HOST}@${REALM}"
readonly HIVE_PRINCIPAL="hive/${HOST}@${REALM}"
readonly HIVE_CLIENT_PRINCIPAL="hive/presto-master.docker.cluster@${REALM}"
readonly
PRESTO_CLIENT_PRINCIPAL="presto-server/presto-master.docker.cluster@${REALM}"
+readonly PRINCIPAL_PASSWORD="doris-kerberos-test"
declare -a SERVICE_PIDS=()
@@ -70,12 +71,18 @@ wait_for_port() {
return 1
}
+# Keys must stay identical across container rebuilds. Deployments that run
+# Doris on separate hosts from this container provision /keytabs out of band,
+# so a key that is re-randomized on every start makes every such client fail
+# the AS-REP decryption with "GeneralSecurityException: Checksum failed".
+# A fixed password plus the single fixed enctype in kdc.conf yields a stable
+# key, and -norandkey exports that key instead of rolling a new one.
create_keytab() {
local principal=$1
local keytab=$2
- kadmin.local -r "${REALM}" -q "addprinc -randkey ${principal}"
- kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} ${principal}"
+ kadmin.local -r "${REALM}" -q "addprinc -pw ${PRINCIPAL_PASSWORD}
${principal}"
+ kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} -norandkey ${principal}"
}
report_stage() {
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
index 540c07fab66..7e81f220447 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
@@ -52,9 +52,24 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
sql """set enable_file_cache=true"""
- // Check backend configuration prerequisites
// Note: This test case assumes a single backend scenario. Testing with
single backend is logically equivalent
// to testing with multiple backends having identical configurations, but
simpler in logic.
+ // The assumption is load-bearing rather than cosmetic: the HTTP calls
below clear and inspect ONE backend's
+ // file cache while the queries are served by the whole cluster, so with
several backends the inspected cache
+ // never reflects what the query actually cached. Skip instead of
reporting a false failure.
+ def aliveBackends = sql_return_maparray("show backends").findAll {
+ it.Alive.toString().equalsIgnoreCase("true")
+ }
+ if (aliveBackends.size() != 1) {
+ logger.info("skip test_file_cache_query_limit: it assumes a single
backend, found ${aliveBackends.size()}")
+ return
+ }
+ // The backend HTTP/brpc endpoints must be addressed by the backend's own
host. externalEnvIp is the
+ // third-party docker host (hive/es/...), which in a multi-host deployment
runs no backend at all, so
+ // curling it silently yields no file cache metrics.
+ String beHost = aliveBackends[0].Host
+
+ // Check backend configuration prerequisites
def enableFileCacheResult = sql """show backend config like
'enable_file_cache';"""
logger.info("enable_file_cache configuration: " + enableFileCacheResult)
assertFalse(enableFileCacheResult.size() == 0 ||
!enableFileCacheResult[0][3].equalsIgnoreCase("true"),
@@ -139,7 +154,7 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
String brpc_port = brpcPortResult[0][3]
// Search file cache capacity
- def command = ["curl", "-X", "POST", "${externalEnvIp}:${brpc_port}/vars"]
+ def command = ["curl", "-X", "POST", "${beHost}:${brpc_port}/vars"]
def stringCommand = command.collect{it.toString()}
def process = new ProcessBuilder(stringCommand as
String[]).redirectErrorStream(true).start()
@@ -160,7 +175,7 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
logger.info("========================= Start running file cache base test
========================")
// Clear file cache
- command = ["curl", "-X", "POST",
"${externalEnvIp}:${webserver_port}/api/file_cache?op=clear&sync=true"]
+ command = ["curl", "-X", "POST",
"${beHost}:${webserver_port}/api/file_cache?op=clear&sync=true"]
stringCommand = command.collect{it.toString()}
process = new ProcessBuilder(stringCommand as
String[]).redirectErrorStream(true).start()
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
index 6e16af8397c..34f4948c510 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
@@ -31,8 +31,8 @@ final String HIT_RATIO_5M_METRIC_FALSE_MSG =
HIT_RATIO_CHECK_FAILED_PREFIX + "hi
// Constants for normal queue check
final String NORMAL_QUEUE_CHECK_FAILED_PREFIX = "Normal queue check failed: "
-final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be
> 0 and < max_size)"
-final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements
should be > 0 and < max_elements)"
+final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be
> 0 and <= total cache capacity)"
+final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements
should be > 0)"
// Constants for hit and read counts check
final String HIT_AND_READ_COUNTS_CHECK_FAILED_PREFIX = "Hit and read counts
check failed: "
@@ -174,8 +174,20 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
// ===== Normal Queue Metrics Check =====
// curr_size / curr_elements are monitor-published; poll until populated
(> 0) across paths.
// max_size / max_elements come from the queue's static capacity (not
monitor-published), so
- // they are read once without polling. SUM across paths preserves the curr
< max inequality
- // (sum of per-path curr < sum of per-path max, since each curr < max).
+ // they are read once without polling.
+ //
+ // A queue's own max_size is a SOFT limit, not a bound to assert against:
when a queue is over
+ // its share, BlockFileCache::try_reserve_from_other_queue lets it keep
growing as long as the
+ // WHOLE cache still fits (`_cur_cache_size + size > _capacity &&
cur_queue_size + size >
+ // cur_queue_max_size` is the only rejection), evicting from the
under-used queues instead --
+ // see the "Hit the soft limit by self" branch and is_overflow(), which
compares against
+ // _capacity alone. So normal_queue_curr_size legitimately exceeds
normal_queue_max_size
+ // whenever the ttl/index/disposable queues are not full, which depends on
whichever cases ran
+ // before on this shared cache. Asserting curr < max encoded that
non-invariant and was flaky.
+ //
+ // The hard bound the BE actually enforces is the per-cache _capacity, and
by construction in
+ // get_file_cache_settings() capacity == normal + index + ttl + disposable
max sizes (the
+ // normal/query queue is defined as the remainder). Sum across paths and
assert against that.
pollMetric('normal_queue_curr_size', { it > 0 }, metricPollTimeoutSeconds)
pollMetric('normal_queue_curr_elements', { it > 0 },
metricPollTimeoutSeconds)
@@ -188,16 +200,25 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
def normalQueueMaxElementsSum = cacheMetricSum('normal_queue_max_elements')
logger.info("normal_queue_max_elements sum: " + normalQueueMaxElementsSum)
+ def indexQueueMaxSizeSum = cacheMetricSum('index_queue_max_size')
+ def ttlQueueMaxSizeSum = cacheMetricSum('ttl_queue_max_size')
+ def disposableQueueMaxSizeSum = cacheMetricSum('disposable_queue_max_size')
+ Double cacheCapacitySum = (normalQueueMaxSizeSum == null ||
indexQueueMaxSizeSum == null
+ || ttlQueueMaxSizeSum == null || disposableQueueMaxSizeSum ==
null) ? null
+ : normalQueueMaxSizeSum + indexQueueMaxSizeSum +
ttlQueueMaxSizeSum + disposableQueueMaxSizeSum
+ logger.info("total file cache capacity sum (normal+index+ttl+disposable
max_size): " + cacheCapacitySum)
+
boolean hasNormalQueueCurrSize = normalQueueCurrSizeSum != null &&
normalQueueCurrSizeSum > 0
boolean hasNormalQueueMaxSize = normalQueueMaxSizeSum != null &&
normalQueueMaxSizeSum > 0
boolean hasNormalQueueCurrElements = normalQueueCurrElementsSum != null &&
normalQueueCurrElementsSum > 0
boolean hasNormalQueueMaxElements = normalQueueMaxElementsSum != null &&
normalQueueMaxElementsSum > 0
- // Check if current size is less than max size and current elements is
less than max elements
+ // The queue must be in use and must stay within the cache's hard
capacity. max_elements is only
+ // logged: element counts are not bounded by the sum of the per-queue
element caps either, since
+ // a block may be smaller than max_file_block_size.
boolean normalQueueSizeValid = hasNormalQueueCurrSize &&
hasNormalQueueMaxSize &&
- normalQueueCurrSizeSum < normalQueueMaxSizeSum
- boolean normalQueueElementsValid = hasNormalQueueCurrElements &&
hasNormalQueueMaxElements &&
- normalQueueCurrElementsSum < normalQueueMaxElementsSum
+ cacheCapacitySum != null && normalQueueCurrSizeSum <= cacheCapacitySum
+ boolean normalQueueElementsValid = hasNormalQueueCurrElements &&
hasNormalQueueMaxElements
logger.info("Normal queue metrics check result - size valid:
${normalQueueSizeValid}, " +
"elements valid: ${normalQueueElementsValid}")
diff --git a/regression-test/suites/external_table_p2/es/test_es_query.groovy
b/regression-test/suites/external_table_p2/es/test_es_query.groovy
index bfe2ee7170b..c3d6b75ecca 100644
--- a/regression-test/suites/external_table_p2/es/test_es_query.groovy
+++ b/regression-test/suites/external_table_p2/es/test_es_query.groovy
@@ -193,9 +193,12 @@ suite("test_es_query", "p2,external") {
}
}
- def query_catalogs = { ->
+ def query_catalogs = { ->
sql """switch internal"""
- sql """use regression_test_external_table_p0_es"""
+ // test_v1/test_v2 live in this suite's own database, which the
+ // framework derives from the suite directory. Hardcoding the name
+ // breaks whenever the suite moves between p0 and p2.
+ sql """use ${context.dbName}"""
executeWithRetry("""select * from test_v1 where test2='text#1'""",
"sql01", 30)
order_qt_sql01 """select * from test_v1 where test2='text#1'"""
order_qt_sql02 """select * from test_v1 where esquery(test2,
'{"match":{"test2":"text#1"}}')"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
