This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 68261b85839 branch-4.0: [fix](docker)(case) Restore Kerberos/Paimon
external env and fix flaky file cache cases (#66254)
68261b85839 is described below
commit 68261b8583982d09a8129556acdfdb457e132dba
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Aug 1 14:46:18 2026 +0800
branch-4.0: [fix](docker)(case) Restore Kerberos/Paimon external env and
fix flaky file cache cases (#66254)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65939
Problem Summary:
The daily external regression pipeline for `branch-4.0` fails 10 cases.
Eight of them flipped from green to red at one build and have stayed red
for every run since; the other two are long-standing. The cluster itself
is healthy — no crash, no OOM, no leak detected — so these are
environment and case defects, not a runtime regression.
Eight of the failures trace back to #65939, which replaced the external
Docker bootstrap. Three structural gaps came with it:
**1. Kerberos: `GeneralSecurityException: Checksum failed` (6 cases)**
`test_single_hive_kerberos`, `test_two_hive_kerberos`,
`test_non_catalog_kerberos`, `test_iceberg_hadoop_catalog_kerberos`,
`hive_on_hms_and_dlf`, `iceberg_on_hms_and_filesystem_and_dlf`.
The lightweight Kerberos environment creates every principal with
`addprinc -randkey`, so `/keytabs/*.keytab` holds a different key on
every container start. The image-based environment it replaced copied
fixed keys baked into the image (`cp /etc/trino/conf/* /keytabs/`). A
deployment that runs Doris on hosts other than the Docker host
provisions `/keytabs` out of band and cannot follow that rotation, so
its keytab and the KDC no longer share a key. `Checksum failed` is
raised while decrypting the AS-REP, which means the KDC was reached and
the realm, ports and principals are all correct — only the key is wrong.
Fix: derive the keys from a fixed password and export them with `ktadd
-norandkey`. `kdc.conf` already pins a single enctype
(`aes128-cts-hmac-sha1-96`) and the salt is fixed, so the keytab is
reproducible across rebuilds.
**2. Paimon HMS databases are missing after a baseline restore (1
case)**
`test_paimon_hms_catalog` fails with `Unknown database 'hdfs_db'`.
`create_paimon_tables.hql` — which creates `hdfs_db`, `ali_db`, `hw_db`,
`tx_db`, `aws_db` and `gcs_db` — now lives only in
`init-hive-baseline.sh`. `maybe_refresh_hive_data()` runs that script
only when `HIVE_MODE == rebuild` or the baseline restore reported
`missing`. With the default `refresh` mode and a successful baseline
restore, the script never runs and the incremental path has no Paimon
module, so the databases are silently absent on every run. Previously
the container's `hive-metastore.sh` executed the HQL unconditionally on
each start.
Fix: make it a `paimon_hms` module, so it is reached through
`refresh-hive-modules.sh` too. `init-hive-baseline.sh` now goes through
the same module, which records the state marker and keeps the follow-up
refresh pass from executing the same HQL twice.
**3. `test_es_query` hardcodes its old database name (1 case)**
The suite moved from `external_table_p0/es/` to `external_table_p2/es/`
but still runs `use regression_test_external_table_p0_es`, while the
framework derives the database from the suite directory. Fix: `use
${context.dbName}`. A scan of the whole suite tree found no other
cross-tier database reference.
The remaining two failures predate #65939:
**4. `test_file_cache_query_limit` (red on every run)**
It curls `externalEnvIp` for the backend brpc and HTTP endpoints.
`externalEnvIp` is the third-party Docker host (Hive, ES, …), which runs
no backend at all in a multi-host deployment, so `file_cache_capacity`
is never found. Fix: resolve the backend host from `show backends`, and
skip when more than one backend is alive — the case already documents
single-backend as its precondition, and its
clear-one-backend-then-query-the-cluster flow cannot be meaningful
otherwise.
**5. `test_file_cache_statistics` (flaky)**
It asserts `normal_queue_curr_size < normal_queue_max_size`. A queue's
`max_size` is a soft limit: when a queue is over its share,
`BlockFileCache::try_reserve_from_other_queue` lets it keep growing as
long as the whole cache still fits, evicting from the under-used queues
instead — `is_overflow()` compares against `_capacity` alone. So
`curr_size` legitimately exceeds `max_size` whenever the
ttl/index/disposable queues are not full, which depends on which cases
ran before on the shared cache. Fix: assert the bound the backend
actually enforces, the total cache capacity, which by construction in
`get_file_cache_settings()` equals the sum of the four queues'
`max_size`.
Items 1 and 2 apply to `master` and `branch-4.1` as well and should be
forward ported.
> Note for the pipeline owners: item 1 makes the keys stable from now
on, but the pre-provisioned keytabs currently distributed to the FE/BE
hosts were generated from the old image and still will not match. They
need to be regenerated once against this environment (or copied from the
Kerberos container after it reports ready).
---
docker/thirdparties/docker-compose/hive/README.md | 1 +
.../thirdparties/docker-compose/hive/README_ZH.md | 1 +
.../docker-compose/hive/scripts/hive-module-lib.sh | 26 +++++++++++++--
.../hive/scripts/init-hive-baseline.sh | 8 +++--
.../kerberos/entrypoint-hive-master.sh | 11 +++++--
.../cache/test_file_cache_query_limit.groovy | 21 ++++++++++--
.../cache/test_file_cache_statistics.groovy | 37 +++++++++++++++++-----
.../external_table_p2/es/test_es_query.groovy | 7 ++--
8 files changed, 92 insertions(+), 20 deletions(-)
diff --git a/docker/thirdparties/docker-compose/hive/README.md
b/docker/thirdparties/docker-compose/hive/README.md
index 443b1d6ada7..57c6d414b1d 100644
--- a/docker/thirdparties/docker-compose/hive/README.md
+++ b/docker/thirdparties/docker-compose/hive/README.md
@@ -60,6 +60,7 @@ Modules are refreshed incrementally: only modules whose
content SHA changed are
| `test` | `scripts/data/test/` | Lightweight smoke-test datasets |
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | ~77 HQL
files, executed in parallel via `xargs -P` |
| `view` | `scripts/create_view_scripts/create_view.hql` | View definitions |
+| `paimon_hms` |
`scripts/create_external_paimon_scripts/create_paimon_tables.hql` | Paimon
tables registered in HMS (`hdfs_db`, `ali_db`, …); only selected when
`enablePaimonHms=true` |
### Layer 3 — Version-Specific File Selection
diff --git a/docker/thirdparties/docker-compose/hive/README_ZH.md
b/docker/thirdparties/docker-compose/hive/README_ZH.md
index 2b1423e387b..8cc32162dc6 100644
--- a/docker/thirdparties/docker-compose/hive/README_ZH.md
+++ b/docker/thirdparties/docker-compose/hive/README_ZH.md
@@ -60,6 +60,7 @@ Hive 启动被拆分为三层互相独立的抽象：
| `test` | `scripts/data/test/` | 轻量级冒烟测试数据 |
| `preinstalled_hql` | `scripts/create_preinstalled_scripts/*.hql` | 约 77 个
HQL 文件，通过 `xargs -P` 并行执行 |
| `view` | `scripts/create_view_scripts/create_view.hql` | View 定义 |
+| `paimon_hms` |
`scripts/create_external_paimon_scripts/create_paimon_tables.hql` | 注册到 HMS 的
Paimon 表（`hdfs_db`、`ali_db` 等），仅在 `enablePaimonHms=true` 时纳入 |
### Layer 3 — 按版本自动选文件
diff --git a/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
b/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
index 1cd00099c9b..beb26b1ff2e 100644
--- a/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/hive-module-lib.sh
@@ -22,12 +22,19 @@ set -eo pipefail
. /mnt/scripts/hive-common-lib.sh
BOOTSTRAP_GROUPS="$(bootstrap_normalize_groups "${HIVE_BOOTSTRAP_GROUPS:-}")"
-DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf
regression test preinstalled_hql view)
+DEFAULT_MODULES=(default multi_catalog partition_type statistics tvf
regression test preinstalled_hql view paimon_hms)
LAST_REFRESH_DETAIL=""
HIVE_HQL_PARALLEL="${HIVE_HQL_PARALLEL:-${LOAD_PARALLEL}}"
ensure_hive_state_layout
+# The Paimon HMS tables need the Paimon storage handler and the object storage
+# credentials that only the Hive3 stack is configured with, so the module opts
+# in via the same flag the settings env files carry.
+paimon_hms_enabled() {
+ [[ "${enablePaimonHms:-false}" == "true" ]]
+}
+
normalize_hive_modules() {
local raw_modules="${1:-}"
local cleaned_modules="${raw_modules// /}"
@@ -35,14 +42,19 @@ normalize_hive_modules() {
local normalized=()
if [[ -z "${cleaned_modules}" || "${cleaned_modules}" == "all" ]]; then
- printf '%s\n' "${DEFAULT_MODULES[@]}"
+ for module in "${DEFAULT_MODULES[@]}"; do
+ if [[ "${module}" == "paimon_hms" ]] && ! paimon_hms_enabled; then
+ continue
+ fi
+ echo "${module}"
+ done
return 0
fi
IFS=',' read -r -a normalized <<<"${cleaned_modules}"
for module in "${normalized[@]}"; do
case "${module}" in
-
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view)
+
default|multi_catalog|partition_type|statistics|tvf|regression|test|preinstalled_hql|view|paimon_hms)
echo "${module}"
;;
*)
@@ -126,6 +138,9 @@ calc_module_sha() {
view)
files+=("/mnt/scripts/create_view_scripts/create_view.hql")
;;
+ paimon_hms)
+
files+=("/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql")
+ ;;
*)
echo "Unknown module for sha: ${module}" >&2
return 1
@@ -314,6 +329,11 @@ refresh_module() {
LAST_REFRESH_DETAIL="create_view.hql"
run_hive_hql /mnt/scripts/create_view_scripts/create_view.hql
"create_view.hql"
;;
+ paimon_hms)
+ LAST_REFRESH_DETAIL="create_paimon_tables.hql"
+ run_hive_hql
/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql \
+ "create_paimon_tables.hql"
+ ;;
*)
echo "Unknown module for refresh: ${module}" >&2
return 1
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
b/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
index 49424054fbd..02ce6c42f9c 100644
--- a/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/init-hive-baseline.sh
@@ -31,6 +31,10 @@ copy_to_hdfs_if_selected "paimon1"
copy_to_hdfs_if_selected "tvf_data"
copy_to_hdfs_if_selected "preinstalled_data"
-if [[ ${enablePaimonHms:-false} == "true" ]]; then
- run_hive_hql
/mnt/scripts/create_external_paimon_scripts/create_paimon_tables.hql
"create_paimon_table.hql"
+# Go through the module framework rather than calling the HQL directly: this
+# script only runs on the full-init path, while a baseline restore reaches the
+# Paimon tables through refresh-hive-modules.sh. Recording the module state
+# here also keeps the follow-up refresh pass from running the same HQL twice.
+if paimon_hms_enabled; then
+ refresh_module paimon_hms
fi
diff --git
a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
index 5dd4adf46d4..6735ae7ee40 100644
--- a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
+++ b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -37,6 +37,7 @@ readonly HTTP_PRINCIPAL="HTTP/${HOST}@${REALM}"
readonly HIVE_PRINCIPAL="hive/${HOST}@${REALM}"
readonly HIVE_CLIENT_PRINCIPAL="hive/presto-master.docker.cluster@${REALM}"
readonly
PRESTO_CLIENT_PRINCIPAL="presto-server/presto-master.docker.cluster@${REALM}"
+readonly PRINCIPAL_PASSWORD="doris-kerberos-test"
declare -a SERVICE_PIDS=()
@@ -70,12 +71,18 @@ wait_for_port() {
return 1
}
+# Keys must stay identical across container rebuilds. Deployments that run
+# Doris on separate hosts from this container provision /keytabs out of band,
+# so a key that is re-randomized on every start makes every such client fail
+# the AS-REP decryption with "GeneralSecurityException: Checksum failed".
+# A fixed password plus the single fixed enctype in kdc.conf yields a stable
+# key, and -norandkey exports that key instead of rolling a new one.
create_keytab() {
local principal=$1
local keytab=$2
- kadmin.local -r "${REALM}" -q "addprinc -randkey ${principal}"
- kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} ${principal}"
+ kadmin.local -r "${REALM}" -q "addprinc -pw ${PRINCIPAL_PASSWORD}
${principal}"
+ kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} -norandkey ${principal}"
}
report_stage() {
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
index cdeb9e9a77c..bb0c16f44d6 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy
@@ -52,9 +52,24 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
sql """set enable_file_cache=true"""
- // Check backend configuration prerequisites
// Note: This test case assumes a single backend scenario. Testing with
single backend is logically equivalent
// to testing with multiple backends having identical configurations, but
simpler in logic.
+ // The assumption is load-bearing rather than cosmetic: the HTTP calls
below clear and inspect ONE backend's
+ // file cache while the queries are served by the whole cluster, so with
several backends the inspected cache
+ // never reflects what the query actually cached. Skip instead of
reporting a false failure.
+ def aliveBackends = sql_return_maparray("show backends").findAll {
+ it.Alive.toString().equalsIgnoreCase("true")
+ }
+ if (aliveBackends.size() != 1) {
+ logger.info("skip test_file_cache_query_limit: it assumes a single
backend, found ${aliveBackends.size()}")
+ return
+ }
+ // The backend HTTP/brpc endpoints must be addressed by the backend's own
host. externalEnvIp is the
+ // third-party docker host (hive/es/...), which in a multi-host deployment
runs no backend at all, so
+ // curling it silently yields no file cache metrics.
+ String beHost = aliveBackends[0].Host
+
+ // Check backend configuration prerequisites
def enableFileCacheResult = sql """show backend config like
'enable_file_cache';"""
logger.info("enable_file_cache configuration: " + enableFileCacheResult)
assertFalse(enableFileCacheResult.size() == 0 ||
!enableFileCacheResult[0][3].equalsIgnoreCase("true"),
@@ -139,7 +154,7 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
String brpc_port = brpcPortResult[0][3]
// Search file cache capacity
- def command = ["curl", "-X", "POST", "${externalEnvIp}:${brpc_port}/vars"]
+ def command = ["curl", "-X", "POST", "${beHost}:${brpc_port}/vars"]
def stringCommand = command.collect{it.toString()}
def process = new ProcessBuilder(stringCommand as
String[]).redirectErrorStream(true).start()
@@ -160,7 +175,7 @@ suite("test_file_cache_query_limit",
"external_docker,hive,external_docker_hive,
logger.info("========================= Start running file cache base test
========================")
// Clear file cache
- command = ["curl", "-X", "POST",
"${externalEnvIp}:${webserver_port}/api/file_cache?op=clear&sync=true"]
+ command = ["curl", "-X", "POST",
"${beHost}:${webserver_port}/api/file_cache?op=clear&sync=true"]
stringCommand = command.collect{it.toString()}
process = new ProcessBuilder(stringCommand as
String[]).redirectErrorStream(true).start()
diff --git
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
index 6e16af8397c..34f4948c510 100644
---
a/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
+++
b/regression-test/suites/external_table_p0/cache/test_file_cache_statistics.groovy
@@ -31,8 +31,8 @@ final String HIT_RATIO_5M_METRIC_FALSE_MSG =
HIT_RATIO_CHECK_FAILED_PREFIX + "hi
// Constants for normal queue check
final String NORMAL_QUEUE_CHECK_FAILED_PREFIX = "Normal queue check failed: "
-final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be
> 0 and < max_size)"
-final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements
should be > 0 and < max_elements)"
+final String NORMAL_QUEUE_SIZE_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "size validation failed (curr_size should be
> 0 and <= total cache capacity)"
+final String NORMAL_QUEUE_ELEMENTS_VALIDATION_FAILED_MSG =
NORMAL_QUEUE_CHECK_FAILED_PREFIX + "elements validation failed (curr_elements
should be > 0)"
// Constants for hit and read counts check
final String HIT_AND_READ_COUNTS_CHECK_FAILED_PREFIX = "Hit and read counts
check failed: "
@@ -174,8 +174,20 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
// ===== Normal Queue Metrics Check =====
// curr_size / curr_elements are monitor-published; poll until populated
(> 0) across paths.
// max_size / max_elements come from the queue's static capacity (not
monitor-published), so
- // they are read once without polling. SUM across paths preserves the curr
< max inequality
- // (sum of per-path curr < sum of per-path max, since each curr < max).
+ // they are read once without polling.
+ //
+ // A queue's own max_size is a SOFT limit, not a bound to assert against:
when a queue is over
+ // its share, BlockFileCache::try_reserve_from_other_queue lets it keep
growing as long as the
+ // WHOLE cache still fits (`_cur_cache_size + size > _capacity &&
cur_queue_size + size >
+ // cur_queue_max_size` is the only rejection), evicting from the
under-used queues instead --
+ // see the "Hit the soft limit by self" branch and is_overflow(), which
compares against
+ // _capacity alone. So normal_queue_curr_size legitimately exceeds
normal_queue_max_size
+ // whenever the ttl/index/disposable queues are not full, which depends on
whichever cases ran
+ // before on this shared cache. Asserting curr < max encoded that
non-invariant and was flaky.
+ //
+ // The hard bound the BE actually enforces is the per-cache _capacity, and
by construction in
+ // get_file_cache_settings() capacity == normal + index + ttl + disposable
max sizes (the
+ // normal/query queue is defined as the remainder). Sum across paths and
assert against that.
pollMetric('normal_queue_curr_size', { it > 0 }, metricPollTimeoutSeconds)
pollMetric('normal_queue_curr_elements', { it > 0 },
metricPollTimeoutSeconds)
@@ -188,16 +200,25 @@ suite("test_file_cache_statistics",
"external_docker,hive,external_docker_hive,p
def normalQueueMaxElementsSum = cacheMetricSum('normal_queue_max_elements')
logger.info("normal_queue_max_elements sum: " + normalQueueMaxElementsSum)
+ def indexQueueMaxSizeSum = cacheMetricSum('index_queue_max_size')
+ def ttlQueueMaxSizeSum = cacheMetricSum('ttl_queue_max_size')
+ def disposableQueueMaxSizeSum = cacheMetricSum('disposable_queue_max_size')
+ Double cacheCapacitySum = (normalQueueMaxSizeSum == null ||
indexQueueMaxSizeSum == null
+ || ttlQueueMaxSizeSum == null || disposableQueueMaxSizeSum ==
null) ? null
+ : normalQueueMaxSizeSum + indexQueueMaxSizeSum +
ttlQueueMaxSizeSum + disposableQueueMaxSizeSum
+ logger.info("total file cache capacity sum (normal+index+ttl+disposable
max_size): " + cacheCapacitySum)
+
boolean hasNormalQueueCurrSize = normalQueueCurrSizeSum != null &&
normalQueueCurrSizeSum > 0
boolean hasNormalQueueMaxSize = normalQueueMaxSizeSum != null &&
normalQueueMaxSizeSum > 0
boolean hasNormalQueueCurrElements = normalQueueCurrElementsSum != null &&
normalQueueCurrElementsSum > 0
boolean hasNormalQueueMaxElements = normalQueueMaxElementsSum != null &&
normalQueueMaxElementsSum > 0
- // Check if current size is less than max size and current elements is
less than max elements
+ // The queue must be in use and must stay within the cache's hard
capacity. max_elements is only
+ // logged: element counts are not bounded by the sum of the per-queue
element caps either, since
+ // a block may be smaller than max_file_block_size.
boolean normalQueueSizeValid = hasNormalQueueCurrSize &&
hasNormalQueueMaxSize &&
- normalQueueCurrSizeSum < normalQueueMaxSizeSum
- boolean normalQueueElementsValid = hasNormalQueueCurrElements &&
hasNormalQueueMaxElements &&
- normalQueueCurrElementsSum < normalQueueMaxElementsSum
+ cacheCapacitySum != null && normalQueueCurrSizeSum <= cacheCapacitySum
+ boolean normalQueueElementsValid = hasNormalQueueCurrElements &&
hasNormalQueueMaxElements
logger.info("Normal queue metrics check result - size valid:
${normalQueueSizeValid}, " +
"elements valid: ${normalQueueElementsValid}")
diff --git a/regression-test/suites/external_table_p2/es/test_es_query.groovy
b/regression-test/suites/external_table_p2/es/test_es_query.groovy
index bfe2ee7170b..c3d6b75ecca 100644
--- a/regression-test/suites/external_table_p2/es/test_es_query.groovy
+++ b/regression-test/suites/external_table_p2/es/test_es_query.groovy
@@ -193,9 +193,12 @@ suite("test_es_query", "p2,external") {
}
}
- def query_catalogs = { ->
+ def query_catalogs = { ->
sql """switch internal"""
- sql """use regression_test_external_table_p0_es"""
+ // test_v1/test_v2 live in this suite's own database, which the
+ // framework derives from the suite directory. Hardcoding the name
+ // breaks whenever the suite moves between p0 and p2.
+ sql """use ${context.dbName}"""
executeWithRetry("""select * from test_v1 where test2='text#1'""",
"sql01", 30)
order_qt_sql01 """select * from test_v1 where test2='text#1'"""
order_qt_sql02 """select * from test_v1 where esquery(test2,
'{"match":{"test2":"text#1"}}')"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
