morningman opened a new pull request, #66254:
URL: https://github.com/apache/doris/pull/66254
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65939
Problem Summary:
The daily external regression pipeline for `branch-4.0` fails 10 cases.
Eight of them flipped from green to red at one build and have stayed red for
every run since; the other two are long-standing. The cluster itself is healthy
— no crash, no OOM, no leak detected — so these are environment and case
defects, not a runtime regression.
Eight of the failures trace back to #65939, which replaced the external
Docker bootstrap. Three structural gaps came with it:
**1. Kerberos: `GeneralSecurityException: Checksum failed` (6 cases)**
`test_single_hive_kerberos`, `test_two_hive_kerberos`,
`test_non_catalog_kerberos`, `test_iceberg_hadoop_catalog_kerberos`,
`hive_on_hms_and_dlf`, `iceberg_on_hms_and_filesystem_and_dlf`.
The lightweight Kerberos environment creates every principal with `addprinc
-randkey`, so `/keytabs/*.keytab` holds a different key on every container
start. The image-based environment it replaced copied fixed keys baked into the
image (`cp /etc/trino/conf/* /keytabs/`). A deployment that runs Doris on hosts
other than the Docker host provisions `/keytabs` out of band and cannot follow
that rotation, so its keytab and the KDC no longer share a key. `Checksum
failed` is raised while decrypting the AS-REP, which means the KDC was reached
and the realm, ports and principals are all correct — only the key is wrong.
Fix: derive the keys from a fixed password and export them with `ktadd
-norandkey`. `kdc.conf` already pins a single enctype
(`aes128-cts-hmac-sha1-96`) and the salt is fixed, so the keytab is
reproducible across rebuilds.
**2. Paimon HMS databases are missing after a baseline restore (1 case)**
`test_paimon_hms_catalog` fails with `Unknown database 'hdfs_db'`.
`create_paimon_tables.hql` — which creates `hdfs_db`, `ali_db`, `hw_db`,
`tx_db`, `aws_db` and `gcs_db` — now lives only in `init-hive-baseline.sh`.
`maybe_refresh_hive_data()` runs that script only when `HIVE_MODE == rebuild`
or the baseline restore reported `missing`. With the default `refresh` mode and
a successful baseline restore, the script never runs and the incremental path
has no Paimon module, so the databases are silently absent on every run.
Previously the container's `hive-metastore.sh` executed the HQL unconditionally
on each start.
Fix: make it a `paimon_hms` module, so it is reached through
`refresh-hive-modules.sh` too. `init-hive-baseline.sh` now goes through the
same module, which records the state marker and keeps the follow-up refresh
pass from executing the same HQL twice.
**3. `test_es_query` hardcodes its old database name (1 case)**
The suite moved from `external_table_p0/es/` to `external_table_p2/es/` but
still runs `use regression_test_external_table_p0_es`, while the framework
derives the database from the suite directory. Fix: `use ${context.dbName}`. A
scan of the whole suite tree found no other cross-tier database reference.
The remaining two failures predate #65939:
**4. `test_file_cache_query_limit` (red on every run)**
It curls `externalEnvIp` for the backend brpc and HTTP endpoints.
`externalEnvIp` is the third-party Docker host (Hive, ES, …), which runs no
backend at all in a multi-host deployment, so `file_cache_capacity` is never
found. Fix: resolve the backend host from `show backends`, and skip when more
than one backend is alive — the case already documents single-backend as its
precondition, and its clear-one-backend-then-query-the-cluster flow cannot be
meaningful otherwise.
**5. `test_file_cache_statistics` (flaky)**
It asserts `normal_queue_curr_size < normal_queue_max_size`. A queue's
`max_size` is a soft limit: when a queue is over its share,
`BlockFileCache::try_reserve_from_other_queue` lets it keep growing as long as
the whole cache still fits, evicting from the under-used queues instead —
`is_overflow()` compares against `_capacity` alone. So `curr_size` legitimately
exceeds `max_size` whenever the ttl/index/disposable queues are not full, which
depends on which cases ran before on the shared cache. Fix: assert the bound
the backend actually enforces, the total cache capacity, which by construction
in `get_file_cache_settings()` equals the sum of the four queues' `max_size`.
Items 1 and 2 apply to `master` and `branch-4.1` as well and should be
forward ported.
> Note for the pipeline owners: item 1 makes the keys stable from now on,
but the pre-provisioned keytabs currently distributed to the FE/BE hosts were
generated from the old image and still will not match. They need to be
regenerated once against this environment (or copied from the Kerberos
container after it reports ready).
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
The changed cases can only be exercised by the external pipeline (they need
a multi-host cluster plus the full third-party Docker stack), so the following
was verified locally instead:
- `bash -n` on every changed shell script; the existing
`docker/thirdparties/test/run-thirdparties-docker-hive-bootstrap-groups-test.sh`
still passes.
- Groovy parse (CONVERSION phase) on every changed suite, with a mutation
check to confirm the parser actually rejects broken input.
- `normalize_hive_modules` / `module_needs_refresh` / `refresh_module`
driven through a harness: `paimon_hms` is excluded from the `all` expansion
unless `enablePaimonHms=true`, is honoured when named explicitly, runs when no
state marker exists (the baseline-restore case), is skipped on the immediately
following call (no double execution on a full init), and re-runs when the HQL
content changes.
- Keytab reproducibility measured in a container with `krb5-kdc`, rebuilding
the KDC repeatedly with the same realm, principal and enctype and comparing key
bytes: `addprinc -randkey` produced a different key on each round, `addprinc
-pw` + `ktadd -norandkey` produced an identical key on three consecutive rounds.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
The Hive bootstrap README tables were updated in place to list the new
`paimon_hms` module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
