morningman opened a new pull request, #66433:
URL: https://github.com/apache/doris/pull/66433
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65564, #66313, #66417
Problem Summary:
Master port of the remaining external-regression gaps diagnosed on the
branch-4.0 pipeline (#66417). #66313 already carried the keytab and baseline
fixes to master; this PR carries the rest. All three defects were verified to
exist on master by inspection — master shares the same code and the same
lightweight Kerberos environment (#65564, which #65939 backported to 4.0).
**1. Kerberos HDFS writes fail in the p2 suites (SASL client property
missing)**
#65564 created `kerberos/conf/hdfs-site.xml.tpl` with
`dfs.data.transfer.protection=authentication`. A client that does not set the
same property sends a plain `writeBlock` op, `SaslDataTransferServer` closes
the
socket, the client excludes the only DataNode, and the write fails with
could only be written to 0 of the 1 minReplication nodes.
There are 1 datanode(s) running and 1 node(s) are excluded in this
operation.
(through iceberg this surfaces as `IllegalArgumentException: Self-suppression
not permitted`). Metadata operations only reach the NameNode and the HMS, so
`CREATE DATABASE` / `CREATE TABLE` still succeed and only the write fails.
#65564 set the client property in the four `external_table_p0/kerberos`
suites
but not in the p2 suites that talk to the same kerberized HDFS. On the
branch-4.0 pipeline this turned `hive_on_hms_and_dlf` and
`iceberg_on_hms_and_filesystem_and_dlf` red on every run; master carries the
identical omission.
Fix: set `dfs.data.transfer.protection` in every property block that points
at
the kerberized HDFS. A scan of `:8520`/`:8620` across the whole suite tree
finds
seven such blocks still missing it: two each in `hive_on_hms_and_dlf`,
`iceberg_on_hms_and_filesystem_and_dlf` and `test_paimon_hms_catalog`, plus
one
in `hdfs_all_test` (currently returns early because
`refactor_params_hdfs_kerberos_test` is unset — it would fail the moment it
is
enabled). The kerberos block of `test_information_schema_timezone` has the
same
gap but is disabled by an unrelated TODO, so it is left alone.
**2. `test_paimon_hms_catalog`: `Unknown database 'hdfs_db'`**
The lightweight Kerberos environment starts an empty metastore — `schematool
-initSchema` and nothing else. The environment it replaced uploaded
`paimon_data` into HDFS and registered `hdfs_db` through
`create_paimon_hive_table.hql` on every start; `paimon_data/hdfs_db.db/` is
still in the tree but the compose file never mounted it. This restores the
fixture on `kerberos1` only (it owns metastore 9583, the only one the suite
talks to), runs it before `DORIS_KERBEROS_READY` so nothing races the suites,
and aborts the container on failure instead of handing out an environment
silently missing `hdfs_db`.
Two deliberate differences from the branch-4.0 sibling commit:
- **`hdfs_db` only, no `ali_db`/OSS.** Master's pre-#65564 HQL provisioned
exactly `hdfs_db`; the `oss_hms_kerberos` case that reads `ali_db` through
the kerberized metastore exists only on branch-4.0, master's `ali_db` cases
go through the Hive3 metastore, and the golden output here carries only the
`hdfs_kerberos` / `hdfs_new_kerberos` blocks. This keeps the jindo/aliyun
jars and OSS credentials out of the kerberos stack.
- **The fixture is gated on the container role (`HOST`).** Both kerberos
containers run the same entrypoint with the same rendered env switch, but
only `kerberos1` carries the `sql/`/`paimon_data/`/`auxlib/` mounts —
without
the gate, `set -e` would abort `kerberos2` the moment `enablePaimonHms` is
on. (The branch-4.0 sibling needs the same gate; a follow-up on #66417 will
add it.)
**3. `IcebergConnectorMetadata`: report the root cause of failed metadata
commits**
`TableMetadataParser.internalWrite` writes the metadata JSON inside
try-with-resources, and `DFSOutputStream` throws the same exception instance
from `write` and from `close` once its pipeline has failed, so
`Throwable.addSuppressed(this)` replaces the failure with
`IllegalArgumentException: Self-suppression not permitted`. The FE log keeps
the
chain, but the message returned over the MySQL protocol is built from
`e.getMessage()` and carries only the wrapper.
This switches all nineteen metadata-commit paths — `createTable`, the
column/schema updates (including the nested-column and comment variants), the
branch/tag operations, and the partition-field operations — to
`ExceptionUtils.getRootCauseMessage(e)`. The legacy `IcebergMetadataOps`
already reported the root cause for the partition-field trio; the connector
rewrite lost that, so this also restores it. Read paths and the drop/rename
operations cannot hit self-suppression and keep `e.getMessage()`.
**Intentionally not ported from #66417:** the file-cache de-flake. Master's
`test_file_cache_features` (rewritten in #66313's lineage) already waits for
`disk_resource_limit_mode` / `need_evict_cache_in_advance` to clear before
returning, which closes the residual-state window at its source.
**Verification (local):** `fe-connector-iceberg` compiles and passes
checkstyle
(`mvn -pl fe-connector/fe-connector-iceberg package`); the four changed
suites
compile with the offline groovy compiler; `bash -n` passes on both shell
scripts; the compose/env/conf templates render with no leftover placeholders,
the rendered XML parses, kerberos1 gets exactly the three new mounts while
kerberos2 is unchanged; the settings resolution and the entrypoint role-gate
were exercised for all switch/role combinations. The containers themselves
need
the CI environment and have not been run.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
