morningman opened a new pull request, #66417:
URL: https://github.com/apache/doris/pull/66417
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65939, #66254
Problem Summary:
Follow-up to #66254. The daily `Regression External` pipeline for
`branch-4.0`
(build 482/483) still fails 4 cases. None of them is a runtime problem: the
cluster is healthy, 522 cases pass, and there is no crash, OOM or core dump.
Two root causes remain, both of them the tail of #65939.
**1. Kerberos HDFS writes fail (3 p2 cases, red on every run since #65939)**
`hive_on_hms_and_dlf` fails with
could only be written to 0 of the 1 minReplication nodes.
There are 1 datanode(s) running and 1 node(s) are excluded in this
operation.
and `iceberg_on_hms_and_filesystem_and_dlf` fails with
`IllegalArgumentException: Self-suppression not permitted`, which is what
`TableMetadataParser.internalWrite` turns the same failure into.
The DataNode is alive — the client excluded it. `jni.log` shows why:
`java.net.SocketException: Connection reset` in
`DataStreamer.createBlockOutputStream`. Connection **reset**, not refused:
the
socket was established and the DataNode closed it. #65939 created
`kerberos/conf/hdfs-site.xml.tpl` — the previous environment carried its
server
configuration inside the image, so this file is new — and it sets
`dfs.data.transfer.protection=authentication`. A client that does not set the
same property sends a plain `writeBlock` op, `SaslDataTransferServer` does
not
find the SASL magic number and closes the socket, the client excludes the
only
DataNode, and the next `addBlock` has no target left. Metadata operations
only
reach the NameNode and the HMS, so `CREATE DATABASE` / `CREATE TABLE` still
succeed and only the write fails.
The same commit added the client property to the four
`external_table_p0/kerberos`
cases but not to the p2 cases that talk to the same kerberized HDFS. Both
suites
were green through build 468 and red from build 469, the first run after
#65939.
They were held at `GeneralSecurityException: Checksum failed` until #66254
fixed
the keytab, which is why this second gap only surfaced now.
Fix: set `dfs.data.transfer.protection` in every property block that points
at
the kerberized HDFS. A scan of `:8520` / `:8620` across the whole suite tree
found
seven such blocks still missing it: two each in `hive_on_hms_and_dlf`,
`iceberg_on_hms_and_filesystem_and_dlf` and `test_paimon_hms_catalog`, plus
one
in `hdfs_all_test`, which writes to HDFS but currently returns early because
`refactor_params_hdfs_kerberos_test` is unset — it would fail the moment it
is
enabled. The kerberos block of `test_information_schema_timezone` has the
same
gap but is disabled by an unrelated TODO, so it is left alone.
**2. `test_paimon_hms_catalog`: `Unknown database 'hdfs_db'`**
The lightweight Kerberos environment from #65939 starts an empty metastore —
`schematool -dbType derby -initSchema` and nothing else. The environment it
replaced provisioned the Paimon fixture on every start (`hadoop fs -put
/tmp/paimon_data/*`, `create_paimon_hive_table.hql`, and the Paimon/jindo
jars in
the image's hive auxlib). None of that survived; `paimon_data/hdfs_db.db/` is
still in the tree but the compose file never mounted it.
This restores the fixture on `kerberos1` only — it owns metastore 9583, the
only
one the suite talks to, so `kerberos2` stays as light as it is today.
`hdfs_db` is
backed by `paimon_data`; `ali_db` carries no data and points at the same OSS
warehouse the Hive3 stack registers under that name, so the two metastores
share
one copy. Registering it still needs the `oss://` scheme to resolve, because
`HiveMetaStore.create_table_core` puts any explicit LOCATION through
`Warehouse.getDnsPath` → `Path.getFileSystem`; hence the jindo filesystem
settings
in `hive-site.xml` and the aliyun jars next to the Paimon one. The fixture
runs
before `DORIS_KERBEROS_READY`, so nothing races the suites, and a failure
aborts
the container instead of handing out an environment silently missing the two
databases.
The golden output already contains the `hdfs_kerberos`, `hdfs_new_kerberos`
and
`oss_hms_kerberos` blocks, so no baseline has to be regenerated.
**3. `test_file_cache_statistics` (flaky, ~25% pass rate)**
#66254 relaxed the upper bound after build 474 reported
`normal_queue_curr_size`
2.44e8 above a `max_size` of 1.68e8. Build 482 fails the other way round,
with
`normal_queue_curr_size` exactly 0 while `hits_ratio` is 3.76 — the queue is
empty
rather than over its share.
`test_file_cache_features` runs immediately before it and drives every
backend into
`disk_resource_limit_mode` and `need_evict_cache_in_advance` on purpose, by
lowering
the enter thresholds to 2%. Both modes drain the shared cache, and while
`disk_resource_limit_mode` is on `BlockFileCache::try_reserve` admits
nothing: it
multiplies the requested size by 5 and only succeeds if it evicted that
much, which
an already drained cache never can. `setBeConfigTemporary` restores the
configs, but
the backend only recomputes the two flags in `run_background_monitor`, once
per
`file_cache_background_monitor_interval_ms` (5s in this pipeline). The next
suite
starts 37ms later, so its warm-up queries can land inside that window and
cache
nothing — and since the case then only polls a passive metric, it can never
recover.
Both polls time out 30s later, which matches the log exactly.
Fix, in two parts:
- `test_file_cache_features` waits until both metrics are back to 0 before
it returns,
so it leaves the cluster in the state it found. The wait uses `max()`
rather than
`limit 1`, because one row is one arbitrary (backend, cache_path) pair and
any path
still in the mode reports a 1.
- `test_file_cache_statistics` re-runs the read on every poll of the normal
queue
metrics, the same self-healing shape its hit/read-count section already
uses.
The upper bound from #66254 is kept: build 474 shows that mode is real too.
**4. `IcebergMetadataOps`: report the root cause of failed metadata commits**
While diagnosing (1), the failure surfaced to the user as
`Failed to create table: ..., error message is:Self-suppression not
permitted`,
which says nothing about what went wrong. `TableMetadataParser.internalWrite`
writes the metadata JSON inside try-with-resources, and `DFSOutputStream`
throws
the same exception instance from `write` and from `close` once its pipeline
has
failed, so the generated `addSuppressed` call becomes `addSuppressed(this)`
and
`Throwable` replaces the failure with `IllegalArgumentException`. Every path
that
commits Iceberg metadata goes through the same parser and can be masked the
same
way. `addPartitionField`, `dropPartitionField` and `replacePartitionField`
already
report `ExceptionUtils.getRootCauseMessage(e)`; this applies it to the
remaining
eleven commit paths so all fourteen agree. The surrounding message is a
literal
prefix that already names the operation and the object, so replacing only the
trailing part loses no context, and no regression case asserts on this text.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
