This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 845aef6a9ad [chore](ci) Own pipeline configuration and restore
validated Cloud S3 suites (#70444)
845aef6a9ad is described below
commit 845aef6a9ad7c964117e363b0f99614c1cc349d1
Author: Dongyang Li <[email protected]>
AuthorDate: Wed Nov 11 13:41:04 2026 +0800
[chore](ci) Own pipeline configuration and restore validated Cloud S3
suites (#70444)
### Background
Rootless CI previously appended fixed settings and suite exclusions
after copying configuration from the cases package. This split
configuration ownership between the repository and deployment scripts.
The regression framework remains unmodified.
### What problem does this PR solve?
Issue Number: None
Related PR: #70407 (earlier source-owned compatibility exclusions)
PR Type: Build/CI/Chore
Problem Summary:
Keep fixed CI settings and remaining exclusions in the existing profile
configuration files so changes can be reviewed here. Restore 50 CloudP0
suites that passed targeted validation using the existing S3 case
credentials. Preserve the original load-suite initialization before
ordinary regression suites. Keep the temporary-token suite excluded
until its STS prerequisites are configured and validated.
### Details
- **Affected:** P0, CloudP0, VaultP0, External and NonConcurrent
configuration ownership. CloudP0 removes exactly 50 exclusions: 43
entries from the adapter migration and 7 previously source-owned
entries. Other exclusions, groups and directory filters are unchanged.
`test_s3_load_with_tmp_token` remains excluded because
`regressionAliyunStsRegion` and `regressionAliyunStsRoleArn` are not
configured in rootless CI; successful static-credential tests do not
establish AssumeRole coverage.
- **Developer and release impact:** Master only. The existing
`datatype_p0/nested_types/load.groovy` suite initializes its parent
database through `SuiteContext`; the framework runs load suites before
ordinary suites. Full CloudP0 keeps that original initialization, with
no added deployment SQL. Targeted selectors must include the parent load
suite rather than bypass it. No framework, case, assertion or
expected-result edits. Runtime addresses, directories, interpreter
identity and credentials remain deployment inputs.
- **License and third-party:** No dependencies or licenses added.
- **Rollback:** The one-line `ci-config-version` marker distinguishes
source-owned configuration from older cases packages. Removing it would
select the retained legacy adapter and append the old exclusions again.
Unmarked master and release-branch packages keep that complete legacy
behavior. The marker does not replace commit or artifact verification.
- **Behavior change:** CloudP0 restores the 50 validated suites while
retaining the original load-suite initialization. Other profiles retain
their effective configuration.
---
regression-test/pipeline/CI_CONFIG.md | 12 +++++
regression-test/pipeline/ci-config-version | 1 +
.../pipeline/cloud_p0/conf/be_custom.conf | 3 +-
.../pipeline/cloud_p0/conf/ms_custom.conf | 3 +-
.../cloud_p0/conf/regression-conf-custom.groovy | 54 ++++++++++++++++------
regression-test/pipeline/external/conf/fe.conf | 6 +++
.../pipeline/external/conf/regression-conf.groovy | 25 ++++++++++
.../pipeline/nonConcurrent/conf/fe.conf | 6 +++
.../nonConcurrent/conf/regression-conf.groovy | 10 ++++
regression-test/pipeline/p0/conf/fe.conf | 6 +++
.../pipeline/p0/conf/regression-conf.groovy | 10 ++++
.../pipeline/vault_p0/conf/be_custom.conf | 3 +-
.../pipeline/vault_p0/conf/ms_custom.conf | 3 +-
.../vault_p0/conf/regression-conf-custom.groovy | 11 +++++
14 files changed, 134 insertions(+), 19 deletions(-)
diff --git a/regression-test/pipeline/CI_CONFIG.md
b/regression-test/pipeline/CI_CONFIG.md
new file mode 100644
index 00000000000..598ab5630b8
--- /dev/null
+++ b/regression-test/pipeline/CI_CONFIG.md
@@ -0,0 +1,12 @@
+# CI configuration
+
+Fixed settings and suite exclusions live in each profile's existing
configuration
+files. `DORIS_CI_PRISTINE_FRAMEWORK=true` enables the reviewed settings for CI
+running the unmodified regression framework. Deployment supplies runtime
addresses,
+paths and credentials.
+
+`ci-config-version` contains `1` so deployment can distinguish these
source-owned
+settings from older cases packages. Marked packages use the source
configuration;
+unmarked packages retain the legacy adapter additions, including their
exclusions.
+Removing the marker would select that fallback again. Unknown versions are
rejected.
+The marker does not replace the artifact manifest or PR commit checks.
diff --git a/regression-test/pipeline/ci-config-version
b/regression-test/pipeline/ci-config-version
new file mode 100644
index 00000000000..d00491fd7e5
--- /dev/null
+++ b/regression-test/pipeline/ci-config-version
@@ -0,0 +1 @@
+1
diff --git a/regression-test/pipeline/cloud_p0/conf/be_custom.conf
b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
index c4aa0619d9f..a904314fe4c 100644
--- a/regression-test/pipeline/cloud_p0/conf/be_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/be_custom.conf
@@ -63,7 +63,8 @@ fail_when_segment_rows_not_in_rowset_meta=true
# enable to use python udf
enable_python_udf_support=true
-python_env_mode=conda
+# The CI image supplies a Python venv; conda is not installed in this runtime.
+python_env_mode = venv
python_conda_root_path=/opt/miniforge3
max_python_process_num=16
diff --git a/regression-test/pipeline/cloud_p0/conf/ms_custom.conf
b/regression-test/pipeline/cloud_p0/conf/ms_custom.conf
index c1a44037f42..954829c7011 100644
--- a/regression-test/pipeline/cloud_p0/conf/ms_custom.conf
+++ b/regression-test/pipeline/cloud_p0/conf/ms_custom.conf
@@ -1,4 +1,5 @@
# below lines will be appended to the default doris_cloud.conf when deploying
meta service
meta_schema_value_version = 1
-enable_logging_conflict_keys = true
+# Existing CI mitigation for observed MetaService memory growth during
conflict-key logging.
+enable_logging_conflict_keys = false
cloud_txn_lazy_commit_fuzzy_possibility = 50
diff --git
a/regression-test/pipeline/cloud_p0/conf/regression-conf-custom.groovy
b/regression-test/pipeline/cloud_p0/conf/regression-conf-custom.groovy
index bc312fd632b..03db1f3e5c4 100644
--- a/regression-test/pipeline/cloud_p0/conf/regression-conf-custom.groovy
+++ b/regression-test/pipeline/cloud_p0/conf/regression-conf-custom.groovy
@@ -120,8 +120,6 @@ pythonUdfRuntimeVersion = "3.12.11"
// Historical injected-adapter failures below are not evidence of
original-framework defects.
if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
excludeSuites += "," + [
- // Historical injected OUTFILE rewriter rejected two statement
terminators; original framework cause not established.
- "dml_into_outfile",
// Historical local-rootless BE filesystem adapter failure; exact
rejected path/operation remains unresolved.
"test_export_basic",
// Historical local-rootless BE filesystem adapter failure; exact
rejected path/operation remains unresolved.
@@ -131,13 +129,7 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
// Historical local-rootless BE filesystem adapter failure; exact
rejected path/operation remains unresolved.
"test_export_table_with_label_retry",
// Historical local-rootless BE filesystem adapter failure; exact
rejected path/operation remains unresolved.
- "test_nereids_show_export",
- // Historical CREATE REPOSITORY used public-bucket ANONYMOUS SQL while
migration backup routing was disabled.
- "test_temp_table",
- // Historical injected TVF rewriter left repeated-TVF SQL using
original public locations; original framework cause not established.
- "test_uuid_csv_schema",
- // Historical injected TVF rewriter left repeated-TVF SQL using
original public locations; original framework cause not established.
- "test_uuid_parquet_native"
+ "test_nereids_show_export"
].join(",")
// Temporary environment exclusions using the existing suite-name filter.
@@ -146,12 +138,6 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
// and export_p0/outfile/parquet/test_outfile_parquet.groovy. No whole
directory is excluded.
// The two independent-provider credential cases already have the p2 group
in master.
excludeSuites += "," + [
- // shared parent database prerequisite
- "test_nested_types_insert_into_with_s3",
- // shared parent database prerequisite
- "test_nestedtypes_csv_insert_into_with_s3",
- // shared parent database prerequisite
- "test_nestedtypes_json_insert_into_with_s3",
// root SSH/SCP or remote filesystem prerequisite
"test_export_view",
// mysqldump column-statistics compatibility
@@ -171,3 +157,41 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
// This does not establish a wrong query result.
excludeSuites += ",scalar_agg_pushdown"
}
+
+// Move remaining compatibility exclusions from the CI adapter into reviewed
source.
+// These observations do not establish a universal framework defect.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ excludeSuites += "," + [
+ // Build 477: original suite assertion/result mismatch; exact cause
pending, not attributed to OOM. Restore: Resolve recorded failure and pass the
isolated original suite before removing this exclusion.
+ "test_stream_load_error_url",
+ // Build 477: original suite assertion/result mismatch; exact cause
pending, not attributed to OOM. Restore: Resolve recorded failure and pass the
isolated original suite before removing this exclusion.
+ "test_csv_with_none_utf8_data",
+ // Build 477: original suite assertion/result mismatch; exact cause
pending, not attributed to OOM. Restore: Resolve recorded failure and pass the
isolated original suite before removing this exclusion.
+ "rewrite_when_dml",
+ // Requires regressionAliyunStsRegion and regressionAliyunStsRoleArn
plus AssumeRole access.
+ // These inputs are not configured in rootless CI; static S3
credentials do not cover this suite.
+ "test_s3_load_with_tmp_token",
+ // Build 477: error-URL expectation differs; SQL returns please use
show load instead. Restore: Resolve recorded failure and pass the isolated
original suite before removing this exclusion.
+ "test_insert_strict_fail_url",
+ // Build 477: error-URL expectation differs; SQL returns please use
show load instead. Restore: Resolve recorded failure and pass the isolated
original suite before removing this exclusion.
+ "test_insert_partition_fail_url",
+ // Build 477: error-URL expectation differs; SQL returns please use
show load instead. Restore: Resolve recorded failure and pass the isolated
original suite before removing this exclusion.
+ "test_insert_docs_demo",
+ // Build 434: active near initial MS RSS surge (19:18:47-19:19:02);
temporal correlation only, not proven cause. Restore: Developer MS
investigation plus isolated/repeated memory-tracked pass, then re-enable
individually.
+ "multi_slot",
+ // Build 434: active near initial MS RSS surge (19:18:47-19:19:02);
temporal correlation only, not proven cause. Restore: Developer MS
investigation plus isolated/repeated memory-tracked pass, then re-enable
individually.
+ "variant_mv_rowstore_crash",
+ // Build 434: active near initial MS RSS surge (19:18:47-19:19:02);
temporal correlation only, not proven cause. Restore: Developer MS
investigation plus isolated/repeated memory-tracked pass, then re-enable
individually.
+ "test_variant_search_subcolumn_snii"
+ ].join(",")
+}
+
+// Reviewed defaults for CI using the unmodified framework and rootless
runtime.
+// Preserve other environments; addresses, local paths, credentials and build
IDs are injected at deployment.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ // Use the existing read-only regression fixtures in Hong Kong.
+ s3BucketName = "doris-regression-hk"
+ s3Endpoint = "oss-cn-hongkong.aliyuncs.com"
+ s3Region = "oss-cn-hongkong"
+ s3Provider = "S3"
+}
diff --git a/regression-test/pipeline/external/conf/fe.conf
b/regression-test/pipeline/external/conf/fe.conf
index 34fa5a440a5..0994db451a3 100644
--- a/regression-test/pipeline/external/conf/fe.conf
+++ b/regression-test/pipeline/external/conf/fe.conf
@@ -105,3 +105,9 @@ hms_events_polling_interval_ms=700
KRB5_CONFIG=/keytabs/krb5.conf
+
+# Match the regression client ports used by the single-container CI cluster.
+query_port = 9131
+
+# Match the regression client ports used by the single-container CI cluster.
+http_port = 8131
diff --git a/regression-test/pipeline/external/conf/regression-conf.groovy
b/regression-test/pipeline/external/conf/regression-conf.groovy
index eff4c724778..3d1c5953080 100644
--- a/regression-test/pipeline/external/conf/regression-conf.groovy
+++ b/regression-test/pipeline/external/conf/regression-conf.groovy
@@ -241,3 +241,28 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
"test_parquet_meta_tvf"
].join(",")
}
+
+// Reviewed defaults for CI using the unmodified framework and rootless
runtime.
+// Preserve other environments; addresses, local paths, credentials and build
IDs are injected at deployment.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ // Use the existing read-only regression fixtures in Hong Kong.
+ s3BucketName = "doris-regression-hk"
+ s3Endpoint = "oss-cn-hongkong.aliyuncs.com"
+ s3Region = "oss-cn-hongkong"
+ s3FixtureProtocol = "https"
+ ciFixtureBaseUrl =
"https://doris-regression-hk.oss-cn-hongkong.aliyuncs.com";
+ ciFixtureStsEnabled = true
+ ciScopedDynamicFixturesEnabled = true
+ ciFixtureProfile = "external"
+ // Preserve the existing local filesystem contract for the rootless
external lane.
+ backendFilesystemTransport = "local-rootless"
+ ciTrinoLocalDispatchEnabled = true
+ backendLocalPathAllowlist = "[\"/tmp/test_export_view\"]"
+ // Preserve the existing build-scoped backup repository; its build ID is
injected separately.
+ ciBackupRepositoryEnabled = true
+ ciBackupRepositoryBucket = "doris-ci-artifacts-1949087886039699-hk"
+ ciBackupRepositoryEndpoint =
"https://oss-cn-hongkong-internal.aliyuncs.com";
+ ciBackupRepositoryRegion = "oss-cn-hongkong"
+ ciBackupRepositoryProfile = "external"
+ ciBackupRepositoryPrefix = "external"
+}
diff --git a/regression-test/pipeline/nonConcurrent/conf/fe.conf
b/regression-test/pipeline/nonConcurrent/conf/fe.conf
index d283b2a15cf..2e4e995e23a 100644
--- a/regression-test/pipeline/nonConcurrent/conf/fe.conf
+++ b/regression-test/pipeline/nonConcurrent/conf/fe.conf
@@ -94,3 +94,9 @@ max_spilled_profile_num = 2000
check_table_lock_leaky=true
enable_table_stream=true
+
+# Match the regression client ports used by the single-container CI cluster.
+query_port = 9131
+
+# Match the regression client ports used by the single-container CI cluster.
+http_port = 8131
diff --git a/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
b/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
index 749f8bad040..e1bb96e77c2 100644
--- a/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
+++ b/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
@@ -180,3 +180,13 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
"test_analyze_long_string"
].join(",")
}
+
+// Reviewed defaults for CI using the unmodified framework and rootless
runtime.
+// Preserve other environments; addresses, local paths, credentials and build
IDs are injected at deployment.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ // Use the existing read-only regression fixtures in Hong Kong.
+ s3BucketName = "doris-regression-hk"
+ s3Endpoint = "oss-cn-hongkong.aliyuncs.com"
+ s3Region = "oss-cn-hongkong"
+ s3Provider = "S3"
+}
diff --git a/regression-test/pipeline/p0/conf/fe.conf
b/regression-test/pipeline/p0/conf/fe.conf
index 61c6a5a4962..343078f485a 100644
--- a/regression-test/pipeline/p0/conf/fe.conf
+++ b/regression-test/pipeline/p0/conf/fe.conf
@@ -96,3 +96,9 @@ check_table_lock_leaky=true
max_bucket_num_per_partition=512
enable_table_stream=true
max_remote_file_system_cache_num=1000
+
+# Match the regression client ports used by the single-container CI cluster.
+query_port = 9131
+
+# Match the regression client ports used by the single-container CI cluster.
+http_port = 8131
diff --git a/regression-test/pipeline/p0/conf/regression-conf.groovy
b/regression-test/pipeline/p0/conf/regression-conf.groovy
index df30ebf4885..7065aad8194 100644
--- a/regression-test/pipeline/p0/conf/regression-conf.groovy
+++ b/regression-test/pipeline/p0/conf/regression-conf.groovy
@@ -347,3 +347,13 @@ if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") ==
"true") {
"test_replace_table"
].join(",")
}
+
+// Reviewed defaults for CI using the unmodified framework and rootless
runtime.
+// Preserve other environments; addresses, local paths, credentials and build
IDs are injected at deployment.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ // Use the existing read-only regression fixtures in Hong Kong.
+ s3BucketName = "doris-regression-hk"
+ s3Endpoint = "oss-cn-hongkong.aliyuncs.com"
+ s3Region = "oss-cn-hongkong"
+ s3Provider = "S3"
+}
diff --git a/regression-test/pipeline/vault_p0/conf/be_custom.conf
b/regression-test/pipeline/vault_p0/conf/be_custom.conf
index ccfdac33c88..c042334ef38 100644
--- a/regression-test/pipeline/vault_p0/conf/be_custom.conf
+++ b/regression-test/pipeline/vault_p0/conf/be_custom.conf
@@ -43,7 +43,8 @@ enable_brpc_connection_check=true
# enable to use python udf
enable_python_udf_support=true
-python_env_mode=venv
+# The CI image supplies a Python venv; conda is not installed in this runtime.
+python_env_mode = venv
python_venv_interpreter_paths=/usr/bin/python
max_python_process_num=16
diff --git a/regression-test/pipeline/vault_p0/conf/ms_custom.conf
b/regression-test/pipeline/vault_p0/conf/ms_custom.conf
index 64ba3116f5f..ad00150db29 100644
--- a/regression-test/pipeline/vault_p0/conf/ms_custom.conf
+++ b/regression-test/pipeline/vault_p0/conf/ms_custom.conf
@@ -1,3 +1,4 @@
# below lines will be appended to the default doris_cloud.conf when deploying
meta service
meta_schema_value_version = 1
-enable_logging_conflict_keys = true
+# Existing CI mitigation for observed MetaService memory growth during
conflict-key logging.
+enable_logging_conflict_keys = false
diff --git
a/regression-test/pipeline/vault_p0/conf/regression-conf-custom.groovy
b/regression-test/pipeline/vault_p0/conf/regression-conf-custom.groovy
index 3308682b0ba..69af614ee8b 100644
--- a/regression-test/pipeline/vault_p0/conf/regression-conf-custom.groovy
+++ b/regression-test/pipeline/vault_p0/conf/regression-conf-custom.groovy
@@ -45,3 +45,14 @@ extMinioRegion = "us-east-1"
extMinioBucket = "test-bucket"
s3Endpoint = "oss-cn-hongkong-internal.aliyuncs.com"
+
+// Reviewed defaults for CI using the unmodified framework and rootless
runtime.
+// Preserve other environments; addresses, local paths, credentials and build
IDs are injected at deployment.
+if (System.getenv("DORIS_CI_PRISTINE_FRAMEWORK") == "true") {
+ // Use the existing read-only regression fixtures in Hong Kong.
+ s3BucketName = "doris-regression-hk"
+ s3Endpoint = "oss-cn-hongkong.aliyuncs.com"
+ s3Region = "oss-cn-hongkong"
+ // Vault fixtures use the OSS provider; the other S3 lanes retain S3.
+ s3Provider = "OSS"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
