This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7856e3bf67f40406bab66b9626b7cf6c25d440aa Author: Gabriel <[email protected]> AuthorDate: Sat Oct 10 18:32:58 2026 +0800 [fix](regression) Backport external and Kerberos fixes to branch-4.1 (#68150, #66433) (#68869) Backport #68150 and #66433 to branch-4.1. Trino startup/query failures and unfinished HDFS Broker Load jobs could appear as empty-result mismatches. Kerberized HDFS clients lacked the DataNode SASL setting, and the lightweight Kerberos metastore lacked the Paimon fixture. Iceberg metadata write failures could expose only a self-suppression wrapper. This backport adds bounded readiness/load checks with failure diagnostics, configures HDFS transfer protection, restores the Paimon fixture before the Kerberos readiness marker, and reports the root cause of failed Iceberg metadata commits. Branch adaptation: branch-4.1 uses `IcebergMetadataOps`, so the root-cause changes are applied to its create-table, schema-update, and branch/tag commit paths. Its partition-spec update helper already reports root causes. The newer connector implementation is not introduced. Validation: - FE Checkstyle passes: `mvn -pl fe-core -am checkstyle:check`. - All six changed suites compile with Groovy 4.0.19. - Eight local control-flow scenarios pass for Trino readiness/query errors and Broker Load completion/cancellation/timeouts. - Shell syntax, Kerberos settings isolation, both container role/switch combinations, env/config rendering, and rendered XML checks pass. - An isolated harness compiles the actual adapted create-table/schema-commit methods with stubbed dependencies: the original masks the storage failure, while the backport preserves the root message, operation context, and exception cause. - `git diff --check` passes. Full FE build and external integration runs remain pending CI. Release note: Improved external regression setup and diagnostics; Iceberg metadata commit errors now expose the underlying cause. Co-authored-by: Mingyu Chen (Rayner) <[email protected]> --- .../kerberos/entrypoint-hive-master.sh | 26 ++++++++++++ .../docker-compose/kerberos/hadoop-hive.env.tpl | 4 ++ .../docker-compose/kerberos/kerberos.yaml.tpl | 6 +++ .../kerberos/sql/create_paimon_hive_table.hql | 20 +++++++++ docker/thirdparties/run-thirdparties-docker.sh | 47 +++++++++++++++++++++- .../datasource/iceberg/IcebergMetadataOps.java | 16 ++++---- .../iceberg/test_iceberg_deletion_vector.groovy | 42 ++++++++++++------- .../refactor_storage_param/hdfs_all_test.groovy | 2 + .../tvf/test_s3_tvf_number_range.groovy | 32 +++++++-------- .../paimon/test_paimon_hms_catalog.groovy | 4 +- .../hive_on_hms_and_dlf.groovy | 4 +- .../iceberg_on_hms_and_filesystem_and_dlf.groovy | 4 +- 12 files changed, 165 insertions(+), 42 deletions(-) diff --git a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh index 6735ae7ee40..885a415d996 100644 --- a/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh +++ b/docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh @@ -136,9 +136,35 @@ kdestroy report_stage "initialize-hive-metastore" schematool -dbType derby -initSchema +# The Paimon table declares no columns, so the metastore itself resolves its +# schema through the storage handler. The handler class must therefore be on the +# metastore service's own classpath, not merely on the DDL client's. +if [[ -d /opt/doris/auxlib ]]; then + export HIVE_AUX_JARS_PATH=/opt/doris/auxlib +fi start_service hive --service metastore -p "${HMS_PORT}" wait_for_port "${HOST}" "${HMS_PORT}" "Hive Metastore" +# Register the Paimon fixture consumed by test_paimon_hms_catalog. This runs +# before the readiness marker below on purpose: run-thirdparties-docker.sh +# releases the pipeline on DORIS_KERBEROS_READY, so anything published later +# would race the suites. A failure here aborts the container (set -e) instead of +# handing out an environment that is silently missing hdfs_db. +# +# Both kerberos containers run this entrypoint with the same env switch, but the +# fixture and its mounts (sql/, paimon_data/, auxlib/) belong to kerberos1 only - +# its metastore (9583) is the one the suite talks to. Gate on the container role, +# not on the mounts, so a broken mount on kerberos1 still fails loudly. +if [[ "${enablePaimonHms:-false}" == "true" && "${HOST:-}" == "hadoop-master" ]]; then + report_stage "load-paimon-hms" + export KRB5CCNAME=FILE:/tmp/hive-admin.ccache + kinit -kt /data/keytabs/hive.keytab "${HIVE_PRINCIPAL}" + hdfs dfs -mkdir -p /user/hive/warehouse + hdfs dfs -put -f /opt/doris/paimon_data/* /user/hive/warehouse/ + hive -f /opt/doris/sql/create_paimon_hive_table.hql + kdestroy +fi + touch /tmp/SUCCESS echo "Minimal Kerberos HDFS and Hive Metastore environment is ready" echo "DORIS_KERBEROS_READY" diff --git a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl index 792d0c0491d..ff95e56f1ae 100644 --- a/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl +++ b/docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl @@ -29,3 +29,7 @@ PRESTO_CLIENT_KEYTAB=${PRESTO_CLIENT_KEYTAB} HADOOP_CONF_DIR=/opt/doris/conf HIVE_CONF_DIR=/opt/doris/conf KRB5_CONFIG=/etc/krb5.conf +# Paimon-on-kerberized-HMS fixture switch. Read from hive-3x_settings.env by +# start_kerberos() rather than duplicated into kerberos*_settings.env, so that +# the pipeline keeps patching exactly one file and the two stacks cannot drift. +enablePaimonHms=${enablePaimonHms} diff --git a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl index d21496516af..8e903ed6d7f 100644 --- a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl +++ b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl @@ -33,11 +33,17 @@ services: <<: *kerberos-service container_name: doris-${CONTAINER_UID}-kerberos1 hostname: hadoop-master + # auxlib/sql/paimon_data are mounted on kerberos1 only: test_paimon_hms_catalog + # is the sole consumer and it talks to this metastore (9583). Leaving kerberos2 + # untouched keeps the second container as light as it was. volumes: - ./conf/kerberos1:/opt/doris/conf:ro - ./conf/kerberos1/krb5.conf:/etc/krb5.conf:ro - ./data/kerberos1:/data - ./two-kerberos-hives:/keytabs + - ./auxlib:/opt/doris/auxlib:ro + - ./sql:/opt/doris/sql:ro + - ./paimon_data:/opt/doris/paimon_data:ro env_file: - ./hadoop-hive-1.env diff --git a/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql new file mode 100644 index 00000000000..69a6f476286 --- /dev/null +++ b/docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql @@ -0,0 +1,20 @@ +-- Paimon table registered in the kerberized Hive Metastore, consumed by the +-- hdfs_kerberos / hdfs_new_kerberos cases of +-- external_table_p2/paimon/test_paimon_hms_catalog.groovy. +-- +-- hdfs_db is backed by ../paimon_data, which the entrypoint uploads into HDFS. +-- +-- The table declares no columns, so the metastore derives them through the +-- Paimon storage handler (metastore.storage.schema.reader.impl is +-- SerDeStorageSchemaReader) -- the Paimon jar has to be on the metastore's own +-- classpath, not just on the classpath of whoever runs this script. + +CREATE DATABASE IF NOT EXISTS hdfs_db; + +USE hdfs_db; + +DROP TABLE IF EXISTS external_test_table; + +CREATE EXTERNAL TABLE external_test_table + STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' +LOCATION 'hdfs:///user/hive/warehouse/hdfs_db.db/external_test_table'; diff --git a/docker/thirdparties/run-thirdparties-docker.sh b/docker/thirdparties/run-thirdparties-docker.sh index 63154a15473..14babf73026 100755 --- a/docker/thirdparties/run-thirdparties-docker.sh +++ b/docker/thirdparties/run-thirdparties-docker.sh @@ -1574,6 +1574,46 @@ validate_kerberos_container() { fi } +# Aux jar for the Paimon fixture in the kerberized metastore: the Paimon storage +# handler the metastore loads to derive the fixture table's schema. Same source +# and same skip-if-present caching the Hive3 stack uses in prepare-hive-data.sh. +download_kerberos_paimon_jars() { + local auxlib_dir="$1" + local url_prefix="https://${s3BucketName}.${s3Endpoint}/regression/docker/hive3"; + local jars=( + paimon-hive-connector-3.1-1.3-SNAPSHOT.jar + ) + local jar + + for jar in "${jars[@]}"; do + if [[ -f "${auxlib_dir}/${jar}" ]]; then + echo "Reuse cached kerberos aux jar ${jar}" + continue + fi + echo "Download kerberos aux jar ${jar}" + curl -sSfL -o "${auxlib_dir}/${jar}" "${url_prefix}/${jar}" + done +} + +# enablePaimonHms lives in hive-3x_settings.env, which the pipeline already +# patches with the real value. Read it from there instead of duplicating the +# switch into kerberos*_settings.env, where the two copies would drift and one +# stack would silently lose the fixture. Sourced in a subshell so that hive3's +# FS_PORT / HMS_PORT cannot leak into the kerberos settings sourced a few lines +# below. +resolve_kerberos_paimon_env() { + local hive3_settings="${ROOT}/docker-compose/hive/hive-3x_settings.env" + + enablePaimonHms="false" + if [[ -f "${hive3_settings}" ]]; then + read -r enablePaimonHms < <( + . "${hive3_settings}" >/dev/null 2>&1 + printf '%s\n' "${enablePaimonHms:-false}" + ) + fi + export enablePaimonHms +} + start_kerberos() { echo "RUN_KERBEROS" local KERBEROS_DIR="${ROOT}/docker-compose/kerberos" @@ -1588,8 +1628,13 @@ start_kerberos() { export CONTAINER_UID=${CONTAINER_UID} envsubst <"${KERBEROS_DIR}/kerberos.yaml.tpl" >"${KERBEROS_DIR}/kerberos.yaml" + # auxlib must exist even when the fixture is off: kerberos1 mounts it read-only. mkdir -p "${KERBEROS_DIR}/conf/kerberos1" "${KERBEROS_DIR}/conf/kerberos2" \ - "${KERBEROS_DIR}/two-kerberos-hives" + "${KERBEROS_DIR}/two-kerberos-hives" "${KERBEROS_DIR}/auxlib" + resolve_kerberos_paimon_env + if [[ "${enablePaimonHms}" == "true" && "${STOP}" -ne 1 ]]; then + download_kerberos_paimon_jars "${KERBEROS_DIR}/auxlib" + fi for i in {1..2}; do . "${KERBEROS_DIR}/kerberos${i}_settings.env" envsubst <"${KERBEROS_DIR}/hadoop-hive.env.tpl" >"${KERBEROS_DIR}/hadoop-hive-${i}.env" diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java index ccc693f4ad2..0e605f9790d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java @@ -345,9 +345,10 @@ public class IcebergMetadataOps implements ExternalMetadataOps { try { return executeCatalogOperation(() -> performCreateTable(createTableInfo)); } catch (Exception e) { + // Metadata writes can be wrapped by a self-suppression error; expose the underlying failure. throw new DdlException( - "Failed to create table: " + createTableInfo.getTableName() + ", error message is:" + e.getMessage(), - e); + "Failed to create table: " + createTableInfo.getTableName() + ", error message is:" + + ExceptionUtils.getRootCauseMessage(e), e); } } @@ -582,7 +583,7 @@ public class IcebergMetadataOps implements ExternalMetadataOps { } catch (Exception e) { throw new RuntimeException( "Failed to create or replace branch: " + branchName + " in table: " + icebergTable.name() - + ", error message is: " + e.getMessage(), e); + + ", error message is: " + ExceptionUtils.getRootCauseMessage(e), e); } } } @@ -650,7 +651,7 @@ public class IcebergMetadataOps implements ExternalMetadataOps { } catch (Exception e) { throw new RuntimeException( "Failed to create or replace tag: " + tagName + " in table: " + icebergTable.name() - + ", error message is: " + e.getMessage(), e); + + ", error message is: " + ExceptionUtils.getRootCauseMessage(e), e); } } } @@ -672,7 +673,7 @@ public class IcebergMetadataOps implements ExternalMetadataOps { } catch (Exception e) { throw new RuntimeException( "Failed to drop tag: " + tagName + " in table: " + icebergTable.name() - + ", error message is: " + e.getMessage(), e); + + ", error message is: " + ExceptionUtils.getRootCauseMessage(e), e); } } } @@ -695,7 +696,7 @@ public class IcebergMetadataOps implements ExternalMetadataOps { } catch (Exception e) { throw new RuntimeException( "Failed to drop branch: " + branchName + " in table: " + icebergTable.name() - + ", error message is: " + e.getMessage(), e); + + ", error message is: " + ExceptionUtils.getRootCauseMessage(e), e); } } } @@ -792,7 +793,8 @@ public class IcebergMetadataOps implements ExternalMetadataOps { try { lease.getAuthenticator().execute(commit); } catch (Exception e) { - throw new UserException(failureMessage + ", error message is: " + e.getMessage(), e); + // Preserve the storage failure hidden by metadata writer cleanup exceptions. + throw new UserException(failureMessage + ", error message is: " + ExceptionUtils.getRootCauseMessage(e), e); } } diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy index 9b06efc8a85..8d513f0c9f5 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_deletion_vector.groovy @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +import java.util.concurrent.TimeUnit + suite("test_iceberg_deletion_vector", "p0,external,nonConcurrent") { String enabled = context.config.otherConfigs.get("enableIcebergTest") if (enabled == null || !enabled.equalsIgnoreCase("true")) { @@ -495,22 +497,34 @@ s3.path-style-access=true 30 ) executeCommand("${dockerCommand} restart ${trinoContainerName}", true, 60) - String trinoRows = "" - for (int i = 0; i < 12; i++) { - Thread.sleep(5000) - trinoRows = normalizeExternalRows(executeCommand( - "${dockerCommand} exec ${trinoContainerName} trino --output-format TSV " + - "--catalog iceberg --schema format_v3 --execute " + - "\"SELECT id, batch, data " + - "FROM dv_delete_matrix_equality_and_dv ORDER BY id\"", - false, - 120 - )) - if (!trinoRows.isEmpty()) { + String trinoCommand = "${dockerCommand} exec ${trinoContainerName} trino --output-format TSV " + + "--catalog iceberg --schema format_v3 --execute " + // A running container does not imply a ready coordinator. Retry startup separately so a + // failed data query cannot be mistaken for a successful query returning no rows. + long trinoReadyDeadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(300) + def readiness = [exitCode: -1, stdout: "", stderr: "Readiness probe has not run"] + while (System.nanoTime() < trinoReadyDeadline) { + readiness = executeCommandWithStatus(trinoCommand + '"SELECT 1"', 10, false, false) + if (readiness.exitCode == 0 && readiness.stdout.trim() == "1") { break - } + } + Thread.sleep(1000) } - assertEquals(expectedRows, trinoRows) + assertTrue(readiness.exitCode == 0 && readiness.stdout.trim() == "1", + "Trino did not become ready within 300 seconds. Exit code: ${readiness.exitCode}\n" + + "stdout:\n${readiness.stdout}\nstderr:\n${readiness.stderr}") + + def trinoResult = executeCommandWithStatus( + trinoCommand + '"SELECT id, batch, data ' + + 'FROM dv_delete_matrix_equality_and_dv ORDER BY id"', + 120 + ) + assertEquals(0, trinoResult.exitCode, + "Trino comparison query failed. stdout:\n${trinoResult.stdout}\nstderr:\n${trinoResult.stderr}") + String trinoRows = normalizeExternalRows(trinoResult.stdout) + assertEquals(expectedRows, trinoRows, + "Trino comparison query returned unexpected rows. " + + "stdout:\n${trinoResult.stdout}\nstderr:\n${trinoResult.stderr}") def profileCounterValues = { String profileText, String counterName -> def values = [] diff --git a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy index 25ca5ebf6da..c2a8a8cdec5 100644 --- a/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy +++ b/regression-test/suites/external_table_p0/refactor_storage_param/hdfs_all_test.groovy @@ -74,9 +74,11 @@ suite("refactor_params_hdfs_all_test", "p0,external,kerberos,external_docker,ext assert insertResult.get(0).get(0) == 1 } + // Explicit client properties must match the DataNode SASL protection to allow writes. def hdfsNonXmlParams = "\"fs.defaultFS\" = \"hdfs://${externalEnvIp}:8520\",\n" + "\"dfs.namenode.kerberos.principal\" = \"hdfs/[email protected]\",\n" + "\"dfs.client.use.datanode.hostname\" = \"true\",\n" + + "\"dfs.data.transfer.protection\" = \"authentication\",\n" + "\"hadoop.security.token.service.use_ip\" = \"false\",\n" + "\"hadoop.kerberos.min.seconds.before.relogin\" = \"5\",\n" + "\"hadoop.security.authentication\" = \"kerberos\",\n" + diff --git a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy index 3e84bce35ba..216c869f131 100644 --- a/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_s3_tvf_number_range.groovy @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +import java.util.concurrent.TimeUnit + suite("test_s3_tvf_number_range", "p0,external") { String ak = getS3AK() @@ -351,29 +353,25 @@ suite("test_s3_tvf_number_range", "p0,external") { // Helper closure to check load result def check_hdfs_load_result = {checklabel -> - def max_try_milli_secs = 10000 - def success = false - while(max_try_milli_secs) { - def result = sql """ SHOW LOAD WHERE LABEL = '${checklabel}' """ + // Broker Load is asynchronous: cancellation or an unfinished job must fail here, + // before the caller compares rows in the freshly truncated table. + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(120) + def result = [] + while (System.nanoTime() < deadline) { + result = sql """ SHOW LOAD WHERE LABEL = '${checklabel}' """ if (result.size() > 0) { def state = result[0][2] // State column if (state == "FINISHED") { sql "sync" - success = true - break - } else if (state == "CANCELLED") { - logger.error("HDFS load job ${checklabel} was cancelled: ${result[0]}") - break + return } + assertTrue(state != "CANCELLED", + "HDFS load job ${checklabel} was cancelled. Status: ${result}") } - sleep(1000) // wait 1 second every time - max_try_milli_secs-=1000 - } - - if (!success) { - def result = sql """ SHOW LOAD WHERE LABEL = '${checklabel}' """ - logger.error("HDFS load job ${checklabel} failed or timeout. Status: ${result}") + sleep(1000) } + assertTrue(false, + "HDFS load job ${checklabel} did not finish within 120 seconds. Last status: ${result}") } // Test 12: HDFS Broker Load Single range {1..3} - should load {1,2,3} @@ -407,4 +405,4 @@ suite("test_s3_tvf_number_range", "p0,external") { } } sql """ DROP TABLE IF EXISTS ${test_table} """ -} \ No newline at end of file +} diff --git a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy index 01b98dc02b2..c1810ca24be 100644 --- a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy +++ b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy @@ -119,11 +119,12 @@ suite("test_paimon_hms_catalog", "p2,external,paimon,new_catalog_property") { 'fs.defaultFS' = 'hdfs://${extHiveHmsHost}:${extHiveHdfsHost}' """ - // kerberos + // Both Kerberos property forms must match the DataNode SASL protection to allow writes. String hdfs_kerberos_properties = """ "fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "hadoop.security.authentication" = "kerberos", "hadoop.kerberos.principal"="hive/[email protected]", @@ -134,6 +135,7 @@ suite("test_paimon_hms_catalog", "p2,external,paimon,new_catalog_property") { "fs.defaultFS" = "hdfs://${extHiveHmsHost}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "hdfs.authentication.type" = "kerberos", "hdfs.authentication.kerberos.principal"="hive/[email protected]", diff --git a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy index 62771e46ebc..4055a5aaa0c 100644 --- a/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy +++ b/regression-test/suites/external_table_p2/refactor_catalog_param/hive_on_hms_and_dlf.groovy @@ -421,11 +421,12 @@ suite("hive_on_hms_and_dlf", "p2,external,new_catalog_property") { String hdfs_properties = """ "hadoop.username" = "doris" """ - // kerberos + // Both Kerberos property forms must match the DataNode SASL protection to allow writes. String hdfs_kerberos_properties = """ "fs.defaultFS" = "hdfs://${externalEnvIp}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "hadoop.security.authentication" = "kerberos", @@ -436,6 +437,7 @@ suite("hive_on_hms_and_dlf", "p2,external,new_catalog_property") { "fs.defaultFS" = "hdfs://${externalEnvIp}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "hdfs.authentication.type" = "kerberos", "hdfs.authentication.kerberos.principal"="hive/[email protected]", diff --git a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy index fa3cff25572..98ab7ad6039 100644 --- a/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy +++ b/regression-test/suites/external_table_p2/refactor_catalog_param/iceberg_on_hms_and_filesystem_and_dlf.groovy @@ -474,11 +474,12 @@ suite("iceberg_on_hms_and_filesystem_and_dlf", "p2,external,new_catalog_property String hdfs_properties = """ "hadoop.username" = "doris" """ - // kerberos + // Both Kerberos property forms must match the DataNode SASL protection to allow writes. String hdfs_kerberos_properties = """ "fs.defaultFS" = "hdfs://${externalEnvIp}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "hadoop.security.authentication" = "kerberos", "io-impl" = "org.apache.doris.datasource.iceberg.fileio.DelegateFileIO", @@ -490,6 +491,7 @@ suite("iceberg_on_hms_and_filesystem_and_dlf", "p2,external,new_catalog_property "fs.defaultFS" = "hdfs://${externalEnvIp}:8520", "dfs.namenode.kerberos.principal" = "hdfs/[email protected]", "dfs.client.use.datanode.hostname" = "true", + "dfs.data.transfer.protection" = "authentication", "hadoop.security.token.service.use_ip" = "false", "io-impl" = "org.apache.doris.datasource.iceberg.fileio.DelegateFileIO", "hdfs.authentication.type" = "kerberos", --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
