github-actions[bot] commented on code in PR #65564: URL: https://github.com/apache/doris/pull/65564#discussion_r3578691748
########## docker/thirdparties/docker-compose/kerberos/conf/hdfs-site.xml.tpl: ########## @@ -0,0 +1,113 @@ +<?xml version="1.0"?> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<configuration> + <property> + <name>dfs.namenode.name.dir</name> + <value>file:///data/hdfs/name</value> + </property> + <property> + <name>dfs.datanode.data.dir</name> + <value>file:///data/hdfs/data</value> + </property> + <property> + <name>dfs.replication</name> + <value>1</value> + </property> + <property> + <name>dfs.permissions.enabled</name> + <value>false</value> + </property> + <property> + <name>dfs.namenode.datanode.registration.ip-hostname-check</name> + <value>false</value> + </property> + <property> + <name>dfs.block.access.token.enable</name> Review Comment: The new image now runs HDFS in secure mode, but only the containers receive the rendered `core-site.xml`/`hdfs-site.xml`. After `docker compose up --wait`, the runner publishes just `doris-krb5.conf` to `/keytabs/krb5.conf`, and the external FE/BE configs point only at that file. The Doris-side properties in the Kerberos suites provide a client principal/keytab and `fs.defaultFS`, but they never pass the HDFS service-principal/client settings that this template now requires, such as `dfs.namenode.kerberos.principal` and the hostname/token-service settings from `core-site.xml`. The container health check can still pass because it uses `/opt/doris/conf` inside the container, while FE/BE HDFS operations such as the HMS catalog read/write, EXPORT/OUTFILE, HDFS TVF, and Iceberg Hadoop catalog run from the host Doris processes without those rendered configs. Please either expose the rendered client `core-site.xml`/`hdfs-site.xml` through `/keytabs` and set `hdfs.config.resources` or equ ivalent for the tests, or add the needed per-test HDFS service-principal settings for both realms. ########## docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh: ########## @@ -15,83 +15,112 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. + set -euo pipefail -source /usr/local/common/hive-configure.sh -source /usr/local/common/event-hook.sh - -echo "Configuring hive" -configure /etc/hive/conf/hive-site.xml hive HIVE_SITE_CONF -configure /etc/hive/conf/hiveserver2-site.xml hive HIVE_SITE_CONF -configure /etc/hadoop/conf/core-site.xml core CORE_CONF -configure /etc/hadoop/conf/hdfs-site.xml hdfs HDFS_CONF -configure /etc/hadoop/conf/yarn-site.xml yarn YARN_CONF -configure /etc/hadoop/conf/mapred-site.xml mapred MAPRED_CONF -configure /etc/hive/conf/beeline-site.xml beeline BEELINE_SITE_CONF - -echo "Copying kerberos keytabs to keytabs/" -mkdir -p /etc/hadoop-init.d/ - -if [ "$1" == "1" ]; then - cp /etc/trino/conf/* /keytabs/ -elif [ "$1" == "2" ]; then - cp /etc/trino/conf/hive-presto-master.keytab /keytabs/other-hive-presto-master.keytab - cp /etc/trino/conf/presto-server.keytab /keytabs/other-presto-server.keytab -else - echo "Invalid index parameter. Exiting." - exit 1 -fi -cd /usr/hdp/3.1.0.0-78/hive/auxlib -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/jdom-1.1.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/aliyun-java-sdk-core-3.4.0.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/aliyun-java-sdk-ecs-4.2.0.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/aliyun-java-sdk-ram-3.0.0.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/aliyun-java-sdk-sts-3.0.0.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/jindo-core-6.3.4.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/jindo-core-linux-el7-aarch64-6.3.4.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/jindo-sdk-6.3.4.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/aws-java-sdk-bundle-1.11.375.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/hadoop-huaweicloud-3.1.1-hw-54.5.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/hadoop-cos-3.1.0-8.3.22.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/cos_api-bundle-5.6.244.4.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/hadoop-aws-3.2.1.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/paimon-hive-connector-3.1-1.3-SNAPSHOT.jar -curl -O https://s3BucketName.s3Endpoint/regression/docker/hive3/gcs-connector-hadoop3-2.2.24-shaded.jar - - -/usr/local/hadoop-run.sh & - -# check healthy hear -echo "Waiting for hadoop to be healthy" - -for i in {1..60}; do - if /usr/local/health.sh; then - echo "Hadoop is healthy" + +: "${HOST:?}" +: "${REALM:?}" +: "${KDC_PORT:?}" +: "${FS_PORT:?}" +: "${HMS_PORT:?}" +: "${HIVE_CLIENT_KEYTAB:?}" +: "${PRESTO_CLIENT_KEYTAB:?}" + +export HADOOP_CONF_DIR=/opt/doris/conf +export HIVE_CONF_DIR=/opt/doris/conf +export KRB5_KDC_PROFILE=/opt/doris/conf/kdc.conf +export PATH="/usr/local/sbin:/usr/sbin:${PATH}" +export HADOOP_CLIENT_OPTS="-Xms192m -Xmx320m" + +readonly HDFS_PRINCIPAL="hdfs/${HOST}@${REALM}" +readonly HTTP_PRINCIPAL="HTTP/${HOST}@${REALM}" +readonly HIVE_PRINCIPAL="hive/${HOST}@${REALM}" +readonly HIVE_CLIENT_PRINCIPAL="hive/presto-master.docker.cluster@${REALM}" +readonly PRESTO_CLIENT_PRINCIPAL="presto-server/presto-master.docker.cluster@${REALM}" + +declare -a SERVICE_PIDS=() + +stop_services() { + kill "${SERVICE_PIDS[@]}" 2>/dev/null || true + wait "${SERVICE_PIDS[@]}" 2>/dev/null || true +} + +trap stop_services EXIT INT TERM + +start_service() { + "$@" & + SERVICE_PIDS+=("$!") +} + +wait_for_port() { + local host=$1 + local port=$2 + local service=$3 + + for _ in {1..120}; do + if (exec 3<>"/dev/tcp/${host}/${port}") 2>/dev/null; then + exec 3>&- + exec 3<&- + return + fi + sleep 1 + done + + echo "Timed out waiting for ${service} on ${host}:${port}" >&2 + return 1 +} + +create_keytab() { + local principal=$1 + local keytab=$2 + + kadmin.local -r "${REALM}" -q "addprinc -randkey ${principal}" + kadmin.local -r "${REALM}" -q "ktadd -k ${keytab} ${principal}" +} + +mkdir -p /data/hdfs/data /data/hdfs/name /data/kdc /data/keytabs /data/metastore /keytabs +rm -rf /tmp/hadoop-server-conf +cp -R "${HADOOP_CONF_DIR}" /tmp/hadoop-server-conf +sed -i "s#hdfs://${HOST}:${FS_PORT}#hdfs://127.0.0.1:${FS_PORT}#" \ + /tmp/hadoop-server-conf/core-site.xml + +kdb5_util create -s -r "${REALM}" -P doris-kerberos-test +create_keytab "${HDFS_PRINCIPAL}" /data/keytabs/hdfs.keytab +create_keytab "${HTTP_PRINCIPAL}" /data/keytabs/spnego.keytab +create_keytab "${HIVE_PRINCIPAL}" /data/keytabs/hive.keytab +create_keytab "${HIVE_CLIENT_PRINCIPAL}" "/keytabs/${HIVE_CLIENT_KEYTAB}" +create_keytab "${PRESTO_CLIENT_PRINCIPAL}" "/keytabs/${PRESTO_CLIENT_KEYTAB}" +chmod 644 /keytabs/*.keytab + +start_service krb5kdc -n -r "${REALM}" +wait_for_port 127.0.0.1 "${KDC_PORT}" "Kerberos KDC" + +export HDFS_NAMENODE_OPTS="-Xms128m -Xmx256m" +export HDFS_DATANODE_OPTS="-Xms96m -Xmx192m" +hdfs namenode -format -force -nonInteractive +start_service hdfs namenode +wait_for_port "${HOST}" "${FS_PORT}" "HDFS NameNode" +start_service env HADOOP_CONF_DIR=/tmp/hadoop-server-conf hdfs datanode +wait_for_port "${HOST}" "${DFS_DN_PORT}" "HDFS DataNode" + +export KRB5CCNAME=FILE:/tmp/hdfs-admin.ccache +kinit -kt /data/keytabs/hdfs.keytab "${HDFS_PRINCIPAL}" +for _ in {1..120}; do + if hdfs dfsadmin -Dfs.defaultFS="hdfs://127.0.0.1:${FS_PORT}" -report 2>/dev/null \ + | grep -q '^Live datanodes (1):'; then break fi - echo "Hadoop is not healthy yet. Retrying in 60 seconds..." - sleep 5 + sleep 1 done +hdfs dfsadmin -Dfs.defaultFS="hdfs://127.0.0.1:${FS_PORT}" -report \ + | grep -q '^Live datanodes (1):' +kdestroy + +schematool -dbType derby -initSchema Review Comment: This removes the old Paimon HMS bootstrap but does not replace it. Previously the Kerberos entrypoint could copy `paimon_data` into HDFS and run `create_paimon_hive_table.hql`, which registered `hdfs_db.external_test_table`. After this change the new entrypoint only initializes the Derby schema and starts the metastore, and the `paimon_data` mounts plus the HQL file are gone. However `test_paimon_hms_catalog.groovy` still runs the Kerberos HMS variants against `hdfs_db` and selects `external_test_table`, and its expected output still has the `hdfs_kerberos`/`support_hdfs_kerberos` rows. When `enablePaimonTest` runs with this environment, those cases no longer have a table or data to read. Please keep an equivalent Paimon bootstrap in the new environment, or move creation/import of `hdfs_db.external_test_table` into that suite before the Kerberos HMS `testQuery` calls. ########## docker/thirdparties/docker-compose/kerberos/hadoop-hive.env.tpl: ########## @@ -1,95 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 # -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -HIVE_SITE_CONF_javax_jdo_option_ConnectionURL=jdbc:mysql://127.0.0.1:${MYSQL_PORT}/metastore -HIVE_SITE_CONF_javax_jdo_option_ConnectionDriverName=com.mysql.jdbc.Driver -HIVE_SITE_CONF_javax_jdo_option_ConnectionUserName=root -HIVE_SITE_CONF_javax_jdo_option_ConnectionPassword=root -HIVE_SITE_CONF_datanucleus_autoCreateSchema=false -HIVE_SITE_CONF_hive_metastore_port=${HMS_PORT} -HIVE_SITE_CONF_hive_metastore_uris=thrift://${IP_HOST}:${HMS_PORT} -HIVE_SITE_CONF_hive_server2_thrift_bind_host=0.0.0.0 -HIVE_SITE_CONF_hive_server2_thrift_port=${HS_PORT} -HIVE_SITE_CONF_hive_server2_webui_port=0 -HIVE_SITE_CONF_hive_compactor_initiator_on=true -HIVE_SITE_CONF_hive_compactor_worker_threads=2 -HIVE_SITE_CONF_metastore_storage_schema_reader_impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader -BEELINE_SITE_CONF_beeline_hs2_jdbc_url_tcpUrl=jdbc:hive2://${HOST}:${HS_PORT}/default;user=hdfs;password=hive -BEELINE_SITE_CONF_beeline_hs2_jdbc_url_httpUrl=jdbc:hive2://${HOST}:${HS_PORT}/default;user=hdfs;password=hive - - -CORE_CONF_fs_defaultFS=hdfs://${HOST}:${FS_PORT} -CORE_CONF_hadoop_http_staticuser_user=root -CORE_CONF_hadoop_proxyuser_hue_hosts=* -CORE_CONF_hadoop_proxyuser_hue_groups=* - -HDFS_CONF_dfs_webhdfs_enabled=true -HDFS_CONF_dfs_permissions_enabled=false -HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false -HDFS_CONF_dfs_datanode_address=${HOST}:${DFS_DN_PORT} -HDFS_CONF_dfs_datanode_http_address=${HOST}:${DFS_DN_HTTP_PORT} -HDFS_CONF_dfs_datanode_ipc_address=${HOST}:${DFS_DN_IPC_PORT} -HDFS_CONF_dfs_namenode_http___address=${HOST}:${DFS_NN_HTTP_PORT} -YARN_CONF_yarn_log___aggregation___enable=true -YARN_CONF_yarn_resourcemanager_recovery_enabled=true -YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore -YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate -YARN_CONF_yarn_nodemanager_remote___app___log___dir=/var/log/hadoop-yarn/apps -YARN_CONF_yarn_log_server_url=http://${HOST}:${YARM_LOG_SERVER_PORT}/jobhistory/logs -YARN_CONF_yarn_timeline___service_enabled=false -YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true -YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true -YARN_CONF_yarn_resourcemanager_hostname=${HOST} -MAPRED_CONF_mapreduce_shuffle_port=${MAPREDUCE_SHUFFLE_PORT} -YARN_CONF_yarn_timeline___service_hostname=${HOST} -YARN_CONF_yarn_resourcemanager_address=${HOST}:${YARN_RM_PORT} -YARN_CONF_yarn_resourcemanager_scheduler_address=${HOST}:${YARN_RM_SCHEDULER_PORT} -YARN_CONF_yarn_resourcemanager_resource___tracker_address=${HOST}:${YARN_RM_TRACKER_PORT} -YARN_CONF_yarn_resourcemanager_admin_address=${HOST}:${YARN_RM_ADMIN_PORT} -YARN_CONF_yarn_resourcemanager_webapp_address=${HOST}:${YARN_RM_WEBAPP_PORT} -YARN_CONF_yarn_nodemanager_localizer_address=${HOST}:${YARN_NM_LOCAL_PORT} -YARN_CONF_yarn_nodemanager_webapp_address=${HOST}:${YARN_NM_WEBAPP_PORT} +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. -HIVE_SITE_CONF_fs_s3_impl=org.apache.hadoop.fs.s3a.S3AFileSystem -HIVE_SITE_CONF_fs_s3a_impl=org.apache.hadoop.fs.s3a.S3AFileSystem -HIVE_SITE_CONF_fs_s3a_access_key=${AWSAk} -HIVE_SITE_CONF_fs_s3a_secret_key=${AWSSk} -HIVE_SITE_CONF_fs_s3a_endpoint=${AWSEndpoint} -HIVE_SITE_CONF_fs_AbstractFileSystem_obs_impl=org.apache.hadoop.fs.obs.OBS -HIVE_SITE_CONF_fs_obs_access_key=${OBSAk} -HIVE_SITE_CONF_fs_obs_secret_key=${OBSSk} -HIVE_SITE_CONF_fs_obs_endpoint=${OBSEndpoint} -HIVE_SITE_CONF_fs_cosn_credentials_provider=org.apache.hadoop.fs.auth.SimpleCredentialProvider -HIVE_SITE_CONF_fs_cosn_userinfo_secretId=${COSAk} -HIVE_SITE_CONF_fs_cosn_userinfo_secretKey=${COSSk} -HIVE_SITE_CONF_fs_cosn_bucket_region=${COSRegion} -HIVE_SITE_CONF_fs_cosn_impl=org.apache.hadoop.fs.CosFileSystem -HIVE_SITE_CONF_fs_cos_impl=org.apache.hadoop.fs.CosFileSystem -HIVE_SITE_CONF_fs_AbstractFileSystem_cosn_impl=org.apache.hadoop.fs.CosN -HIVE_SITE_CONF_fs_oss_impl=com.aliyun.jindodata.oss.JindoOssFileSystem -HIVE_SITE_CONF_fs_AbstractFileSystem_oss_impl=com.aliyun.jindodata.oss.JindoOSS -HIVE_SITE_CONF_fs_oss_accessKeyId=${OSSAk} -HIVE_SITE_CONF_fs_oss_accessKeySecret=${OSSSk} -HIVE_SITE_CONF_fs_oss_endpoint=${OSSEndpoint} -HIVE_SITE_CONF_fs_AbstractFileSystem_gs_impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS -HIVE_SITE_CONF_fs_gs_project_id=${GCSProjectId} -HIVE_SITE_CONF_google_cloud_auth_service_account_enable=true -HIVE_SITE_CONF_fs_gs_auth_service_account_email=${GCSAccountEmail} -HIVE_SITE_CONF_fs_gs_auth_service_account_private_key_id=${GCSAccountPrivateKeyId} -HIVE_SITE_CONF_fs_gs_auth_service_account_private_key=${GCSAccountPrivateKey} -HIVE_SITE_CONF_fs_gs_proxy_address=${GCSProxyAddress} -enablePaimonHms=${enablePaimonHms} \ No newline at end of file +HOST=${HOST} Review Comment: The old Kerberos HMS image did more than start HMS: it also downloaded the object-store filesystem jars into Hive's auxlib and generated `HIVE_SITE_CONF_fs_*` settings for S3A, OBS, COS/COSN, OSS, and GCS. This replacement env now only carries Kerberos/port/keytab paths, and the new Dockerfile installs only Kerberos packages on top of `apache/hive:3.1.3`, so the HMS server no longer has those filesystem implementations or credentials. That breaks unchanged Kerberized HMS object-store cases such as `hive_on_hms_and_dlf.groovy`, which still creates HMS databases at `obs://`, `oss://`, `cosn://`/`cos://`, and optional `gs://` locations via the Kerberos metastore, and the Iceberg HMS suite keeps the same Kerberos object-store variants. Please restore the connector jars and Hive/Hadoop `fs.*` settings for the schemes still covered by Kerberos HMS tests, or remove/retarget those Kerberos HMS object-store cases so they no longer depend on the container-side HMS filesystem support. ########## regression-test/suites/external_table_p0/kerberos/test_single_hive_kerberos.groovy: ########## @@ -48,6 +48,24 @@ suite("test_single_hive_kerberos", "p0,external") { ); """ sql """ switch hms_kerberos """ + sql """ CREATE DATABASE IF NOT EXISTS test_krb_hive_db """ + sql """ USE test_krb_hive_db """ + sql """ DROP TABLE IF EXISTS test_krb_hive_tbl """ + sql """ + CREATE TABLE test_krb_hive_tbl ( Review Comment: The removed container-side fixture created this table explicitly as Parquet (`ParquetHiveSerDe` plus `MapredParquetInputFormat`). The replacement here creates a Hive table through Doris without any `file_format` property, so `HiveMetadataOps` falls back to `Config.hive_default_file_format`, which is `orc`. That means the Kerberos read path no longer exercises the same Hive-created Parquet table that the old environment provided; it exercises a Doris-created ORC table instead. If the goal is an equivalent fixture, please add `PROPERTIES ("file_format" = "parquet")` here and in the matching two-Hive fixture, or keep an explicit Parquet bootstrap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
