abstractdog commented on code in PR #6640: URL: https://github.com/apache/hive/pull/6640#discussion_r4045179225
########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> Review Comment: need a more generic name here, this is a yarn cluster, not only suitable for localization tests ``` Hive Integration - Tez on YARN integration tests ``` ########## itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/ComposeImageSubstitutor.java: ########## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hive.tez.yarn; + +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.ImageNameSubstitutor; + +/** Replaces Testcontainers' pinned docker:24.0.2 compose helper with a newer CLI for modern daemons. + * Registered via testcontainers.properties; override with -Dtez.yarn.compose.image=docker:tag. */ +public class ComposeImageSubstitutor extends ImageNameSubstitutor { + + private static final String REPLACEMENT_IMAGE = + System.getProperty("tez.yarn.compose.image", "docker:27.5.1"); Review Comment: do we need to pin exact minor version? isn't `docker:27` suitable? ########## itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/ComposeImageSubstitutor.java: ########## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hive.tez.yarn; + +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.ImageNameSubstitutor; + +/** Replaces Testcontainers' pinned docker:24.0.2 compose helper with a newer CLI for modern daemons. Review Comment: doesn't mentioning `24.0.2` explicitly go out-of-date over time? ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/Dockerfile: ########## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Custom Hadoop image for the Tez-on-YARN localization integration tests. Review Comment: more generic, not only suitable for "localization" ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> Review Comment: this one should be skipped if tests are not running: ``` <configuration> <skip>${skipTests}</skip> ... ``` ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.tez</includeGroupIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + <execution> + <id>stage-tez-mr-client-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.hadoop,commons-collections</includeGroupIds> + <includeArtifactIds>hadoop-mapreduce-client-core,hadoop-mapreduce-client-common,commons-collections</includeArtifactIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <id>assemble-tez-dist</id> Review Comment: this one should be skipped if tests are not running: ``` <configuration> <skip>${skipTests}</skip> ... ``` it can be costly, no need to run if it's just test compilation ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.tez</includeGroupIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + <execution> + <id>stage-tez-mr-client-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.hadoop,commons-collections</includeGroupIds> + <includeArtifactIds>hadoop-mapreduce-client-core,hadoop-mapreduce-client-common,commons-collections</includeArtifactIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <id>assemble-tez-dist</id> + <!-- Must run after the dependency-copy executions in process-test-resources. --> + <phase>test-compile</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <tar destfile="${project.build.directory}/tez-libs.tar.gz" compression="gzip"> + <tarfileset dir="${project.build.directory}/tez-stage" includes="*.jar"/> + </tar> + </target> + </configuration> + </execution> + <execution> + <id>generate-custom-hosts-file</id> Review Comment: this one should be skipped if tests are not running: ``` <configuration> <skip>${skipTests}</skip> ... ``` ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest + hostname: namenode + command: ["hdfs", "namenode"] + env_file: + - config + environment: + ENSURE_NAMENODE_DIR: /tmp/hadoop-hadoop/dfs/name + ports: + - "8020:8020" + - "9870:9870" + healthcheck: + test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9870'"] + interval: 5s + timeout: 3s + retries: 30 + start_period: 10s + + datanode: + image: hive-it-hadoop-jdk21:latest + hostname: datanode + command: ["hdfs", "datanode"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "9866:9866" + - "9864:9864" Review Comment: comment for which port is what ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> Review Comment: it might need some slf4j hacking, but I can imagine that using system.out, some details about the cluster can be displayed after the surefire message: ``` [INFO] Running org.apache.hive.tez.yarn.StartTezYarnCluster ``` like: ``` ====================================================================== Tez-on-YARN cluster is up. Ctrl-C this JVM to tear it down. ====================================================================== HDFS URI hdfs://namenode:8020 NameNode web UI http://localhost:9870 ResourceManager RPC resourcemanager:8032 ResourceManager UI http://localhost:8088 NodeManager address nodemanager:41000 Tez libs on HDFS hdfs://namenode:8020/tmp/hive-tez-yarn/tez-libs.tar.gz#tez HDFS warehouse hdfs://namenode:8020/tmp/hive-tez-loc/warehouse HiveServer2 JDBC jdbc:hive2://localhost:10000/default;auth=noSasl Log file (slf4j) /Users/lbodor/apache/hive/itests/tez-yarn-it/target/...log ====================================================================== Beeline: beeline -u 'jdbc:hive2://localhost:10000/default' -n hive ====================================================================== ``` while all the other messages go to the log file under target as it's going now ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and +task containers — a scenario that in-process mini-clusters (MiniTezCluster, MiniDFSCluster) Review Comment: mention the exact root cause why mini clusters don't exercise that code path as a motivation for this integration testing mechanism ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and +task containers — a scenario that in-process mini-clusters (MiniTezCluster, MiniDFSCluster) +do not exercise. + +## Prerequisites + +* Java 21 +* Maven 3.6.3 or later +* Docker Desktop (or Docker Engine) with at least **4 GB** of memory assigned + +Docker Desktop 25+ exposes a newer Docker API than Testcontainers' default helper image +expects. This module ships `ComposeImageSubstitutor` and `testcontainers.properties` to +substitute a compatible helper image automatically; no extra configuration is required on +supported setups. + +## First-time setup + +Build the full Hive distribution once to populate `$HIVE_HOME/lib/` and install all +artifacts to `~/.m2`: + +```bash +mvn clean install -DskipTests -Pitests,dist +``` + +## Running the automated tests + +This module is **opt-in**: tests are skipped by default. You must pass `-Pitests,tez-yarn` +(or `-Drun.tez.yarn.tests=true`) for Surefire to execute anything. Running +`mvn test -pl itests/tez-yarn-it` without those flags reports zero tests by design. + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it +``` + +If you have not run a full install recently, add `-am` to build required upstream modules: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -am +``` + +To run a single test: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar +``` + +## Starting a keep-alive cluster for manual testing + +`StartTezYarnCluster` starts the full HDFS + YARN + HiveServer2 stack and blocks until +`Ctrl+C`. An optional port can be passed via `-Dtez.yarn.cluster.hs2.port` (default: 10000). + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=StartTezYarnCluster \ + -Dtez.yarn.cluster.run=true +``` + +Once ready, the log prints the JDBC URL and Beeline command. Connect from a second terminal: + +```bash +beeline -u 'jdbc:hive2://localhost:10000/default;auth=noSasl' -n hive Review Comment: why is `auth=noSasl` needed? ########## itests/tez-yarn-it/src/test/java/org/apache/hive/tez/yarn/ComposeImageSubstitutor.java: ########## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.hive.tez.yarn; + +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.ImageNameSubstitutor; + +/** Replaces Testcontainers' pinned docker:24.0.2 compose helper with a newer CLI for modern daemons. + * Registered via testcontainers.properties; override with -Dtez.yarn.compose.image=docker:tag. */ Review Comment: format javadoc, it should be written like: ``` /** * First sentence. * Second sentence. */ ``` double-check for the same ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and Review Comment: `to verify that hive-exec.jar is correctly localized for Tez AM and task containers` be more generic here also: for future readers and users, this module can be used as a "real" mini hadoop cluster, not only for localization testing ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and +task containers — a scenario that in-process mini-clusters (MiniTezCluster, MiniDFSCluster) +do not exercise. + +## Prerequisites + +* Java 21 +* Maven 3.6.3 or later +* Docker Desktop (or Docker Engine) with at least **4 GB** of memory assigned + +Docker Desktop 25+ exposes a newer Docker API than Testcontainers' default helper image +expects. This module ships `ComposeImageSubstitutor` and `testcontainers.properties` to +substitute a compatible helper image automatically; no extra configuration is required on +supported setups. + +## First-time setup + +Build the full Hive distribution once to populate `$HIVE_HOME/lib/` and install all +artifacts to `~/.m2`: + +```bash +mvn clean install -DskipTests -Pitests,dist +``` + +## Running the automated tests + +This module is **opt-in**: tests are skipped by default. You must pass `-Pitests,tez-yarn` +(or `-Drun.tez.yarn.tests=true`) for Surefire to execute anything. Running +`mvn test -pl itests/tez-yarn-it` without those flags reports zero tests by design. + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it +``` + +If you have not run a full install recently, add `-am` to build required upstream modules: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -am +``` + +To run a single test: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar +``` + +## Starting a keep-alive cluster for manual testing + +`StartTezYarnCluster` starts the full HDFS + YARN + HiveServer2 stack and blocks until +`Ctrl+C`. An optional port can be passed via `-Dtez.yarn.cluster.hs2.port` (default: 10000). + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=StartTezYarnCluster \ + -Dtez.yarn.cluster.run=true +``` + +Once ready, the log prints the JDBC URL and Beeline command. Connect from a second terminal: + +```bash +beeline -u 'jdbc:hive2://localhost:10000/default;auth=noSasl' -n hive +``` + +Run a Tez-on-YARN query to exercise jar localization (including `INSERT ... VALUES`): + +```sql +CREATE TABLE test_tez (id INT, name STRING) STORED AS ORC; Review Comment: `set hive.fetch.task.conversion=none;` this way, the select will also use Tez ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then Review Comment: nit: `. The` two spaces before "The" ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the Review Comment: nit: `. This` multiple spaces before "This" ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> Review Comment: should inherit version from parent pomt ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> Review Comment: feature request: as an alternative for complicated staging, we need to make users able to download a tez.tar.gz or use a local one, I can imagine something like: ``` # download from Apache dist (or a mirror) mvn test -Ptez-yarn -Dtez.dist.source=download \ [-Dtez.dist.download.url=https://…] # reuse a tarball already on disk mvn test -Ptez-yarn -Dtez.dist.source=local \ -Dtez.dist.tarball=/absolute/path/to/tez.tar.gz ``` this makes Tez engineers (hi there!) happy, like: ``` cd ~/apache/tez && mvn -DskipTests clean install cd ~/apache/hive/itests/tez-yarn-it mvn test -Ptez-yarn -Dtez.dist.source=local \ -Dtez.dist.tarball=$HOME/apache/tez/tez-dist/target/tez-0.10.6-SNAPSHOT-minimal.tar.gz ``` if `tez.dist.source` is not defined, it can fallback to the default staging mechanism ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.tez</includeGroupIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + <execution> + <id>stage-tez-mr-client-jars</id> Review Comment: this one should be skipped if tests are not running: ``` <configuration> <skip>${skipTests}</skip> ... ``` ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.tez</includeGroupIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + <execution> + <id>stage-tez-mr-client-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.hadoop,commons-collections</includeGroupIds> + <includeArtifactIds>hadoop-mapreduce-client-core,hadoop-mapreduce-client-common,commons-collections</includeArtifactIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <id>assemble-tez-dist</id> + <!-- Must run after the dependency-copy executions in process-test-resources. --> + <phase>test-compile</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <tar destfile="${project.build.directory}/tez-libs.tar.gz" compression="gzip"> + <tarfileset dir="${project.build.directory}/tez-stage" includes="*.jar"/> + </tar> + </target> + </configuration> + </execution> + <execution> + <id>generate-custom-hosts-file</id> + <!-- Must run after the parent "setup-test-dirs" execution (process-test-resources). --> + <phase>test-compile</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <mkdir dir="${test.conf.dir}"/> + <copy file="${basedir}/src/test/resources/custom_hosts_file" tofile="${test.conf.dir}/custom_hosts_file" overwrite="true"/> + <exec executable="hostname" output="${test.conf.dir}/local_hostname.txt" failonerror="true"/> + <loadfile property="hive.tez.yarn.it.local.hostname" srcFile="${test.conf.dir}/local_hostname.txt"> + <filterchain> + <striplinebreaks/> + </filterchain> + </loadfile> + <!-- + When -Djdk.net.hosts.file is set, Java hostname resolution uses only that file. + Ensure the local machine hostname is resolvable so Hive can start/compile queries. + --> + <echo file="${test.conf.dir}/custom_hosts_file" append="true">127.0.0.1 ${hive.tez.yarn.it.local.hostname}</echo> + </target> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-surefire-plugin</artifactId> + <configuration> + <!-- Opt-in: -Ptez-yarn or -Drun.tez.yarn.tests=true --> Review Comment: I think this can be simplified further, and the `run.tez.yarn.tests` can be dropped completely ``` <properties> <hive.path.to.root>../..</hive.path.to.root> <!-- This module needs Docker; skip by default. Opt in with -Ptez-yarn, or override with the standard -DskipTests=false. Any -DskipTests on the CLI still wins. --> <skipTests>true</skipTests> </properties> <build> <plugins> <plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <argLine>@{argLine} -Dtez.dist.path=${project.build.directory}/tez-libs.tar.gz</argLine> <!-- no <skipTests> here — inherited from the property above --> </configuration> </plugin> </plugins> </build> <profiles> <profile> <id>tez-yarn</id> <properties> <skipTests>false</skipTests> </properties> </profile> </profiles> ``` ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest + hostname: namenode + command: ["hdfs", "namenode"] + env_file: + - config + environment: + ENSURE_NAMENODE_DIR: /tmp/hadoop-hadoop/dfs/name + ports: + - "8020:8020" + - "9870:9870" + healthcheck: + test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9870'"] + interval: 5s + timeout: 3s + retries: 30 + start_period: 10s + + datanode: + image: hive-it-hadoop-jdk21:latest + hostname: datanode + command: ["hdfs", "datanode"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "9866:9866" + - "9864:9864" + + resourcemanager: + image: hive-it-hadoop-jdk21:latest + hostname: resourcemanager + command: ["yarn", "resourcemanager"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "8032:8032" + - "8088:8088" Review Comment: comment for which port is what ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest + hostname: namenode + command: ["hdfs", "namenode"] + env_file: + - config + environment: + ENSURE_NAMENODE_DIR: /tmp/hadoop-hadoop/dfs/name + ports: + - "8020:8020" + - "9870:9870" + healthcheck: + test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9870'"] + interval: 5s + timeout: 3s + retries: 30 + start_period: 10s + + datanode: + image: hive-it-hadoop-jdk21:latest + hostname: datanode + command: ["hdfs", "datanode"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "9866:9866" + - "9864:9864" + + resourcemanager: + image: hive-it-hadoop-jdk21:latest + hostname: resourcemanager + command: ["yarn", "resourcemanager"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "8032:8032" + - "8088:8088" + # No healthcheck: RM webapp binds to hostname "resourcemanager", not localhost. + + nodemanager: + image: hive-it-hadoop-jdk21:latest + hostname: nodemanager + command: ["yarn", "nodemanager"] + env_file: + - config + depends_on: + resourcemanager: + condition: service_started + ports: + - "41000:41000" Review Comment: comment for which port is what ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest + hostname: namenode + command: ["hdfs", "namenode"] + env_file: + - config + environment: + ENSURE_NAMENODE_DIR: /tmp/hadoop-hadoop/dfs/name + ports: + - "8020:8020" + - "9870:9870" + healthcheck: + test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9870'"] + interval: 5s + timeout: 3s + retries: 30 + start_period: 10s + + datanode: + image: hive-it-hadoop-jdk21:latest + hostname: datanode + command: ["hdfs", "datanode"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "9866:9866" + - "9864:9864" + + resourcemanager: + image: hive-it-hadoop-jdk21:latest + hostname: resourcemanager + command: ["yarn", "resourcemanager"] + env_file: + - config + depends_on: + namenode: + condition: service_healthy + ports: + - "8032:8032" + - "8088:8088" + # No healthcheck: RM webapp binds to hostname "resourcemanager", not localhost. + + nodemanager: + image: hive-it-hadoop-jdk21:latest + hostname: nodemanager Review Comment: is there a ui for nodemanager? if so, it should be used and advised in the startup banner of `StartTezYarnCluster` ########## itests/tez-yarn-it/pom.xml: ########## @@ -0,0 +1,251 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it</artifactId> + <version>4.3.0-SNAPSHOT</version> + <relativePath>../pom.xml</relativePath> + </parent> + <artifactId>hive-it-tez-yarn</artifactId> + <packaging>jar</packaging> + <name>Hive Integration - Tez on YARN localization tests</name> + <properties> + <hive.path.to.root>../..</hive.path.to.root> + <run.tez.yarn.tests>false</run.tez.yarn.tests> + </properties> + <dependencies> + <dependency> + <groupId>org.testcontainers</groupId> + <artifactId>testcontainers</artifactId> + <scope>test</scope> + </dependency> + <!-- Explicit Tez dependencies: Hive's Tez compilation/execution path needs these classes + on the in-process HS2 JVM classpath. In the main Hive build they are often profile-gated. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-api</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-dag</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-library</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-runtime-internals</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>tez-mapreduce</artifactId> + <scope>test</scope> + </dependency> + <!-- Provides org.apache.tez.hadoop.shim.HadoopShimsLoader, which DAGAppMaster.serviceInit() + loads at Tez AM startup. Normally transitive via tez-api; declared explicitly so it is + reliably on the classpath of the in-process HiveServer2 JVM. --> + <dependency> + <groupId>org.apache.tez</groupId> + <artifactId>hadoop-shim</artifactId> + <version>${tez.version}</version> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-service</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-jdbc</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-it-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.awaitility</groupId> + <artifactId>awaitility</artifactId> + <version>4.2.1</version> + <scope>test</scope> + </dependency> + </dependencies> + <build> + <plugins> + <!-- Copy Tez framework jars (org.apache.tez group) and the Hadoop MR client jars + required by the Tez AM into target/tez-stage/. The antrun execution below then + packs them into a tez-libs.tar.gz that the integration tests upload to the + containerised HDFS cluster and reference via tez.lib.uris. This replaces the + previous approach of scanning the test JVM's classpath at runtime, which was + fragile and did not resemble real cluster setups. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <executions> + <execution> + <id>stage-tez-framework-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.tez</includeGroupIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + <execution> + <id>stage-tez-mr-client-jars</id> + <phase>process-test-resources</phase> + <goals> + <goal>copy-dependencies</goal> + </goals> + <configuration> + <outputDirectory>${project.build.directory}/tez-stage</outputDirectory> + <includeGroupIds>org.apache.hadoop,commons-collections</includeGroupIds> + <includeArtifactIds>hadoop-mapreduce-client-core,hadoop-mapreduce-client-common,commons-collections</includeArtifactIds> + <excludeClassifiers>tests</excludeClassifiers> + <stripVersion>false</stripVersion> + <overWriteIfNewer>true</overWriteIfNewer> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <executions> + <execution> + <id>assemble-tez-dist</id> + <!-- Must run after the dependency-copy executions in process-test-resources. --> + <phase>test-compile</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <tar destfile="${project.build.directory}/tez-libs.tar.gz" compression="gzip"> + <tarfileset dir="${project.build.directory}/tez-stage" includes="*.jar"/> + </tar> + </target> + </configuration> + </execution> + <execution> + <id>generate-custom-hosts-file</id> + <!-- Must run after the parent "setup-test-dirs" execution (process-test-resources). --> + <phase>test-compile</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <mkdir dir="${test.conf.dir}"/> + <copy file="${basedir}/src/test/resources/custom_hosts_file" tofile="${test.conf.dir}/custom_hosts_file" overwrite="true"/> + <exec executable="hostname" output="${test.conf.dir}/local_hostname.txt" failonerror="true"/> + <loadfile property="hive.tez.yarn.it.local.hostname" srcFile="${test.conf.dir}/local_hostname.txt"> + <filterchain> + <striplinebreaks/> + </filterchain> + </loadfile> + <!-- + When -Djdk.net.hosts.file is set, Java hostname resolution uses only that file. + Ensure the local machine hostname is resolvable so Hive can start/compile queries. + --> + <echo file="${test.conf.dir}/custom_hosts_file" append="true">127.0.0.1 ${hive.tez.yarn.it.local.hostname}</echo> + </target> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-surefire-plugin</artifactId> + <configuration> + <!-- Opt-in: -Ptez-yarn or -Drun.tez.yarn.tests=true --> + <skipTests>true</skipTests> + <!-- Resolve docker-network hostnames (namenode, resourcemanager, …) to 127.0.0.1 + so HDFS URIs embedded in Tez LocalResources are reachable from both the host JVM + and inside YARN containers (via the shared docker network alias). --> + <argLine>${maven.test.jvm.args} -Djdk.net.hosts.file=${test.conf.dir}/custom_hosts_file -Dtez.dist.path=${project.build.directory}/tez-libs.tar.gz</argLine> Review Comment: what about inheriting the parent: ``` <argLine>@{argLine} ...</argLine> ``` ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest + hostname: namenode + command: ["hdfs", "namenode"] + env_file: + - config + environment: + ENSURE_NAMENODE_DIR: /tmp/hadoop-hadoop/dfs/name + ports: + - "8020:8020" + - "9870:9870" Review Comment: comment for which port is what ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and +task containers — a scenario that in-process mini-clusters (MiniTezCluster, MiniDFSCluster) +do not exercise. + +## Prerequisites + +* Java 21 +* Maven 3.6.3 or later +* Docker Desktop (or Docker Engine) with at least **4 GB** of memory assigned + +Docker Desktop 25+ exposes a newer Docker API than Testcontainers' default helper image +expects. This module ships `ComposeImageSubstitutor` and `testcontainers.properties` to +substitute a compatible helper image automatically; no extra configuration is required on +supported setups. + +## First-time setup + +Build the full Hive distribution once to populate `$HIVE_HOME/lib/` and install all +artifacts to `~/.m2`: + +```bash +mvn clean install -DskipTests -Pitests,dist +``` + +## Running the automated tests + +This module is **opt-in**: tests are skipped by default. You must pass `-Pitests,tez-yarn` +(or `-Drun.tez.yarn.tests=true`) for Surefire to execute anything. Running +`mvn test -pl itests/tez-yarn-it` without those flags reports zero tests by design. + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it +``` + +If you have not run a full install recently, add `-am` to build required upstream modules: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -am +``` + +To run a single test: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar +``` + +## Starting a keep-alive cluster for manual testing + +`StartTezYarnCluster` starts the full HDFS + YARN + HiveServer2 stack and blocks until +`Ctrl+C`. An optional port can be passed via `-Dtez.yarn.cluster.hs2.port` (default: 10000). + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=StartTezYarnCluster \ + -Dtez.yarn.cluster.run=true +``` + +Once ready, the log prints the JDBC URL and Beeline command. Connect from a second terminal: + +```bash +beeline -u 'jdbc:hive2://localhost:10000/default;auth=noSasl' -n hive +``` + +Run a Tez-on-YARN query to exercise jar localization (including `INSERT ... VALUES`): + +```sql +CREATE TABLE test_tez (id INT, name STRING) STORED AS ORC; +INSERT INTO test_tez VALUES (1, 'hello'), (2, 'world'); +SELECT * FROM test_tez; +``` + +## Troubleshooting Review Comment: maybe manual docker-compose command to tear down hadoop cluster if it's left over by any chance ########## itests/tez-yarn-it/README.md: ########## @@ -0,0 +1,119 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Tez-on-YARN localization integration test + +An opt-in integration test module that runs Hive-on-Tez against a real Docker-containerized +HDFS + YARN cluster to verify that `hive-exec.jar` is correctly localized for Tez AM and +task containers — a scenario that in-process mini-clusters (MiniTezCluster, MiniDFSCluster) +do not exercise. + +## Prerequisites + +* Java 21 +* Maven 3.6.3 or later +* Docker Desktop (or Docker Engine) with at least **4 GB** of memory assigned + +Docker Desktop 25+ exposes a newer Docker API than Testcontainers' default helper image +expects. This module ships `ComposeImageSubstitutor` and `testcontainers.properties` to +substitute a compatible helper image automatically; no extra configuration is required on +supported setups. + +## First-time setup + +Build the full Hive distribution once to populate `$HIVE_HOME/lib/` and install all +artifacts to `~/.m2`: + +```bash +mvn clean install -DskipTests -Pitests,dist +``` + +## Running the automated tests + +This module is **opt-in**: tests are skipped by default. You must pass `-Pitests,tez-yarn` +(or `-Drun.tez.yarn.tests=true`) for Surefire to execute anything. Running +`mvn test -pl itests/tez-yarn-it` without those flags reports zero tests by design. + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it +``` + +If you have not run a full install recently, add `-am` to build required upstream modules: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -am +``` + +To run a single test: + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar +``` + +## Starting a keep-alive cluster for manual testing + +`StartTezYarnCluster` starts the full HDFS + YARN + HiveServer2 stack and blocks until +`Ctrl+C`. An optional port can be passed via `-Dtez.yarn.cluster.hs2.port` (default: 10000). + +```bash +mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it \ + -Dtest=StartTezYarnCluster \ + -Dtez.yarn.cluster.run=true +``` + +Once ready, the log prints the JDBC URL and Beeline command. Connect from a second terminal: Review Comment: update this to mention the banner on the standard output if it's really implemented in the meantime ########## itests/tez-yarn-it/src/test/docker/hadoop-yarn/docker-compose.yml: ########## @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# HDFS + YARN cluster for tez-yarn-it. Image built by TezYarnClusterContainer; config via env_file. + +services: + namenode: + image: hive-it-hadoop-jdk21:latest Review Comment: remove jdk21 from the image name: this can become a tag later if needed to distinguish between java versions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
