abstractdog commented on code in PR #473:
URL: https://github.com/apache/tez/pull/473#discussion_r3490405085
##########
tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java:
##########
@@ -2420,4 +2423,12 @@ static Set<String> getPropertySet() {
@ConfigurationScope(Scope.AM)
@ConfigurationProperty
public static final String TEZ_AM_STANDALONE_CONFS = TEZ_AM_PREFIX +
"standalone.confs";
+
+ /**
+ * String value. Namespace in ZooKeeper registry for the TezChild
+ */
+ @ConfigurationScope(Scope.VERTEX)
+ @ConfigurationProperty
+ public static final String TEZ_TASK_REGISTRY_NAMESPACE = TEZ_TASK_PREFIX +
"registry.namespace";
+ public static final String TEZ_TASK_REGISTRY_NAMESPACE_DEFAULT =
"/tez_am/workers";
Review Comment:
`/tez_am/workers` looks a bit strange, a component is an `am` or a `worker`
as the zookeeper registry hasn't been released yet, we're free to modify the
default values, what about:
for AM: `/tez-am`
for tasks: `/tez-worker`
so if we consider the root `/tez-external-sessions` prefix, they'll end up
in:
```
/tez-external-sessions/tez-am
/tez-external-sessions/tez-worker
```
this way we can stick to hyphen for the static prefixes, and underscore for
the dynamic container ids
additional changes needed: moving this below to `TezConstants` class and
making it public:
```
ZK_NAMESPACE_PREFIX = "/tez-external-sessions";
```
##########
tez-dag/src/main/java/org/apache/tez/dag/app/launcher/NoOpContainerLauncher.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tez.dag.app.launcher;
+
+import org.apache.tez.serviceplugins.api.ContainerLaunchRequest;
+import org.apache.tez.serviceplugins.api.ContainerLauncher;
+import org.apache.tez.serviceplugins.api.ContainerLauncherContext;
+import org.apache.tez.serviceplugins.api.ContainerStopRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A No-Op Container Launcher where TezChild processes are started externally
(e.g., via Docker
+ * Compose or Kubernetes).
+ */
+public class NoOpContainerLauncher extends ContainerLauncher {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NoOpContainerLauncher.class);
+
+ public NoOpContainerLauncher(ContainerLauncherContext
containerLauncherContext) {
+ super(containerLauncherContext);
+ }
+
+ /** This method is no-op it's just informing AM to change container state */
+ @Override
+ public void launchContainer(ContainerLaunchRequest launchRequest) {
+ LOG.info("Container is externally managed: {}",
launchRequest.getContainerId());
+
+ // Immediately tell AM that the container is "launched"
+ getContext().containerLaunched(launchRequest.getContainerId());
+ }
+
+ /** This method is no-op it's just informing AM to change container state */
+ @Override
+ public void stopContainer(ContainerStopRequest stopRequest) {
+ LOG.info("Lifecycle is externally managed for container: {}",
stopRequest.getContainerId());
Review Comment:
message should contain the actual fact also that this is a "stop" call
##########
tez-dag/src/main/java/org/apache/tez/dag/app/launcher/NoOpContainerLauncher.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tez.dag.app.launcher;
+
+import org.apache.tez.serviceplugins.api.ContainerLaunchRequest;
+import org.apache.tez.serviceplugins.api.ContainerLauncher;
+import org.apache.tez.serviceplugins.api.ContainerLauncherContext;
+import org.apache.tez.serviceplugins.api.ContainerStopRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A No-Op Container Launcher where TezChild processes are started externally
(e.g., via Docker
+ * Compose or Kubernetes).
+ */
+public class NoOpContainerLauncher extends ContainerLauncher {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NoOpContainerLauncher.class);
+
+ public NoOpContainerLauncher(ContainerLauncherContext
containerLauncherContext) {
+ super(containerLauncherContext);
+ }
+
+ /** This method is no-op it's just informing AM to change container state */
+ @Override
+ public void launchContainer(ContainerLaunchRequest launchRequest) {
+ LOG.info("Container is externally managed: {}",
launchRequest.getContainerId());
Review Comment:
message should contain the actual fact also that this is a "launch" call
##########
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/TezRuntimeUtils.java:
##########
@@ -271,6 +271,9 @@ public static BaseHttpConnection getHttpConnection(boolean
asyncHttp, URL url,
public static int deserializeShuffleProviderMetaData(ByteBuffer meta)
throws IOException {
+ if (meta == null) {
+ return 0;
+ }
Review Comment:
hm, need to know what is this a workaround for
##########
tez-dist/src/docker/entrypoint.sh:
##########
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -xeou pipefail
+
+: "${TEZ_COMPONENT:="AM"}"
+
+echo "--> Starting Tez Component: $TEZ_COMPONENT"
+
+if [[ "$TEZ_COMPONENT" == "AM" ]]; then
+ echo "--> Routing to Tez AM Entrypoint"
+ exec /am-entrypoint.sh "$@"
+elif [[ "$TEZ_COMPONENT" == "CHILD" ]]; then
+ echo "--> Routing to Tez Child Entrypoint"
+ exec /child-entrypoint.sh "$@"
Review Comment:
"child" might imply they are actually created out of a "parent" task, which
is not the case, as they are just simple, independent workers
currently, we usually refer to them as "task" containers
just thinking aloud: so before renaming every occurrence of "child" to
"task", we can also consider using "worker" consistently, especially because
the `TezChild` behavior is going to be altered
##########
tez-dist/src/docker/docker-compose.yml:
##########
@@ -142,3 +176,7 @@ volumes:
name: zookeeper_datalog
zookeeper_logs:
name: zookeeper_logs
+ tez_shared_data:
+ name: tez_shared_data
+ tez_shared_tmp:
+ name: tez_shared_tmp
Review Comment:
when the implementation is ready, please don't forget to comment here what's
supposed to be in tmp, what's in shared and what's local
also especially interested what's "tez_shared_tmp": it looks strange for the
first sight that containers share some temporary data: I believe, the naming
should be more specific to the use case than "tmp"
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
Review Comment:
this section could also be removed: the docker-compose file should make this
part clear
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+5. To override the tez-site.xml in docker image use:
+
+ * Set the `TEZ_CUSTOM_CONF_DIR` environment variable in `am.env` /
+ `child.env` or via the `docker run` command (e.g.,
+ `/opt/tez/custom-conf`).
+
+ ```bash
+ export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "$TEZ_SITE_PATH:/opt/tez/custom-conf/tez-site.xml" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
Review Comment:
this section can be replaced with a some tips pointing to a starter script:
```
export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml #this is a
path on the host machine that the developer can mess with :D
./start-tez.sh
```
this will make the testing with custom configuration extremely easy
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+5. To override the tez-site.xml in docker image use:
+
+ * Set the `TEZ_CUSTOM_CONF_DIR` environment variable in `am.env` /
+ `child.env` or via the `docker run` command (e.g.,
+ `/opt/tez/custom-conf`).
+
+ ```bash
+ export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "$TEZ_SITE_PATH:/opt/tez/custom-conf/tez-site.xml" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+6. To add plugin jars in docker image use:
+
+ * The plugin directory path inside the Docker container is fixed at
+ `/opt/tez/plugins`.
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "/path/to/your/local/plugins:/opt/tez/plugins" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+7. Using Docker Compose (Local Testing Cluster):
+
+ The provided `docker-compose.yml` offers a complete, minimal Hadoop
+ ecosystem to test Tez in a distributed manner locally without setting
+ up a real cluster.
+
+ **Services Included:**
+
+ * **namenode & datanode:** A minimal Apache Hadoop HDFS cluster
Review Comment:
even if I was in favor of using hadoop components earlier, know I believe we
should try to move away from `namenode`and `datanode`, and simply share folders
for the containers
I know this might need a re-design at some point, but it's worth the effort
e.g. hive's docker compose now runs a full cluster without any hadoop:
https://github.com/apache/hive/blob/9e38b8a3c58720899f413fe2531fed4138fb0a7f/packaging/src/docker/docker-compose.yml#L109-L110
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+5. To override the tez-site.xml in docker image use:
+
+ * Set the `TEZ_CUSTOM_CONF_DIR` environment variable in `am.env` /
+ `child.env` or via the `docker run` command (e.g.,
+ `/opt/tez/custom-conf`).
+
+ ```bash
+ export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "$TEZ_SITE_PATH:/opt/tez/custom-conf/tez-site.xml" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+6. To add plugin jars in docker image use:
+
+ * The plugin directory path inside the Docker container is fixed at
+ `/opt/tez/plugins`.
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "/path/to/your/local/plugins:/opt/tez/plugins" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+7. Using Docker Compose (Local Testing Cluster):
+
+ The provided `docker-compose.yml` offers a complete, minimal Hadoop
+ ecosystem to test Tez in a distributed manner locally without setting
+ up a real cluster.
+
+ **Services Included:**
+
+ * **namenode & datanode:** A minimal Apache Hadoop HDFS cluster
+ (lean image)
+
+ * **zookeeper:** Required by the Tez AM for standalone session
+ discovery
+
+ * **tez-am:** It automatically waits for Zookeeper and HDFS to
+ be healthy before starting up.
+
+ * **tez-child:** TBD
Review Comment:
this list of containers is not a must-have here, docker-compose must be
clean and self-descriptive
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
##########
@@ -517,45 +531,112 @@ public static TezChild newTezChild(Configuration conf,
String host, int port, St
hadoopShim);
}
- public static void main(String[] args) throws IOException,
InterruptedException, TezException {
+ public static void main(String[] args) throws Exception {
TezClassLoader.setupTezClassLoader();
final Configuration defaultConf = new Configuration();
+ String frameworkMode = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+
+ String host, appId, tokenIdentifier, containerIdentifier;
+ int port, attemptNumber;
+ Credentials credentials = new Credentials();
+
+ if (STANDALONE_ZOOKEEPER.name().equalsIgnoreCase(frameworkMode)) {
+ DAGProtos.ConfigurationProto confProtoBefore =
TezUtilsInternal.loadConfProtoFromText();
+ TezUtilsInternal.addUserSpecifiedTezConfiguration(
+ defaultConf, confProtoBefore.getConfKeyValuesList());
+
+ ZkAMRegistryClient registry = ZkAMRegistryClient.getClient(defaultConf);
+ registry.start();
+
+ // TODO: Expose retry counter and sleep time as config — in ZKconfig or
TezConfig?
+ while (!registry.isInitialized()) {
+ TimeUnit.SECONDS.sleep(5);
+ }
+
+ List<AMRecord> records = registry.getAllRecords();
+ if (records.isEmpty()) {
+ throw new RuntimeException("No AM found in ZooKeeper registry");
+ }
+ // TODO: Should we always get the first or there should be some
sophisticated logic?
+ AMRecord amRecord = records.getFirst();
+
+ host = amRecord.getHostName();
+ String portRange =
+ defaultConf.getTrimmed(TezConfiguration.TEZ_AM_TASK_AM_PORT_RANGE,
"12000-12000");
+ port = Integer.parseInt(portRange.split("[-,]")[0]);
+
+ appId = amRecord.getApplicationId().toString();
Review Comment:
this part is crucial from design perspective: the current implementation
here assumes the following behavior:
1. AM starts first
2. AM registers its application id in zookeeper
3. Task container starts
4. Task starts a registry client, and picks the first discoverable AM
let's assume a kubernetes cluster later, multiple tenants, so multiple
applications, and therefore concurrent DAGs
I'm afraid that with the current implementation, we simply lose our control
to start an application with the desired resources (so the desired amount of
task containers), because task containers just start and pick the first AM from
the zookeeper registry
instead, we should have a clear connection between them, which might be the
application_id/container_id (which is similar to yarn world tez container
mode), and when the task container starts, it already knows which application
it belongs to: I guess here this logical connection is around because you
already pass a container id for the TezChild
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
##########
@@ -517,45 +531,112 @@ public static TezChild newTezChild(Configuration conf,
String host, int port, St
hadoopShim);
}
- public static void main(String[] args) throws IOException,
InterruptedException, TezException {
+ public static void main(String[] args) throws Exception {
TezClassLoader.setupTezClassLoader();
final Configuration defaultConf = new Configuration();
+ String frameworkMode = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+
+ String host, appId, tokenIdentifier, containerIdentifier;
+ int port, attemptNumber;
+ Credentials credentials = new Credentials();
+
+ if (STANDALONE_ZOOKEEPER.name().equalsIgnoreCase(frameworkMode)) {
+ DAGProtos.ConfigurationProto confProtoBefore =
TezUtilsInternal.loadConfProtoFromText();
+ TezUtilsInternal.addUserSpecifiedTezConfiguration(
+ defaultConf, confProtoBefore.getConfKeyValuesList());
+
+ ZkAMRegistryClient registry = ZkAMRegistryClient.getClient(defaultConf);
+ registry.start();
+
+ // TODO: Expose retry counter and sleep time as config — in ZKconfig or
TezConfig?
+ while (!registry.isInitialized()) {
+ TimeUnit.SECONDS.sleep(5);
+ }
+
+ List<AMRecord> records = registry.getAllRecords();
+ if (records.isEmpty()) {
+ throw new RuntimeException("No AM found in ZooKeeper registry");
+ }
+ // TODO: Should we always get the first or there should be some
sophisticated logic?
+ AMRecord amRecord = records.getFirst();
+
+ host = amRecord.getHostName();
+ String portRange =
+ defaultConf.getTrimmed(TezConfiguration.TEZ_AM_TASK_AM_PORT_RANGE,
"12000-12000");
+ port = Integer.parseInt(portRange.split("[-,]")[0]);
+
+ appId = amRecord.getApplicationId().toString();
+ tokenIdentifier = appId;
+ attemptNumber = 1;
+
+ String baseContainerId = appId.replace("application_", "container_");
+ int randomSeq = (int) (Math.random() * 900000) + 100000;
+ containerIdentifier = baseContainerId + "_01_" + randomSeq;
+
+ String zkQuorum =
defaultConf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+ ZkConfig zkconfig = new ZkConfig(defaultConf);
+ zkWorkerClient = CuratorFrameworkFactory.newClient(zkQuorum,
zkconfig.getRetryPolicy());
+ zkWorkerClient.start();
+
+ // Create Ephemeral node representing this worker
+ String workerPath = zkconfig.getZkTaskNameSpace() + "/" + appId + "/" +
containerIdentifier;
+ zkWorkerClient
+ .create()
+ .creatingParentsIfNeeded()
+ .withMode(CreateMode.EPHEMERAL)
+ .forPath(workerPath,
host.getBytes(java.nio.charset.StandardCharsets.UTF_8));
+
+ LOG.info("Registered TezChild Worker in ZK at path: {}", workerPath);
+
+ // FIX: Deterministic token
+ JobTokenIdentifier identifier = new JobTokenIdentifier(new Text(appId));
+ Token<JobTokenIdentifier> sessionToken =
+ new Token<>(identifier,
TezCommonUtils.createJobTokenSecretManager(defaultConf));
+ credentials.addToken(new Text("SessionToken"), sessionToken);
+
+ LOG.info("ZK Mode: Discovered AM {} at {}:{}", appId, host, port);
+
+ } else {
+ assert args.length == 5;
+ host = args[0];
+ port = Integer.parseInt(args[1]);
+ containerIdentifier = args[2];
+ tokenIdentifier = args[3];
+ attemptNumber = Integer.parseInt(args[4]);
Review Comment:
instead of if else, acquiring these could be refactored to separate methods
or even classes
also, if we feel that `TezChild` has changed so badly, I would also consider
introducing class hierarchy for that (not sure at the moment if composition or
inheritance), because another type of `TezChild` could also take care of
special things in the same JVM (e.g. shuffle handler)
##########
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/FetcherOrderedGrouped.java:
##########
@@ -180,7 +180,7 @@ public FetcherOrderedGrouped(HttpConnectionParams
httpConnectionParams,
@VisibleForTesting
protected void fetchNext() throws InterruptedException, IOException {
try {
- if (localDiskFetchEnabled && mapHost.getHost().equals(localShuffleHost)
&& mapHost.getPort() == localShufflePort) {
Review Comment:
we should clearly distinguish between local disk fetch and the containerized
approach in the first implementation
how local disk fetch optimization works on a classic yarn cluster is that
containers on the same host can copy data quickly: in docker containers we
cannot assume that the local folder used for shuffle intermediate data is
shared in any way, so to make this clear in the code, fetchers should be able
to know from the config framework mode, that we're in standalone zookeeper mode
and **should not** set up local fetch I believe
later, this same code path could still be optimized: e.g. fetchers to know
that even if they are in TezChild containers, they are on the same node + the
folder is shared, so they can leverage local fetch, but in the meantime, for
simplicity's sake, we can just disable it (TODO + clarifying code comment can
tell that this area needs further optimization)
however, a http fetch assumes a shuffle handler on the producer side, which
might be an additional burden on this POC, because another design decision has
to be made: how to run a ShuffleHandler, this is up to your fantasy
I think by a docker-compose you cannot co-locate tezchild-1 +
shufflehandler-1, tez-child2 + shufflehandler-2 (to make them share the same
hostname, you need to find a way to run a shuffle handler parallely for each
tezchild, sharing the same local folders, so shuffle handlers can serve them)
note that if shufflehandler somehow runs in the same container/jvm, they see
the same hostname, so you're sorted out with the local fetch optimization
this is obviously not the perfect production setup, but this ticket is the
ground POC of running tez DAGs in a distributed way without yarn
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
##########
@@ -517,45 +531,112 @@ public static TezChild newTezChild(Configuration conf,
String host, int port, St
hadoopShim);
}
- public static void main(String[] args) throws IOException,
InterruptedException, TezException {
+ public static void main(String[] args) throws Exception {
TezClassLoader.setupTezClassLoader();
final Configuration defaultConf = new Configuration();
+ String frameworkMode = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+
+ String host, appId, tokenIdentifier, containerIdentifier;
+ int port, attemptNumber;
+ Credentials credentials = new Credentials();
+
+ if (STANDALONE_ZOOKEEPER.name().equalsIgnoreCase(frameworkMode)) {
+ DAGProtos.ConfigurationProto confProtoBefore =
TezUtilsInternal.loadConfProtoFromText();
+ TezUtilsInternal.addUserSpecifiedTezConfiguration(
+ defaultConf, confProtoBefore.getConfKeyValuesList());
+
+ ZkAMRegistryClient registry = ZkAMRegistryClient.getClient(defaultConf);
+ registry.start();
+
+ // TODO: Expose retry counter and sleep time as config — in ZKconfig or
TezConfig?
+ while (!registry.isInitialized()) {
+ TimeUnit.SECONDS.sleep(5);
+ }
+
+ List<AMRecord> records = registry.getAllRecords();
+ if (records.isEmpty()) {
+ throw new RuntimeException("No AM found in ZooKeeper registry");
+ }
+ // TODO: Should we always get the first or there should be some
sophisticated logic?
+ AMRecord amRecord = records.getFirst();
+
+ host = amRecord.getHostName();
+ String portRange =
+ defaultConf.getTrimmed(TezConfiguration.TEZ_AM_TASK_AM_PORT_RANGE,
"12000-12000");
+ port = Integer.parseInt(portRange.split("[-,]")[0]);
+
+ appId = amRecord.getApplicationId().toString();
+ tokenIdentifier = appId;
+ attemptNumber = 1;
+
+ String baseContainerId = appId.replace("application_", "container_");
+ int randomSeq = (int) (Math.random() * 900000) + 100000;
+ containerIdentifier = baseContainerId + "_01_" + randomSeq;
+
+ String zkQuorum =
defaultConf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+ ZkConfig zkconfig = new ZkConfig(defaultConf);
+ zkWorkerClient = CuratorFrameworkFactory.newClient(zkQuorum,
zkconfig.getRetryPolicy());
+ zkWorkerClient.start();
+
+ // Create Ephemeral node representing this worker
+ String workerPath = zkconfig.getZkTaskNameSpace() + "/" + appId + "/" +
containerIdentifier;
+ zkWorkerClient
+ .create()
+ .creatingParentsIfNeeded()
+ .withMode(CreateMode.EPHEMERAL)
+ .forPath(workerPath,
host.getBytes(java.nio.charset.StandardCharsets.UTF_8));
+
+ LOG.info("Registered TezChild Worker in ZK at path: {}", workerPath);
+
+ // FIX: Deterministic token
+ JobTokenIdentifier identifier = new JobTokenIdentifier(new Text(appId));
+ Token<JobTokenIdentifier> sessionToken =
+ new Token<>(identifier,
TezCommonUtils.createJobTokenSecretManager(defaultConf));
+ credentials.addToken(new Text("SessionToken"), sessionToken);
+
+ LOG.info("ZK Mode: Discovered AM {} at {}:{}", appId, host, port);
+
+ } else {
+ assert args.length == 5;
+ host = args[0];
+ port = Integer.parseInt(args[1]);
+ containerIdentifier = args[2];
+ tokenIdentifier = args[3];
+ attemptNumber = Integer.parseInt(args[4]);
+
+ DAGProtos.ConfigurationProto confProto =
+
TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name()));
+ TezUtilsInternal.addUserSpecifiedTezConfiguration(
+ defaultConf, confProto.getConfKeyValuesList());
+ }
Thread.setDefaultUncaughtExceptionHandler(new
YarnUncaughtExceptionHandler());
final String pid = System.getenv().get("JVM_PID");
+ String[] localDirs =
+
TezCommonUtils.getTrimmedStrings(System.getenv(Environment.LOCAL_DIRS.name()));
-
- assert args.length == 5;
- String host = args[0];
- int port = Integer.parseInt(args[1]);
- final String containerIdentifier = args[2];
- final String tokenIdentifier = args[3];
- final int attemptNumber = Integer.parseInt(args[4]);
- final String[] localDirs =
TezCommonUtils.getTrimmedStrings(System.getenv(Environment.LOCAL_DIRS
- .name()));
- CallerContext.setCurrent(new
CallerContext.Builder("tez_"+tokenIdentifier).build());
- LOG.info("TezChild starting with PID=" + pid + ", containerIdentifier=" +
containerIdentifier);
+ CallerContext.setCurrent(new CallerContext.Builder("tez_" +
tokenIdentifier).build());
+ LOG.info("TezChild starting with PID={}, containerIdentifier={}", pid,
containerIdentifier);
Review Comment:
in docker containers, pid is going to be 1 always, so depending on the
framework mode, `JVM_PID` is not always valid to be choosen
##########
tez-dist/src/docker/docker-compose.yml:
##########
@@ -126,6 +130,36 @@ services:
datanode:
condition: service_started
+ tez-child-1:
+ image: apache/tez:${TEZ_VERSION:-1.0.0-SNAPSHOT}
+ container_name: tez-child-1
+ hostname: tez-child-1
+ networks:
+ - hadoop-network
+ volumes:
+ - tez_shared_data:/data
+ - tez_shared_tmp:/tmp
Review Comment:
as implied in another comment, let's assume first that tez task containers
do have a local (not shared) folder for intermediate data, so they can be later
scheduled to any node in the world and keep on working (we decide local fetch
optimization later)
##########
tez-dist/src/docker/docker-compose.yml:
##########
@@ -126,6 +130,36 @@ services:
datanode:
condition: service_started
+ tez-child-1:
+ image: apache/tez:${TEZ_VERSION:-1.0.0-SNAPSHOT}
+ container_name: tez-child-1
+ hostname: tez-child-1
+ networks:
+ - hadoop-network
Review Comment:
nit: let's try not to use "hadoop" in the network name, as especially with
this patch, we're moving away from the classic hadoop setups
##########
tez-dist/src/docker/docker-compose.yml:
##########
@@ -126,6 +130,36 @@ services:
datanode:
condition: service_started
+ tez-child-1:
Review Comment:
instead of defining separate tez child containers, try to simply scale them,
like `llapdaemon` in hive, see:
https://github.com/apache/hive/blob/9e38b8a3c58720899f413fe2531fed4138fb0a7f/packaging/src/docker/docker-compose.yml#L165
and the starter script:
https://github.com/apache/hive/blob/9e38b8a3c58720899f413fe2531fed4138fb0a7f/packaging/src/docker/start-hive.sh#L32
you can also add a `start-tez.sh` script here and refer to it in README: a
dev/user should be able to start a tez cluster on top of docker-compose without
any prior knowledge, just by this script
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
Review Comment:
I think if we have a docker-compose file, these separate docker run commands
are not supposed to be documented at all, it's verbose, and we're not
encouraging users to do so, instead, use a starter script that utilizes the
docker-compose file
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
Review Comment:
it's strange that the docker run command for an AM doesn't contain the
artificial application id, but same for task container does, I wish this could
become shorter...not sure yet how to proceed, maybe after the docker-compose
file and a corresponding starter script is going to take care of this
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+5. To override the tez-site.xml in docker image use:
+
+ * Set the `TEZ_CUSTOM_CONF_DIR` environment variable in `am.env` /
+ `child.env` or via the `docker run` command (e.g.,
+ `/opt/tez/custom-conf`).
+
+ ```bash
+ export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "$TEZ_SITE_PATH:/opt/tez/custom-conf/tez-site.xml" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+6. To add plugin jars in docker image use:
+
+ * The plugin directory path inside the Docker container is fixed at
+ `/opt/tez/plugins`.
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "/path/to/your/local/plugins:/opt/tez/plugins" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
Review Comment:
this section can be replaced with some tips pointing to jars:
```
export TEZ_PLUGIN_JARS=/path/to/your/local/plugins
./start-tez.sh
```
this will make the testing with custom jars extremely easy
##########
tez-dist/src/docker/README.md:
##########
@@ -0,0 +1,196 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Tez Docker
+
+This directory contains a unified Docker implementation for running TezAM
+and TezChild process from a single container image. Based on
+`TEZ_COMPONENT` environment variable the entrypoint is dynamically selected
+
+1. Building the docker image:
+
+ ```bash
+ mvn clean install -DskipTests -Pdocker
+ ```
+
+ Alternatively, you can build it explicitly via the provided script:
+
+ ```bash
+ ./tez-dist/src/docker/build.sh -tez <version> -repo apache
+ ```
+
+2. Local Zookeeper Setup (Standalone):
+
+ If you are running the AM container use the official Docker
+ image (Refer to docker-compose.yml):
+
+ ```bash
+ docker pull zookeeper:3.8.4
+
+ docker run -d \
+ --name zookeeper-server \
+ -p 2181:2181 \
+ -p 8080:8080 \
+ -e ZOO_MY_ID=1 \
+ zookeeper:3.8.4
+ ```
+
+3. Running the Tez containers explicitly:
+
+ **Running the Tez AM:**
+
+ ```bash
+ export TEZ_VERSION=1.0.0-SNAPSHOT
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+ * `TEZ_VERSION` corresponds to the Maven `${project.version}`.
+ Set this environment variable in your shell before running the commands.
+
+ * Expose ports using the `-p` flag based on the
+ `tez.am.client.am.port-range` property in `tez-site.xml`.
+
+ * The `--hostname` flag configures the container's hostname, allowing
+ services on the host (e.g., macOS) to connect to it.
+
+ * Ensure the `--env-file` flag is included, or at a minimum, pass
+ `-e TEZ_FRAMEWORK_MODE=STANDALONE_ZOOKEEPER` and `-e TEZ_COMPONENT=AM`
+ to the `docker run` command.
+
+ **Running the Tez Child:**
+
+ The child container requires specific arguments (`<am-host> <am-port>
+ <container-id> <token-id> <attempt-number>`) to connect back to the
+ Application Master.
+
+ Assuming your AM is running on `localhost` port `10001`, and the AM
+ assigned the container ID `container_1703023223000_0001_01_000001`:
+
+ ```bash
+ docker run --rm \
+ --network host \
+ --env-file tez-dist/src/docker/child.env \
+ --name tez-child \
+ --hostname localhost \
+ apache/tez:1.0.0-SNAPSHOT \
+ localhost 10001 container_1703023223000_0001_01_000001 dummy_token_abc 1
+ ```
+
+4. Debugging the Tez containers:
+ Uncomment the `JAVA_TOOL_OPTIONS` in `am.env` (or `child.env` for
+ port 5006) and expose the debug port using `-p` flag:
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 -p 5005:5005 \
+ --env-file tez-dist/src/docker/am.env \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+5. To override the tez-site.xml in docker image use:
+
+ * Set the `TEZ_CUSTOM_CONF_DIR` environment variable in `am.env` /
+ `child.env` or via the `docker run` command (e.g.,
+ `/opt/tez/custom-conf`).
+
+ ```bash
+ export TEZ_SITE_PATH=$(pwd)/tez-dist/src/docker/conf/tez-site.xml
+
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "$TEZ_SITE_PATH:/opt/tez/custom-conf/tez-site.xml" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+6. To add plugin jars in docker image use:
+
+ * The plugin directory path inside the Docker container is fixed at
+ `/opt/tez/plugins`.
+
+ ```bash
+ docker run --rm \
+ -p 10001:10001 \
+ --env-file tez-dist/src/docker/am.env \
+ -v "/path/to/your/local/plugins:/opt/tez/plugins" \
+ --name tez-am \
+ --hostname localhost \
+ apache/tez:$TEZ_VERSION
+ ```
+
+7. Using Docker Compose (Local Testing Cluster):
+
+ The provided `docker-compose.yml` offers a complete, minimal Hadoop
+ ecosystem to test Tez in a distributed manner locally without setting
+ up a real cluster.
+
+ **Services Included:**
+
+ * **namenode & datanode:** A minimal Apache Hadoop HDFS cluster
+ (lean image)
+
+ * **zookeeper:** Required by the Tez AM for standalone session
+ discovery
+
+ * **tez-am:** It automatically waits for Zookeeper and HDFS to
+ be healthy before starting up.
+
+ * **tez-child:** TBD
+
+ **To start the full cluster:**
+
+ ```bash
+ docker-compose -f tez-dist/src/docker/docker-compose.yml up -d
+ ```
+
+ **To monitor the Application Master logs:**
+
+ ```bash
+ docker-compose -f tez-dist/src/docker/docker-compose.yml logs -f tez-am
+ ```
+
+ **To shut down the cluster and clean up volumes (HDFS/Zookeeper data):**
+
+ ```bash
+ docker-compose -f tez-dist/src/docker/docker-compose.yml down -v
+ ```
+
+8. To mount custom plugins or JARs required by Tez AM (e.g., for split
+ generation — typically the hive-exec jar, but in general, any UDFs or
+ dependencies previously managed via YARN localization:
+
+ * Create a directory `tez-plugins` and add all required jars.
+
+ * Uncomment the following lines in docker compose under the `tez-am`
+ and `tez-child` services to mount this directory as a volume to
+ `/opt/tez/plugins` in the docker container.
+
+ ```yaml
+ volumes:
+ - ./tez-plugins:/opt/tez/plugins
+ ```
Review Comment:
this should be taken care of by the same as above in 6)
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
##########
@@ -517,45 +531,112 @@ public static TezChild newTezChild(Configuration conf,
String host, int port, St
hadoopShim);
}
- public static void main(String[] args) throws IOException,
InterruptedException, TezException {
+ public static void main(String[] args) throws Exception {
TezClassLoader.setupTezClassLoader();
final Configuration defaultConf = new Configuration();
+ String frameworkMode = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+
+ String host, appId, tokenIdentifier, containerIdentifier;
+ int port, attemptNumber;
+ Credentials credentials = new Credentials();
+
+ if (STANDALONE_ZOOKEEPER.name().equalsIgnoreCase(frameworkMode)) {
+ DAGProtos.ConfigurationProto confProtoBefore =
TezUtilsInternal.loadConfProtoFromText();
+ TezUtilsInternal.addUserSpecifiedTezConfiguration(
+ defaultConf, confProtoBefore.getConfKeyValuesList());
+
+ ZkAMRegistryClient registry = ZkAMRegistryClient.getClient(defaultConf);
+ registry.start();
+
+ // TODO: Expose retry counter and sleep time as config — in ZKconfig or
TezConfig?
+ while (!registry.isInitialized()) {
+ TimeUnit.SECONDS.sleep(5);
+ }
+
+ List<AMRecord> records = registry.getAllRecords();
+ if (records.isEmpty()) {
+ throw new RuntimeException("No AM found in ZooKeeper registry");
+ }
+ // TODO: Should we always get the first or there should be some
sophisticated logic?
+ AMRecord amRecord = records.getFirst();
+
+ host = amRecord.getHostName();
+ String portRange =
+ defaultConf.getTrimmed(TezConfiguration.TEZ_AM_TASK_AM_PORT_RANGE,
"12000-12000");
+ port = Integer.parseInt(portRange.split("[-,]")[0]);
+
+ appId = amRecord.getApplicationId().toString();
+ tokenIdentifier = appId;
+ attemptNumber = 1;
+
+ String baseContainerId = appId.replace("application_", "container_");
+ int randomSeq = (int) (Math.random() * 900000) + 100000;
+ containerIdentifier = baseContainerId + "_01_" + randomSeq;
+
+ String zkQuorum =
defaultConf.get(TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM);
+ ZkConfig zkconfig = new ZkConfig(defaultConf);
+ zkWorkerClient = CuratorFrameworkFactory.newClient(zkQuorum,
zkconfig.getRetryPolicy());
+ zkWorkerClient.start();
+
+ // Create Ephemeral node representing this worker
+ String workerPath = zkconfig.getZkTaskNameSpace() + "/" + appId + "/" +
containerIdentifier;
+ zkWorkerClient
+ .create()
+ .creatingParentsIfNeeded()
+ .withMode(CreateMode.EPHEMERAL)
+ .forPath(workerPath,
host.getBytes(java.nio.charset.StandardCharsets.UTF_8));
Review Comment:
instead of a random + create(), an incremental + retry logic should be
implemented here:
1. try to acquire `n`, pass
2. if fails with `n`, try `n+1`
##########
tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/ShuffleUtils.java:
##########
@@ -316,6 +316,9 @@ static ByteBuffer generateDMEPayload(boolean
sendEmptyPartitionDetails,
if (!sendEmptyPartitionDetails || outputGenerated) {
String host = context.getExecutionContext().getHostName();
+ if (host == null) {
Review Comment:
I believe we should not simply hardcode "localhost" here: if this caused an
NPE, it means ExecutionContext needs to be improved and prepared for the
unmanaged sessions scenario
let's assume that we're going to run `TezChild` containers as part of a
docker-compose and later as k8s pods in a deployment or statefulset: so how we
create an ExecutionContext now should be in line with that kind
e.g. currently we get NM_HOST:
https://github.com/apache/tez/blob/17d15490cedab7d18a4d4c1f83374e30c58a363f/tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java#L563
this is Yarn-specific, but fortunately you worked on TEZ-4689, so whatever
you did there for node context, it should be followed here for execution
context as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
