This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new ff2128f8 [MINOR] docs: update document for tez client plugin. (#987)
ff2128f8 is described below
commit ff2128f82ee58a62f2da46d0bca0dd4cbf0ea38f
Author: zhengchenyu <[email protected]>
AuthorDate: Mon Jul 3 15:50:55 2023 +0800
[MINOR] docs: update document for tez client plugin. (#987)
Co-authored-by: zhengchenyu001 <[email protected]>
---
README.md | 20 +++++++++++++++++++-
docs/client_guide.md | 16 ++++++++++++++++
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index a4246fb5..1f1cd078 100644
--- a/README.md
+++ b/README.md
@@ -22,10 +22,11 @@ It provides the ability to push shuffle data into
centralized storage service,
changing the shuffle style from "local file pull-like style" to "remote block
push-like style".
It brings in several advantages like supporting disaggregated storage
deployment,
super large shuffle jobs, and high elasticity.
-Currently it supports [Apache Spark][1] and [Apache Hadoop MapReduce][2].
+Currently it supports [Apache Spark][1] and [Apache Hadoop MapReduce][2] and
[Apache Tez][3].
[1]: https://spark.apache.org
[2]: https://hadoop.apache.org
+[3]: https://tez.apache.org
[](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml)
[](https://codecov.io/gh/apache/incubator-uniffle)
@@ -248,6 +249,23 @@ The jar for MapReduce is located in
<RSS_HOME>/jars/client/mr/rss-client-mr-XXXX
Note that the RssMRAppMaster will automatically disable slow start (i.e.,
`mapreduce.job.reduce.slowstart.completedmaps=1`)
and job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)
+### Deploy Tez Client
+
+1. Append client jar to pacakge which is set by 'tez.lib.uris'.
+
+In production mode, you can append client jar
(rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
+
+In development mode, you can append client jar
(rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
+
+2. Update tez-site.xml to enable Uniffle.
+
+| Property Name |Default| Description |
+|----------------------------|---|------------------------------|
+| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc
-XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC
org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
+| tez.rss.coordinator.quorum
|coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|
+
+Note that the RssDAGAppMaster will automatically disable slow start (i.e.,
`tez.shuffle-vertex-manager.min-src-fraction=1`,
`tez.shuffle-vertex-manager.max-src-fraction=1`).
+
### Deploy In Kubernetes
We have provided an operator for deploying uniffle in kubernetes environments.
diff --git a/docs/client_guide.md b/docs/client_guide.md
index 7864a2cb..e34e651b 100644
--- a/docs/client_guide.md
+++ b/docs/client_guide.md
@@ -97,6 +97,22 @@ The jar for MapReduce is located in
<RSS_HOME>/jars/client/mr/rss-client-mr-XXXX
Note that the RssMRAppMaster will automatically disable slow start (i.e.,
`mapreduce.job.reduce.slowstart.completedmaps=1`)
and job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)
+### Deploy Tez Client Plugin
+
+1. Append client jar to pacakge which is set by 'tez.lib.uris'.
+
+In production mode, you can append client jar
(rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
+
+In development mode, you can append client jar
(rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
+
+2. Update tez-site.xml to enable Uniffle.
+
+| Property Name |Default| Description |
+|----------------------------|---|------------------------------|
+| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc
-XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC
org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
+| tez.rss.coordinator.quorum
|coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|
+
+Note that the RssDAGAppMaster will automatically disable slow start (i.e.,
`tez.shuffle-vertex-manager.min-src-fraction=1`,
`tez.shuffle-vertex-manager.max-src-fraction=1`).
## Configuration
