TEZ-3360. Tez Custom Shuffle Handler Documentation (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/tez/repo Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/613f1e07 Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/613f1e07 Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/613f1e07 Branch: refs/heads/master Commit: 613f1e0743999c3bf558761c60165bcd2dcd05c7 Parents: 78eee17 Author: Jonathan Eagles <[email protected]> Authored: Mon Aug 15 10:05:26 2016 -0500 Committer: Jonathan Eagles <[email protected]> Committed: Mon Aug 15 10:05:26 2016 -0500 ---------------------------------------------------------------------- TEZ-3334-CHANGES.txt | 1 + docs/src/site/markdown/shuffle-handler.md | 72 ++++++++++++++++++++++++++ docs/src/site/markdown/user_guides.md | 1 + docs/src/site/site.xml | 1 + 4 files changed, 75 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/TEZ-3334-CHANGES.txt ---------------------------------------------------------------------- diff --git a/TEZ-3334-CHANGES.txt b/TEZ-3334-CHANGES.txt index 493614f..f93ec6e 100644 --- a/TEZ-3334-CHANGES.txt +++ b/TEZ-3334-CHANGES.txt @@ -4,6 +4,7 @@ Apache Tez Change Log INCOMPATIBLE CHANGES: ALL CHANGES: + TEZ-3360. Tez Custom Shuffle Handler Documentation TEZ-3411. TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler port TEZ-3412. Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString() TEZ-3410. ShuffleHandler should use Path.SEPARATOR instead "/" http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/shuffle-handler.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/shuffle-handler.md b/docs/src/site/markdown/shuffle-handler.md new file mode 100644 index 0000000..5dc6289 --- /dev/null +++ b/docs/src/site/markdown/shuffle-handler.md @@ -0,0 +1,72 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<head><title>Shuffle Handler</title></head> + +## Tez Shuffle Handler Overview + +A Tez specific shuffle handler allows Tez DAGs to shuffle data in a way that takes advantage of the new features in Tez. +In particular, the Tez shuffle handler allows DAGs to shuffle data more efficiently for Tezâs new data movements types +and runtime optimizations, such as auto-reduce parallelism. Long running Tez sessions will be able to clean up +intermediate data for completed queries and Tez applications can decide to clean up completed intermediate data for +running applications. + +## Setup for the Tez Shuffle Handler +--------- + +_Requires_: **Apache Tez 0.9.0 or above** + +Configuration in the client specify the Tez shuffle handler + +``` +tez-site.xml +------------- +... +<property> + <name>tez.am.shuffle.auxiliary-service.id</name> + <value>tez_shuffle</value> +</property> +... + +``` + +## Deploying the Tez Shuffle Handler + +The Tez Shuffle Handler jar artifact org.apache.org:tez-aux-services needs to be placed into the Node Manager classpath and restarted + +## Setup for Node Manager + +_Requires_: **Apache Hadoop 2.6.0 or above** + +The following configuration needs to be setup in the Node Manager yarn-site.xml to enable the Tez Shuffle Handler + +``` +yarn-site.xml +------------- +... +<property> + <name>yarn.nodemanager.aux-services</name> + <value>tez_shuffle</value> +</property> + +<property> + <name>yarn.nodemanager.aux-services.tez_shuffle.class</name> + <value>org.apache.tez.auxservices.ShuffleHandler</value> +</property> +... + +``` http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/markdown/user_guides.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/user_guides.md b/docs/src/site/markdown/user_guides.md index 890974e..453df16 100644 --- a/docs/src/site/markdown/user_guides.md +++ b/docs/src/site/markdown/user_guides.md @@ -23,4 +23,5 @@ - [Access Control in Tez](./tez_acls.html) - [Embedding Application Specific Data into Tez UI](./tez_ui_user_data.html) - [Tez UI - Overview and installation](./tez-ui.html) + - [Tez Shuffle Handler - Overview and installation](./shuffle-handler.html) http://git-wip-us.apache.org/repos/asf/tez/blob/613f1e07/docs/src/site/site.xml ---------------------------------------------------------------------- diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml index 3317bac..5674951 100644 --- a/docs/src/site/site.xml +++ b/docs/src/site/site.xml @@ -107,6 +107,7 @@ <item name="Install Guide" href="install.html"/> <item name="Local Mode" href="localmode.html"/> <item name="Tez UI" href="tez-ui.html"/> + <item name="Tez Shuffle Handler" href="shuffle-handler.html"/> <item name="User Guides" href="user_guides.html"/> </menu>
