Repository: tez Updated Branches: refs/heads/master b9d5c2056 -> c4dbeeab3
TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN. (hitesh) Project: http://git-wip-us.apache.org/repos/asf/tez/repo Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/c4dbeeab Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/c4dbeeab Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/c4dbeeab Branch: refs/heads/master Commit: c4dbeeab3fc20ccd75367013f2bbffc2fec7f3a3 Parents: b9d5c20 Author: Hitesh Shah <[email protected]> Authored: Mon Jul 27 13:59:21 2015 -0700 Committer: Hitesh Shah <[email protected]> Committed: Mon Jul 27 13:59:21 2015 -0700 ---------------------------------------------------------------------- CHANGES.txt | 1 + docs/src/site/markdown/tez-ui.md | 136 ++++++++++++------------ docs/src/site/markdown/tez_acls.md | 5 + docs/src/site/markdown/tez_ui_user_data.md | 5 +- 4 files changed, 80 insertions(+), 67 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index bd6aa25..dbf0833 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -11,6 +11,7 @@ INCOMPATIBLE CHANGES TEZ-2468. Change the minimum Java version to Java 7. ALL CHANGES: + TEZ-2239. Update Tez UI docs to explain how to configure history url for YARN. TEZ-2602. Throwing EOFException when launching MR job. TEZ-2496. Consider scheduling tasks in ShuffleVertexManager based on the partition sizes from the source. TEZ-2616. Fix build warning by undefined version of maven-findbugs-plugin. http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez-ui.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/tez-ui.md b/docs/src/site/markdown/tez-ui.md index 52332ce..3855fbb 100644 --- a/docs/src/site/markdown/tez-ui.md +++ b/docs/src/site/markdown/tez-ui.md @@ -25,12 +25,74 @@ application data generated during the lifetime of a YARN application. Tez provides its own UI that interfaces with the _Application Timeline Server_ and displays both a live and historical view of the Tez application inside a Tez UI web application. -## Application Timeline Server Setup +__For information on how application-specific data ends up being displayed in the Tez UI, please refer to [this User Guide](tez_ui_user_data.html)__ -_Requires_: **Hadoop 2.6.0 or above** +## Setting up Tez for the Tez UI +--------- + +_Requires_: **Tez 0.6.0 or above** + +For Tez to have a functional UI and for the History links to work correctly from the YARN ResourceManager UI, the following settings need to be enabled in tez-site.xml: + +``` +tez-site.xml +------------- +... +<property> + <description>Enable Tez to use the Timeline Server for History Logging</description> + <name>tez.history.logging.service.class</name> + <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value> +</property> -A minimal _yarn-site.xml_ configuration snippet is provided below. Replace localhost with the actual -hostname if running in a distributed setup. +<property> + <description>URL for where the Tez UI is hosted</description> + <name>tez.tez-ui.history-url.base</name> + <value>http://<webserver-host:9999/tez-ui/</value> +</property> +... + +``` + +## Deploying the Tez UI + +The tez-ui.war can be obtained in 3 ways: + + - Obtain the tez-ui.war from the binary artifacts [for a given release](./releases/). Binary artifacts are not available for all releases and in such cases, please use any of the other options listed below. + - Obtain the tez-ui.war from the [Maven repo](https://repository.apache.org/content/repositories/releases/org/apache/tez/tez-ui/) for a given release. + - Build the tez-ui.war from the source itself. Refer to the README.txt in the tez-ui module for more details. + +Once the tez-ui.war is available, you can extract the contents of the war and host them on any webserver. The Tez UI consists only of html, js and css files +and does not require any complex server-side hosting logic. + +### Configuring the Timeline Server URL and Resource Manager UI URL. +By default, the Tez UI attempts to connect to Timeline Server using the same host as the Tez UI. For example, if the UI is hosted on localhost, the Timeline Server URL is assumed to be http(s)://localhost:8188 and the Resource manager web url is assumed to be http(s)://localhost:8088. + +If the Timeline Server and/or Resource manager is hosted on a different host, the Tez UI needs corresponding changes to be configured in scripts/configs.js ( within the extracted tez-ui.war). Uncomment the following lines and set the hostname and port appropriately. "timelineBaseUrl" maps to YARN Timeline Server and "RMWebUrl" maps to YARN ResourceManager. + +``` + // timelineBaseUrl: 'http://localhost:8188', + // RMWebUrl: 'http://localhost:8088', + +``` + +### Hosting the UI in Tomcat. + +1. Remove any old deployments in $TOMCAT_HOME/webapps +2. Extract the war into $TOMCAT_HOME/webapps/tez-ui/ +3. Modify scripts/config.js as needed. +4. Restart tomcat and the UI should be available under the tez-ui/ path. + +### Hosting the UI using any standalone webserver +1. Untar the war file +2. Modify scripts/config.js as needed. +3. Copy the resulting directory to the document root of the web server. +4. Restart the webserver + +## Additional Setup steps for the Application Timeline Server + +_Requires_: **Apache Hadoop 2.6.0 or above** + +The following configurations need to be setup in yarn-site.xml for correct functional behavior of the Timeline Server with respect to the Tez UI. Replace localhost with the actual hostname if running in a distributed setup. ``` yarn-site.xml @@ -57,75 +119,17 @@ yarn-site.xml <name>yarn.timeline-service.http-cross-origin.enabled</name> <value>true</value> </property> -... -``` - -[Detailed Application Timeline Server Configuration](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServer.html) - -``` -# Starting the Application Timeline Server -$ $HADOOP_PREFIX/sbin/yarn-daemon.sh start timelineserver - -``` -## Tez Setup ---------- - -_Requires_: **Tez 0.6.0 or above** - -A minimal _tez-site.xml_ configuration snippet is provided below - -``` -tez-site.xml -------------- -... -<property> - <description>Log history using the Timeline Server</description> - <name>tez.history.logging.service.class</name> - <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value> -</property> <property> - <description>Publish configuration information to Timeline server.</description> - <name>tez.runtime.convert.user-payload.to.history-text</name> + <description>Publish YARN information to Timeline Server</description> + <name> yarn.resourcemanager.system-metrics-publisher.enabled</name> <value>true</value> </property> ... - ``` -[Detailed Tez - Timeline Server Configuration](tez_yarn_timeline.html) - -[Embedding Application Specific Data into Tez UI](tez_ui_user_data.html) +__For more detailed information (setup, configuration, deployment), please refer to the [Apache Hadoop Documentation on the Application Timeline Server](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServer.html)__ -## Tez UI Install +__For general information on the compatibility matrix of the Tez UI with YARN TimelineServer, please refer to the [Tez - Timeline Server Guide](tez_yarn_timeline.html)__ -### Building Tez UI from sources -For instructions on building Tez UI from sources or setting up a development environment see the -README.txt in tez-ui subfolder. The build generates a war file of the distribution -in the tez-ui/target. Currently the Tez-UI can be hosted using one of the following options. - -### hosting using the war file and tomcat. -1. Remove any old deployments in $TOMCAT_HOME/webapps -2. Copy the war to $TOMCAT_HOME/webapps -3. Restart tomcat and the war will get deployed. The content of the war would be available in -$TOMCAT_HOME/webapps/tez-ui-x.x.x. -### hosting using extracted war file and a standalone webserver -1. untar the war file -2. copy the resulting directory to the document root of the web server. -3. reload/restart the webserver - -### configuring the Timeline Server URL and Resource manager web url. -By default, the Tez UI attempts to connect to Timeline Server using the same host as the Tez UI. -For example, if the UI is hosted on localhost, the Timeline Server URL is assumed to be -http(s)://localhost:8188 and the Resource manager web url is assumed to be http(s)://localhost:8088. - -If the Timeline Server and/or Resource manager is hosted on a different host, the Tez UI needs -corresponding changes to be configured in scripts/configs.js. Uncomment the following lines and -set the hostname and port appropriately. - -``` - // timelineBaseUrl: 'http://localhost:8188', - // RMWebUrl: 'http://localhost:8088', - -``` http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez_acls.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/tez_acls.md b/docs/src/site/markdown/tez_acls.md index 2ac6830..7264c58 100644 --- a/docs/src/site/markdown/tez_acls.md +++ b/docs/src/site/markdown/tez_acls.md @@ -37,6 +37,11 @@ From the above, you can see that All users/groups that have access to do operati Support for ACLs was introduced in Tez 0.5.0. Integration of these ACLs with YARN Timeline is only available from Tez 0.6.0 onwards. +## ACLs and the Tez UI + +For [versions of YARN Timeline Server](./tez_yarn_timeline.html) that support ACLs, the UI will respect the ACLs for such secure clusters. For example, when a particular +authenticated user is viewing the "All DAGs" page on the UI, the user will only be shown DAGs that the user has access to view. + ## How to setup the ACLs By default, ACLs are always enabled in Tez. To disable ACLs, set the following configuration property: http://git-wip-us.apache.org/repos/asf/tez/blob/c4dbeeab/docs/src/site/markdown/tez_ui_user_data.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/tez_ui_user_data.md b/docs/src/site/markdown/tez_ui_user_data.md index 71ffde3..7dd6abd 100644 --- a/docs/src/site/markdown/tez_ui_user_data.md +++ b/docs/src/site/markdown/tez_ui_user_data.md @@ -26,6 +26,7 @@ The Tez UI is built mainly off data stored in [YARN Timeline]. The Tez API, curr To set DAG level information, the API to use is DAG::setDAGInfo. ( Please refer to the Javadocs for more detailed and up-to-date information ) The DAG::setDAGInfo() API expects a to be String passed to it. This string is recommended to be a json-encoded value with the following keys recognized keys by the UI: + - "context": The application context in which this DAG is being used. For example, this could be set to "Hive" or "Pig" if this is being run as part of a Hive or Pig script. - "description": General description on what this DAG is going to do. In the case of Hive, this could be the SQL query text. @@ -34,6 +35,7 @@ The DAG::setDAGInfo() API expects a to be String passed to it. This string is re Each Input/Output/Processor specified in the DAG Plan is specified via a TezEntityDescriptor. Applications specify a user payload that is used to initialize/configure the instance as needed. From a Tez UI point of view, users are usually keen to understand what "work" the particular Input/Output/Processor is doing in addition to any additional configuration information on how the object was initialized/configured. Keeping that in mind, each TezEntityDescriptor supports an api for application developers to specify this information when creating the DAG plan. The API to use for this is setHistoryText(). The setHistoryText() API expects a String to be passed to it. This string is recommended to be a json-encoded value with the following keys recognized keys by the UI: + - "desc" : A simple string describing for the object in question. For example, for a particular Hive Processor, this could be a description of what that particular processor is doing. - "config" : A map of key-value pairs representing the configuration/payload used to initialize the object in question. @@ -44,9 +46,10 @@ By default, the Inputs/Outputs/Processors that are part of the tez-runtime-libra > <value>true</value><br/> > </property><br/> +Please note that this is a data-heavy operation and adds a lot of load on the TimelineServer (due to the additional amount of configuration data generated) and therefore should be disabled for most jobs unless there is a real debugging need. + ## Use of DAG Info and History Text in the Tez UI If the data setup in the DAG Info and History Text conforms to the format expected by the UI, it will be displayed in the Tez UI in an easy to consume manner. In cases where this is not possible, the UI may fall back to either not displaying the data at all or displaying the string as is in a safe manner. [YARN Timeline]:./tez_yarn_timeline.html -
