This is an automated email from the ASF dual-hosted git repository.
abstractdog pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/tez.git
The following commit(s) were added to refs/heads/master by this push:
new 7058de20a TEZ-4633: Set Baseline to JDK 21 and cleanup obsolete JVM
configurations (#506) (Raghav Aggarwal reviewed by Laszlo Bodor)
7058de20a is described below
commit 7058de20a33b758cd379bfc7bcbedd9e27c5694b
Author: Raghav Aggarwal <[email protected]>
AuthorDate: Tue Jun 23 12:36:03 2026 +0530
TEZ-4633: Set Baseline to JDK 21 and cleanup obsolete JVM configurations
(#506) (Raghav Aggarwal reviewed by Laszlo Bodor)
---
docs/src/site/markdown/install.md | 419 ++++++++++++---------
pom.xml | 1 -
.../java/org/apache/tez/client/TezClientUtils.java | 2 +-
.../org/apache/tez/common/JavaOptsChecker.java | 9 -
.../java/org/apache/tez/common/TezCommonUtils.java | 4 +-
.../org/apache/tez/dag/api/TezConfiguration.java | 26 +-
.../org/apache/tez/client/TestTezClientUtils.java | 21 +-
.../org/apache/tez/common/TestJavaOptsChecker.java | 30 +-
tez-tools/tez-javadoc-tools/pom.xml | 94 -----
.../tez/tools/javadoc/model/package-info.java | 25 --
.../tools/javadoc/doclet/ConfigStandardDoclet.java | 230 -----------
.../tez/tools/javadoc/doclet/package-info.java | 25 --
.../org/apache/tez/tools/javadoc/model/Config.java | 51 ---
.../tez/tools/javadoc/model/ConfigProperty.java | 115 ------
.../apache/tez/tools/javadoc/util/HtmlWriter.java | 160 --------
.../org/apache/tez/tools/javadoc/util/Writer.java | 39 --
.../apache/tez/tools/javadoc/util/XmlWriter.java | 107 ------
.../tez/tools/javadoc/util/package-info.java | 25 --
.../tools/javadoc/doclet/ConfigStandardDoclet.java | 59 +--
.../tez/tools/javadoc/doclet/package-info.java | 2 +-
.../org/apache/tez/tools/javadoc/model/Config.java | 16 +-
.../tez/tools/javadoc/model/ConfigProperty.java | 16 +-
.../tez/tools/javadoc/model/package-info.java | 2 +-
.../apache/tez/tools/javadoc/util/HtmlWriter.java | 22 +-
.../org/apache/tez/tools/javadoc/util/Writer.java | 20 +-
.../apache/tez/tools/javadoc/util/XmlWriter.java | 16 +-
.../tez/tools/javadoc/util/package-info.java | 2 +-
27 files changed, 340 insertions(+), 1198 deletions(-)
diff --git a/docs/src/site/markdown/install.md
b/docs/src/site/markdown/install.md
index 44ca862a8..243535a21 100644
--- a/docs/src/site/markdown/install.md
+++ b/docs/src/site/markdown/install.md
@@ -6,7 +6,7 @@
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
@@ -15,92 +15,114 @@
limitations under the License.
-->
-<head><title>Install and Deployment Instructions</title></head>
+# Install/Deploy Instructions for Tez
-Install/Deploy Instructions for Tez
----------------------------------------------------------------------------
-Replace x.y.z with the tez release number that you are using. E.g. 0.5.0. For
Tez
-versions 0.8.3 and higher, Tez needs Apache Hadoop to be of version 2.6.0 or
higher.
-For Tez version 0.9.0 and higher, Tez needs Apache Hadoop to be version 2.7.0
+Replace x.y.z with the tez release number that you are using. E.g. 1.0.0.
+For Tez version 1.0.0 and higher, Tez needs Apache Hadoop to be version 3.5.0
or higher.
-1. Deploy Apache Hadoop using version of 2.7.0 or higher.
- - You need to change the value of the hadoop.version property in the
- top-level pom.xml to match the version of the hadoop branch being used.
+For older Tez versions, the compatibility is as follows:
- ```
- $ hadoop version
- ```
+- Tez 0.9.0 and higher: Apache Hadoop 2.7.0 or higher.
+- Tez 0.8.3 and higher: Apache Hadoop 2.6.0 or higher.
+- Tez 0.5.0 to 0.8.2: Apache Hadoop 2.2.0 or higher.
+
+> **Note:** For a quick local setup, you can use the helper script
+ [tez_run_example.sh][2] which automates these installation,
+ configuration, and deployment steps to run a simple word count example
+ in a pseudo-distributed single-node cluster.
+
+1. Deploy Apache Hadoop using version of 3.5.0 or higher.
-2. Build tez using `mvn clean package -DskipTests=true
-Dmaven.javadoc.skip=true`
- - This assumes that you have already installed JDK8 or later and Maven 3
or later.
- - Tez also requires Protocol Buffers 3.19.4, including the
protoc-compiler.
- * This can be downloaded from
https://github.com/google/protobuf/tags/.
- * On Mac OS X with the homebrew package manager `brew install
protobuf250`
- * For rpm-based linux systems, the yum repos may not have the 3.19.4
version.
- `rpm.pbone.net` has the protobuf-3.19.4 and
protobuf-compiler-3.19.4 packages.
- - If you prefer to run the unit tests, remove skipTests from the
- command above.
- - If you use Eclipse IDE, you can import the projects using
- "Import/Maven/Existing Maven Projects". Eclipse does not
- automatically generate Java sources or include the generated
- sources into the projects. Please build using maven as described
- above and then use Project Properties to include
- "target/generatedsources/java" as a source directory into the
- "Java Build Path" for these projects: tez-api, tez-mapreduce,
- tez-runtime-internals and tez-runtime-library. This needs to be done
- just once after importing the project.
-3. Copy the relevant tez tarball into HDFS, and configure tez-site.xml
- - A tez tarball containing tez and hadoop libraries will be found
- at tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz
- - Assuming that the tez jars are put in /apps/ on HDFS, the
- command would be
+ - You need to change the value of the hadoop.version property in the
+ top-level pom.xml to match the version of the hadoop branch being used.
+ ```bash
+ hadoop version
```
+
+2. Build tez using
+ `mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true`
+
+ - This assumes that you have already installed JDK 21 or later and Maven 3
+ or later.
+ - Tez also requires Protocol Buffers 3.25.5. Note that the
+ `protoc-jar-maven-plugin` used in the build can automatically download
+ and use an embedded `protoc` compiler for most platforms. If you need
+ to use a specific `protoc` binary, you can specify its path using
+ `-Dprotoc.path=/path/to/protoc`.
+ - If you prefer to run the unit tests, remove skipTests from the
+ command above.
+ - If you use Eclipse IDE, you can import the projects using
+ "Import/Maven/Existing Maven Projects". Eclipse does not
+ automatically generate Java sources or include the generated
+ sources into the projects. Please build using maven as described
+ above and then use Project Properties to include
+ "target/generatedsources/java" as a source directory into the
+ "Java Build Path" for these projects: tez-api, tez-mapreduce,
+ tez-runtime-internals and tez-runtime-library. This needs to be done
+ just once after importing the project.
+
+3. Copy the relevant tez tarball into HDFS, and configure tez-site.xml
+
+ - A tez tarball containing tez and hadoop libraries will be found
+ at tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz
+ - Assuming that the tez jars are put in /apps/ on HDFS, the
+ command would be:
+
+ ```bash
hadoop fs -mkdir /apps/tez-x.y.z-SNAPSHOT
- hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz
/apps/tez-x.y.z-SNAPSHOT/
+ hadoop fs -copyFromLocal \
+ tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz /apps/tez-x.y.z-SNAPSHOT/
```
- - tez-site.xml configuration.
- - Set tez.lib.uris to point to the tar.gz uploaded to HDFS.
- Assuming the steps mentioned so far were followed,
- set tez.lib.uris to
`${fs.defaultFS}/apps/tez-x.y.z-SNAPSHOT/tez-x.y.z-SNAPSHOT.tar.gz`
- - Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml,
- or if it is set, the value should be false
- - Please note that the tarball version should match the version of
- the client jars used when submitting Tez jobs to the cluster.
- Please refer to the [Version Compatibility
Guide](https://cwiki.apache.org/confluence/display/TEZ/Version+Compatibility)
- for more details on version compatibility and detecting mismatches.
-4. Optional: If running existing MapReduce jobs on Tez. Modify
- mapred-site.xml to change "mapreduce.framework.name" property from
- its default value of "yarn" to "yarn-tez"
-5. Configure the client node to include the tez-libraries in the hadoop
- classpath
- - Extract the tez minimal tarball created in step 2 to a local directory
- (assuming TEZ_JARS is where the files will be decompressed for
- the next steps)
-
- ```
+ - tez-site.xml configuration.
+ - Set tez.lib.uris to point to the tar.gz uploaded to HDFS.
+ Assuming the steps mentioned so far were followed, set
+ tez.lib.uris to
+ `${fs.defaultFS}/apps/tez-x.y.z-SNAPSHOT/tez-x.y.z-SNAPSHOT.tar.gz`
+ - Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml,
+ or if it is set, the value should be false
+ - Please note that the tarball version should match the version of
+ the client jars used when submitting Tez jobs to the cluster.
+ Please refer to the [Version Compatibility Guide][1]
+ for more details on version compatibility and detecting mismatches.
+
+4. Optional: If running existing MapReduce jobs on Tez. Modify
+ mapred-site.xml to change `mapreduce.framework.name` property from
+ its default value of "yarn" to "yarn-tez"
+
+5. Configure the client node to include the tez-libraries in the hadoop
+ classpath
+
+ - Extract the tez minimal tarball created in step 2 to a local directory
+ (assuming TEZ_JARS is where the files will be decompressed for
+ the next steps)
+
+ ```bash
tar -xvzf tez-dist/target/tez-x.y.z-minimal.tar.gz -C $TEZ_JARS
```
- - set TEZ_CONF_DIR to the location of tez-site.xml
- - Add $TEZ_CONF_DIR, ${TEZ_JARS}/* and ${TEZ_JARS}/lib/* to the
application classpath.
- For example, doing it via the standard Hadoop tool chain would use the
following command
- to set up the application classpath:
+ - set TEZ_CONF_DIR to the location of tez-site.xml
+ - Add $TEZ_CONF_DIR, ${TEZ_JARS}/*and ${TEZ_JARS}/lib/* to the
+ application classpath.
+ - For example, doing it via the standard Hadoop tool chain would use
+ the following command to set up the application classpath:
- ```
+ ```bash
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
```
- - Please note the "*" which is an important requirement when
- setting up classpaths for directories containing jar files.
-6. There is a basic example of using an MRR job in the tez-examples.jar.
- Refer to OrderedWordCount.java in the source code. To run this
- example:
+ - Please note the "*" which is an important requirement when
+ setting up classpaths for directories containing jar files.
- ```
- $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount <input>
<output>
+6. There is a basic example of using an MRR job in the tez-examples.jar.
+ Refer to OrderedWordCount.java in the source code. To run this
+ example:
+
+ ```bash
+ $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar \
+ orderedwordcount <input> <output>
```
This will use the TEZ DAG ApplicationMaster to run the ordered word
@@ -113,147 +135,188 @@ or higher.
and handling multiple input-output pairs. You can use it to run
multiple DAGs serially on different inputs/outputs.
- ```
- $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount <input1>
<output1> <input2> <output2> <input3> <output3> ...
+ ```bash
+ $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount \
+ <input1> <output1> <input2> <output2> <input3> <output3> ...
```
The above will run multiple DAGs for each input-output pair.
To use TEZ sessions, set -DUSE_TEZ_SESSION=true
+ ```bash
+ $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount \
+ -DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
```
- $HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount
-DUSE_TEZ_SESSION=true <input1> <output1> <input2> <output2>
- ```
-7. Submit a MR job as you normally would using something like:
- ```
- $HADOOP_PREFIX/bin/hadoop jar
hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m
1 -r 1
+ > **Note:** The [tez_run_example.sh][2] script mentioned above can also
+ be used to automatically set up a local environment and run the
+ `orderedwordcount` example.
+
+7. Submit a MR job as you normally would using something like:
+
+ ```bash
+ $HADOOP_PREFIX/bin/hadoop jar \
+ hadoop-mapreduce-client-jobclient-3.5.0-SNAPSHOT-tests.jar sleep \
+ -mt 1 -rt 1 -m 1 -r 1
```
This will use the TEZ DAG ApplicationMaster to run the MR job. This
- can be verified by looking at the AM’s logs from the YARN ResourceManager
UI.
- This needs mapred-site.xml to have "mapreduce.framework.name" set to
"yarn-tez"
-
-Various ways to configure tez.lib.uris
----------------------------------------
-
-The `tez.lib.uris` configuration property supports a comma-separated list of
values. The
-types of values supported are:
- - Path to simple file
- - Path to a directory
- - Path to a compressed archive ( tarball, zip, etc).
-
-For simple files and directories, Tez will add all these files and first-level
entries in the
-directories (recursive traversal of dirs is not supported) into the working
directory of the
-Tez runtime and they will automatically be included into the classpath. For
archives i.e.
-files whose names end with generally known compressed archive suffixes such as
'tgz',
-'tar.gz', 'zip', etc. will be uncompressed into the container working
directory too. However,
-given that the archive structure is not known to the Tez framework, the user
is expected to
-configure `tez.lib.uris.classpath` to ensure that the nested directory
structure of an
-archive is added to the classpath. This classpath values should be relative
i.e. the entries
-should start with "./".
-
-Hadoop Installation dependent Install/Deploy Instructions
----------------------------------------------------------
-
-The above install instructions use Tez with pre-packaged Hadoop libraries
included in the package and is the
-recommended method for installation. A full tarball with all dependencies is a
better approach to ensure
-that existing jobs continue to run during a cluster's rolling upgrade.
-
-Although the `tez.lib.uris` configuration options enable a wide variety of
usage patterns, there
-are 2 main alternative modes that are supported by the framework:
-
-1. Mode A: Using a tez tarball on HDFS along with Hadoop libraries available
on the cluster.
+ can be verified by looking at the AM’s logs from the YARN
+ ResourceManager UI.
+ This needs mapred-site.xml to have `mapreduce.framework.name` set to
+ "yarn-tez"
+
+## Various ways to configure tez.lib.uris
+
+The `tez.lib.uris` configuration property supports a comma-separated list of
+values. The types of values supported are:
+
+- Path to simple file
+- Path to a directory
+- Path to a compressed archive ( tarball, zip, etc).
+
+For simple files and directories, Tez will add all these files and first-level
+entries in the directories (recursive traversal of dirs is not supported) into
+the working directory of the Tez runtime and they will automatically be
included
+into the classpath. For archives i.e. files whose names end with generally
known
+compressed archive suffixes such as 'tgz', 'tar.gz', 'zip', etc. will be
+uncompressed into the container working directory too. However, given that the
+archive structure is not known to the Tez framework, the user is expected to
+configure `tez.lib.uris.classpath` to ensure that the nested directory
structure
+of an archive is added to the classpath. This classpath values should be
+relative i.e. the entries should start with "./".
+
+## Hadoop Installation dependent Install/Deploy Instructions
+
+The above install instructions use Tez with pre-packaged Hadoop libraries
+included in the package and is the recommended method for installation.
+A full tarball with all dependencies is a better approach to ensure that
+existing jobs continue to run during a cluster's rolling upgrade.
+
+Although the `tez.lib.uris` configuration options enable a wide variety of
+usage patterns, there are 2 main alternative modes that are supported by the
+framework:
+
+1. Mode A: Using a tez tarball on HDFS along with Hadoop libraries available on
+ the cluster.
2. Mode B: Using a tez tarball along with the Hadoop tarball.
-
-Both these modes will require a tez build without Hadoop dependencies and that
is available at
-tez-dist/target/tez-x.y.z-minimal.tar.gz.
-For Mode A: Tez tarball with using existing cluster Hadoop libraries by
leveraging yarn.application.classpath
--------------------------------------------------------------------------------------------------------------
+Both these modes will require a tez build without Hadoop dependencies and that
+is available at tez-dist/target/tez-x.y.z-minimal.tar.gz.
-This mode is not recommended for clusters that use rolling upgrades.
Additionally, it is the user's responsibility
-to ensure that the tez version being used is compatible with the version of
Hadoop running on the cluster.
-Step 3 above changes as follows. Also subsequent steps should use
tez-dist/target/tez-x.y.z-minimal.tar.gz
-instead of tez-dist/target/tez-x.y.z.tar.gz
+### Mode A: Using cluster Hadoop libraries via yarn.application.classpath
- - A tez build without Hadoop dependencies will be available at
tez-dist/target/tez-x.y.z-minimal.tar.gz
- Assuming that the tez jars are put in /apps/ on HDFS, the command would be
+This mode is not recommended for clusters that use rolling upgrades.
+Additionally, it is the user's responsibility to ensure that the tez version
+being used is compatible with the version of Hadoop running on the cluster.
+Step 3 above changes as follows. Also subsequent steps should use
+tez-dist/target/tez-x.y.z-minimal.tar.gz instead of
+tez-dist/target/tez-x.y.z.tar.gz
- ```
- "hadoop fs -mkdir /apps/tez-x.y.z"
- "hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal.tar.gz
/apps/tez-x.y.z"
+- A tez build without Hadoop dependencies will be available at
+ tez-dist/target/tez-x.y.z-minimal.tar.gz Assuming that the tez jars are put
+ in /apps/ on HDFS, the command would be
+
+ ```bash
+ hadoop fs -mkdir /apps/tez-x.y.z
+ hadoop fs -copyFromLocal \
+ tez-dist/target/tez-x.y.z-minimal.tar.gz /apps/tez-x.y.z
```
- - tez-site.xml configuration
- - Set tez.lib.uris to point to the paths in HDFS containing the tez jars.
Assuming the steps mentioned so far were followed,
-set tez.lib.uris to `${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz`
- - Set tez.use.cluster.hadoop-libs to true
+- tez-site.xml configuration
+ - Set tez.lib.uris to point to the paths in HDFS containing the tez jars.
+ Assuming the steps mentioned so far were followed, set tez.lib.uris to
+ `${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz`
+ - Set tez.use.cluster.hadoop-libs to true
-For Mode B: Tez tarball with Hadoop tarball
---------------------------------------------
+### For Mode B: Tez tarball with Hadoop tarball
-This mode will support rolling upgrades. It is the user's responsibility to
ensure that the
-versions of Tez and Hadoop being used are compatible.
-To do this configuration, we need to change Step 3 of the
-default instructions in the following ways.
+This mode will support rolling upgrades. It is the user's responsibility to
+ensure that the versions of Tez and Hadoop being used are compatible. To do
this
+configuration, we need to change Step 3 of the default instructions in the
+following ways.
- - Assuming that the tez archives/jars are put in /apps/ on HDFS, the command
to put this
-minimal Tez archive into HDFS would be:
+- Assuming that the tez archives/jars are put in /apps/ on HDFS, the command to
+ put this minimal Tez archive into HDFS would be:
- ```
- "hadoop fs -mkdir /apps/tez-x.y.z"
- "hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal.tar.gz
/apps/tez-x.y.z"
- ```
+ ```bash
+ hadoop fs -mkdir /apps/tez-x.y.z
+ hadoop fs -copyFromLocal \
+ tez-dist/target/tez-x.y.z-minimal.tar.gz /apps/tez-x.y.z
+ ```
- - Alternatively, you can put the minimal directory directly into HDFS and
+- Alternatively, you can put the minimal directory directly into HDFS and
reference the jars, instead of using an archive. The command to put
the minimal directory into HDFS would be:
- ```
- "hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal/*
/apps/tez-x.y.z"
- ```
+ ```bash
+ hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal/*
/apps/tez-x.y.z
+ ```
- - After building hadoop, the hadoop tarball will be available at
+- After building hadoop, the hadoop tarball will be available at
hadoop/hadoop-dist/target/hadoop-x.y.z-SNAPSHOT.tar.gz
- - Assuming that the hadoop jars are put in /apps/ on HDFS, the command to
put this
- Hadoop archive into HDFS would be:
-
- ```
- "hadoop fs -mkdir /apps/hadoop-x.y.z"
- "hadoop fs -copyFromLocal hadoop-dist/target/hadoop-x.y.z-SNAPSHOT.tar.gz
/apps/hadoop-x.y.z"
- ```
-
- - tez-site.xml configuration
- - Set tez.lib.uris to point to the the archives and jars that are needed
for Tez/Hadoop.
-
- - Example: When using both Tez and Hadoop archives, set tez.lib.uris to
-
`${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz#tez,${fs.defaultFS}/apps/hadoop-x.y.z/hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce`
-
- - Example: When using Tez jars with a Hadoop archive, set tez.lib.uris to:
-
`${fs.defaultFS}/apps/tez-x.y.z,${fs.defaultFS}/apps/tez-x.y.z/lib,${fs.defaultFS}/apps/hadoop-x.y.z/hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce`
-
- - In tez.lib.uris, the text immediately following the '#' symbol is the
fragment that
- refers to the symlink that will be created for the archive. If no
fragment is given,
- the symlink will be set to the name of the archive. Fragments should not
be given
- to directories or jars.
-
- - If any archives are specified in tez.lib.uris, then
tez.lib.uris.classpath must be set
- to define the classpath for these archives as the archive structure is
not known.
- - Example: Classpath when using both Tez and Hadoop archives, set
tez.lib.uris.classpath to:
+- Assuming that the hadoop jars are put in /apps/ on HDFS, the command to put
+ this Hadoop archive into HDFS would be:
+ ```bash
+ hadoop fs -mkdir /apps/hadoop-x.y.z
+ hadoop fs -copyFromLocal \
+ hadoop-dist/target/hadoop-x.y.z-SNAPSHOT.tar.gz /apps/hadoop-x.y.z
```
-./tez/*:./tez/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/*:./hadoop-mapreduce/hadoo
[...]
- ```
-
- - Example: Classpath when using Tez jars with a Hadoop archive, set
tez.lib.uris.classpath to:
+- tez-site.xml configuration
+ - Set tez.lib.uris to point to the the archives and jars that are needed for
+ Tez/Hadoop.
+
+ - Example: When using both Tez and Hadoop archives, set tez.lib.uris to
+ `${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz#tez,`
+ `${fs.defaultFS}/apps/hadoop-x.y.z/` \
+ `hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce`
+
+ - Example: When using Tez jars with a Hadoop archive, set tez.lib.uris to:
+ `${fs.defaultFS}/apps/tez-x.y.z,${fs.defaultFS}/apps/tez-x.y.z/lib,`
+ `${fs.defaultFS}/apps/hadoop-x.y.z/` \
+ `hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce`
+
+ - In tez.lib.uris, the text immediately following the '#' symbol is the
+ fragment that refers to the symlink that will be created for the archive.
+ If no fragment is given, the symlink will be set to the name of the
archive.
+ Fragments should not be given to directories or jars.
+
+ - If any archives are specified in tez.lib.uris, then
+ tez.lib.uris.classpath must be set to define the classpath for these
+ archives as the archive structure is not known.
+ - Example: Classpath when using both Tez and Hadoop archives, set
+ tez.lib.uris.classpath to:
+
+ ```text
+ ./tez/*:./tez/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/lib/*
```
-./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/*:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/sha
[...]
- ```
-
-[Install instructions for older versions of Tez (pre
0.5.0)](./install_pre_0_5_0.html)
------------------------------------------------------------------------------------
+- Example: Classpath when using Tez jars with a Hadoop archive, set
+ tez.lib.uris.classpath to:
+
+ ```text
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/*:
+ ./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/lib/*
+ ```
+### [Instructions for older Tez versions (pre 0.5.0)](./install_pre_0_5_0.html)
+[1]: https://cwiki.apache.org/confluence/display/TEZ/Version+Compatibility
+[2]:
https://github.com/apache/tez/blob/master/dev-support/bin/tez_run_example.sh
diff --git a/pom.xml b/pom.xml
index 56bca92c7..bd1a268aa 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1488,7 +1488,6 @@
</profile>
</profiles>
- <!-- TODO: TEZ-4597: IncludePublicAnnotationsStandardDoclet is not JDK9+
compatible -->
<reporting>
<plugins>
<plugin>
diff --git a/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java
b/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java
index 15dbd969d..9e6338ddd 100644
--- a/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java
+++ b/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java
@@ -1077,7 +1077,7 @@ public final class TezClientUtils {
amOpts = amOpts + tezConf.get(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT);
- amOpts = amOpts +
TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT;
+ amOpts = amOpts + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT;
amOpts = maybeAddDefaultMemoryJavaOpts(amOpts, capability,
tezConf.getDouble(TezConfiguration.TEZ_CONTAINER_MAX_JAVA_HEAP_FRACTION,
TezConfiguration.TEZ_CONTAINER_MAX_JAVA_HEAP_FRACTION_DEFAULT));
diff --git a/tez-api/src/main/java/org/apache/tez/common/JavaOptsChecker.java
b/tez-api/src/main/java/org/apache/tez/common/JavaOptsChecker.java
index e9f7cb03a..04f2c9f73 100644
--- a/tez-api/src/main/java/org/apache/tez/common/JavaOptsChecker.java
+++ b/tez-api/src/main/java/org/apache/tez/common/JavaOptsChecker.java
@@ -65,15 +65,6 @@ public class JavaOptsChecker {
}
if (gcOpts.size() > 1) {
- // Handle special case for " -XX:+UseParNewGC -XX:+UseConcMarkSweepGC "
- // which can be specified together.
- if (gcOpts.size() == 2) {
- if (gcOpts.contains("UseParNewGC")
- && gcOpts.contains("UseConcMarkSweepGC")) {
- return;
- }
- }
-
LOG.debug("Found clashing GC opts, conflicting GC Values={}", gcOpts);
throw new TezException("Invalid/conflicting GC options found,"
diff --git a/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java
b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java
index 0c74f1bf2..d3980c7c3 100644
--- a/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java
+++ b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java
@@ -575,8 +575,6 @@ public final class TezCommonUtils {
public static int getJavaVersion() {
String javaVersionString = System.getProperty("java.version");
- return javaVersionString.split("\\.")[0].equals("1")
- ? Integer.parseInt(javaVersionString.split("\\.")[1]) // "1.8" -> 8
- : Integer.parseInt(javaVersionString.split("\\.")[0]); // "9.x" -> 9,
"11.x" -> 11
+ return Integer.parseInt(javaVersionString.split("\\.")[0]); // "21.x" -> 21
}
}
diff --git a/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
b/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
index a19700d85..0f7d6d375 100644
--- a/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
+++ b/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
@@ -33,7 +33,6 @@ import
org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.Shell;
import org.apache.hadoop.yarn.api.records.LocalResource;
-import org.apache.tez.common.TezCommonUtils;
import org.apache.tez.common.annotation.ConfigurationClass;
import org.apache.tez.common.annotation.ConfigurationProperty;
import org.apache.tez.dag.api.EdgeProperty.ConcurrentEdgeTriggerType;
@@ -400,7 +399,7 @@ public class TezConfiguration extends Configuration {
public static final String TEZ_AM_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS_DEFAULT =
"-server -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN";
- public static final String TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT =
+ public static final String TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT =
" --add-opens java.base/java.lang=ALL-UNNAMED";
/**
* String value. Command line options provided during the launch of the Tez
@@ -410,11 +409,8 @@ public class TezConfiguration extends Configuration {
@ConfigurationScope(Scope.AM)
@ConfigurationProperty
public static final String TEZ_AM_LAUNCH_CMD_OPTS = TEZ_AM_PREFIX +
"launch.cmd-opts";
- public static final String TEZ_AM_LAUNCH_CMD_OPTS_JDK8_DEFAULT =
- "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA
-XX:+UseParallelGC";
- public static final String TEZ_AM_LAUNCH_CMD_OPTS_JDK9_DEFAULT =
- "-verbose:gc -Xlog:gc*,safepoint::time,uptime -XX:+UseNUMA
-XX:+UseParallelGC";
- public static final String TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT;
+ public static final String TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT =
+ "-Xlog:gc*,safepoint::time,uptime -XX:+UseNUMA -XX:+UseParallelGC";
/**
* String value. Command line options which will be prepended to {@link
@@ -438,21 +434,9 @@ public class TezConfiguration extends Configuration {
@ConfigurationProperty
public static final String TEZ_TASK_LAUNCH_CMD_OPTS = TEZ_TASK_PREFIX
+ "launch.cmd-opts";
- public static final String TEZ_TASK_LAUNCH_CMD_OPTS_JDK8_DEFAULT =
- "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA
-XX:+UseParallelGC";
- public static final String TEZ_TASK_LAUNCH_CMD_OPTS_JDK9_DEFAULT =
- "-verbose:gc -Xlog:gc*,safepoint::time,uptime -XX:+UseNUMA
-XX:+UseParallelGC";
- public static final String TEZ_TASK_LAUNCH_CMD_OPTS_DEFAULT;
+ public static final String TEZ_TASK_LAUNCH_CMD_OPTS_DEFAULT =
+ "-Xlog:gc*,safepoint::time,uptime -XX:+UseNUMA -XX:+UseParallelGC";
- static {
- if (TezCommonUtils.getJavaVersion() >= 9) {
- TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT = TEZ_AM_LAUNCH_CMD_OPTS_JDK9_DEFAULT;
- TEZ_TASK_LAUNCH_CMD_OPTS_DEFAULT = TEZ_TASK_LAUNCH_CMD_OPTS_JDK9_DEFAULT;
- } else {
- TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT = TEZ_AM_LAUNCH_CMD_OPTS_JDK8_DEFAULT;
- TEZ_TASK_LAUNCH_CMD_OPTS_DEFAULT = TEZ_TASK_LAUNCH_CMD_OPTS_JDK8_DEFAULT;
- }
- }
/**
* Double value. Tez automatically determines the Xmx for the JVMs used to
run
diff --git
a/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java
b/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java
index 42ccb1a3f..2d18bc5d3 100644
--- a/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java
+++ b/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java
@@ -538,17 +538,17 @@ public class TestTezClientUtils {
assertEquals(tmpOpts + " "
+ TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS_DEFAULT + " "
+ amCommandOpts
- + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT,
+ + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT,
amOptsConstructed);
// Test2: Setup cluster-default command opts explicitly
String clusterDefaultCommandOpts =
- "-server -Djava.net.preferIPv4Stack=true -XX:+PrintGCDetails
-verbose:gc ";
+ "-server -Djava.net.preferIPv4Stack=true -Xlog:gc ";
tezConf.set(TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS,
clusterDefaultCommandOpts);
amOptsConstructed =
TezClientUtils.constructAMLaunchOpts(tezConf,
Resource.newInstance(1024, 1));
assertEquals(tmpOpts + " " + clusterDefaultCommandOpts + " " +
amCommandOpts
- + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT,
amOptsConstructed);
+ + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT,
amOptsConstructed);
// Test3: Don't setup Xmx explicitly
@@ -560,17 +560,17 @@ public class TestTezClientUtils {
// It's OK for the Xmx value to show up before cluster default options,
since Xmx will not be replaced if it already exists.
assertEquals(
" -Xmx" + ((int) (1024 * factor)) + "m" + " " + tmpOpts + " " +
clusterDefaultCommandOpts + " " +
- amCommandOpts +
TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT,
+ amCommandOpts +
TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT,
amOptsConstructed);
// Test4: Ensure admin options with Xmx does not cause them to be
overridden. This should almost never be done though.
clusterDefaultCommandOpts =
- "-server -Djava.net.preferIPv4Stack=true -XX:+PrintGCDetails
-verbose:gc -Xmx200m";
+ "-server -Djava.net.preferIPv4Stack=true -Xlog:gc -Xmx200m";
tezConf.set(TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS,
clusterDefaultCommandOpts);
amOptsConstructed =
TezClientUtils.constructAMLaunchOpts(tezConf,
Resource.newInstance(1024, 1));
assertEquals(tmpOpts + " " + clusterDefaultCommandOpts + " " +
amCommandOpts
- + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_JDK17_CMD_OPTS_DEFAULT,
amOptsConstructed);
+ + TezConfiguration.TEZ_AM_LAUNCH_CLUSTER_ADD_OPENS_DEFAULT,
amOptsConstructed);
}
@Test(timeout = 5000)
@@ -590,11 +590,10 @@ public class TestTezClientUtils {
// Test2: Setup cluster-default command opts explicitly
String taskClusterDefaultCommandOpts =
- "-server -Djava.net.preferIPv4Stack=true -XX:+PrintGCDetails
-verbose:gc ";
- tezConf.set(TezConfiguration.TEZ_TASK_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS,
- taskClusterDefaultCommandOpts);
- taskOptsConstructed =
- TezClientUtils.addDefaultsToTaskLaunchCmdOpts("", tezConf);
+ "-server -Djava.net.preferIPv4Stack=true -Xlog:gc ";
+ tezConf.set(
+ TezConfiguration.TEZ_TASK_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS,
taskClusterDefaultCommandOpts);
+ taskOptsConstructed = TezClientUtils.addDefaultsToTaskLaunchCmdOpts("",
tezConf);
expected = taskClusterDefaultCommandOpts + " " + taskCommandOpts;
assertTrue(
"Did not find Expected prefix: [" + expected + "] in string [" +
taskOptsConstructed +
diff --git
a/tez-api/src/test/java/org/apache/tez/common/TestJavaOptsChecker.java
b/tez-api/src/test/java/org/apache/tez/common/TestJavaOptsChecker.java
index c945c89b8..1fc510662 100644
--- a/tez-api/src/test/java/org/apache/tez/common/TestJavaOptsChecker.java
+++ b/tez-api/src/test/java/org/apache/tez/common/TestJavaOptsChecker.java
@@ -36,7 +36,7 @@ public class TestJavaOptsChecker {
@Test(timeout = 5000)
public void testMultipleGC() {
// Clashing GC values
- String opts = "-XX:+UseConcMarkSweepGC -XX:+UseG1GC -XX:+UseParallelGC ";
+ String opts = "-XX:+UseSerialGC -XX:+UseG1GC -XX:+UseParallelGC ";
try {
javaOptsChecker.checkOpts(opts);
Assert.fail("Expected check to fail with opts=" + opts);
@@ -49,7 +49,7 @@ public class TestJavaOptsChecker {
@Test(timeout = 5000)
public void testPositiveNegativeOpts() throws TezException {
// Multiple positive GC values
- String opts = "-XX:+UseConcMarkSweepGC -XX:+UseG1GC -XX:+UseParallelGC
-XX:-UseG1GC ";
+ String opts = "-XX:+UseSerialGC -XX:+UseG1GC -XX:+UseParallelGC
-XX:-UseG1GC ";
try {
javaOptsChecker.checkOpts(opts);
Assert.fail("Expected check to fail with opts=" + opts);
@@ -83,32 +83,8 @@ public class TestJavaOptsChecker {
javaOptsChecker.checkOpts(opts);
// Invalid negative can be ignored
- opts = " -XX:+UseG1GC -XX:+UseParallelGC -XX:-UseG1GC
-XX:-UseConcMarkSweepGC ";
+ opts = " -XX:+UseG1GC -XX:+UseParallelGC -XX:-UseG1GC -XX:-UseSerialGC ";
javaOptsChecker.checkOpts(opts);
-
- }
-
- @Test(timeout = 5000)
- public void testSpecialCaseNonConflictingGCOptions() throws TezException {
- String opts = " -XX:+UseParNewGC -XX:+UseConcMarkSweepGC ";
- javaOptsChecker.checkOpts(opts);
-
- opts += " -XX:+DisableExplicitGC ";
- javaOptsChecker.checkOpts(opts);
-
- opts += " -XX:-UseG1GC ";
- javaOptsChecker.checkOpts(opts);
-
- opts += " -XX:+UseG1GC ";
- try {
- javaOptsChecker.checkOpts(opts);
- Assert.fail("Expected check to fail with opts=" + opts);
- } catch (TezException e) {
- Assert.assertTrue(e.getMessage(),
- e.getMessage().contains("Invalid/conflicting GC options found"));
- }
-
-
}
}
diff --git a/tez-tools/tez-javadoc-tools/pom.xml
b/tez-tools/tez-javadoc-tools/pom.xml
index a3927227b..686829b1d 100644
--- a/tez-tools/tez-javadoc-tools/pom.xml
+++ b/tez-tools/tez-javadoc-tools/pom.xml
@@ -47,100 +47,6 @@
</dependency>
</dependencies>
- <profiles>
- <profile>
- <id>platform-macosx</id>
- <activation>
- <file>
- <exists>${java.home}/../Classes/classes.jar</exists>
- </file>
- </activation>
- <dependencies>
- <dependency>
- <groupId>com.sun</groupId>
- <artifactId>tools</artifactId>
- <version>${java.version}</version>
- <scope>system</scope>
- <systemPath>${java.home}/../Classes/classes.jar</systemPath>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>other-os</id>
- <activation>
- <file>
- <exists>${java.home}/../lib/tools.jar</exists>
- </file>
- </activation>
- <dependencies>
- <dependency>
- <groupId>com.sun</groupId>
- <artifactId>tools</artifactId>
- <version>${java.version}</version>
- <scope>system</scope>
- <systemPath>${java.home}/../lib/tools.jar</systemPath>
- </dependency>
- </dependencies>
- </profile>
-
- <profile>
- <id>java8-16</id>
- <activation>
- <jdk>[1.8,16]</jdk>
- </activation>
- <build>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>build-helper-maven-plugin</artifactId>
- <executions>
- <execution>
- <phase>generate-sources</phase>
- <goals>
- <goal>add-source</goal>
- </goals>
- <configuration>
- <sources>
- <source>src/main/java-8-16</source>
- </sources>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
-
- <profile>
- <id>java17</id>
- <activation>
- <jdk>[17,)</jdk>
- </activation>
- <build>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>build-helper-maven-plugin</artifactId>
- <executions>
- <execution>
- <phase>generate-sources</phase>
- <goals>
- <goal>add-source</goal>
- </goals>
- <configuration>
- <sources>
- <source>src/main/java-17</source>
- </sources>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
- </profiles>
-
-
<build>
<plugins>
<plugin>
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/package-info.java
deleted file mode 100644
index 10fcee1f4..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Data model types used by the Tez Javadoc doclets.
- */
-@Private
-package org.apache.tez.tools.javadoc.model;
-
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
deleted file mode 100644
index a67b9dbcd..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
+++ /dev/null
@@ -1,230 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.doclet;
-
-import java.io.IOException;
-import java.util.Map;
-
-import com.sun.javadoc.AnnotationDesc;
-import com.sun.javadoc.AnnotationDesc.ElementValuePair;
-import com.sun.javadoc.ClassDoc;
-import com.sun.javadoc.DocErrorReporter;
-import com.sun.javadoc.FieldDoc;
-import com.sun.javadoc.LanguageVersion;
-import com.sun.javadoc.RootDoc;
-import com.sun.tools.doclets.standard.Standard;
-
-import org.apache.hadoop.classification.InterfaceAudience.Private;
-import org.apache.hadoop.classification.InterfaceStability.Evolving;
-import org.apache.hadoop.classification.InterfaceStability.Unstable;
-import org.apache.tez.common.annotation.ConfigurationClass;
-import org.apache.tez.common.annotation.ConfigurationProperty;
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-import org.apache.tez.tools.javadoc.util.HtmlWriter;
-import org.apache.tez.tools.javadoc.util.XmlWriter;
-
-public final class ConfigStandardDoclet {
-
- private static final String DEBUG_SWITCH = "-debug";
- private static boolean debugMode = false;
-
- private ConfigStandardDoclet() {
- }
-
- public static LanguageVersion languageVersion() {
- return LanguageVersion.JAVA_1_5;
- }
-
- private static void logMessage(String message) {
- if (!debugMode) {
- return;
- }
- System.out.println(message);
- }
-
- public static boolean start(RootDoc root) {
- //look for debug flag
- for (String[] opts : root.options()) {
- for (String opt : opts) {
- if (opt.equals(DEBUG_SWITCH)) {
- debugMode = true;
- break;
- }
- }
- }
-
- logMessage("Running doclet " + ConfigStandardDoclet.class.getSimpleName());
- ClassDoc[] classes = root.classes();
- for (ClassDoc aClass : classes) {
- processDoc(aClass);
- }
-
- return true;
- }
-
- private static void processDoc(ClassDoc doc) {
- logMessage("Parsing : " + doc);
- if (!doc.isClass()) {
- logMessage("Ignoring non-class: " + doc);
- return;
- }
-
- AnnotationDesc[] annotations = doc.annotations();
- boolean isConfigClass = false;
- String templateName = null;
- for (AnnotationDesc annotation : annotations) {
- logMessage("Checking annotation: " + annotation.annotationType());
- if (annotation.annotationType().qualifiedTypeName().equals(
- ConfigurationClass.class.getName())) {
- isConfigClass = true;
- for (ElementValuePair element : annotation.elementValues()) {
- if (element.element().name().equals("templateFileName")) {
- templateName = stripQuotes(element.value().toString());
- }
- }
- break;
- }
- }
-
- if (!isConfigClass) {
- logMessage("Ignoring non-config class: " + doc);
- return;
- }
-
- logMessage("Processing config class: " + doc);
- Config config = new Config(doc.name(), templateName);
- Map<String, ConfigProperty> configProperties =
config.getConfigProperties();
-
- FieldDoc[] fields = doc.fields();
- for (FieldDoc field : fields) {
- if (field.isPrivate()) {
- logMessage("Skipping private field: " + field);
- continue;
- }
- if (!field.isStatic()) {
- logMessage("Skipping non-static field: " + field);
- continue;
- }
-
- if (field.name().endsWith("_PREFIX")) {
- logMessage("Skipping non-config prefix constant field: " + field);
- continue;
- }
- if (field.name().equals("TEZ_SITE_XML")) {
- logMessage("Skipping constant field: " + field);
- continue;
- }
-
- if (field.name().endsWith("_DEFAULT")) {
-
- String name = field.name().substring(0,
- field.name().lastIndexOf("_DEFAULT"));
- if (!configProperties.containsKey(name)) {
- configProperties.put(name, new ConfigProperty());
- }
- ConfigProperty configProperty = configProperties.get(name);
- if (field.constantValue() == null) {
- logMessage("Got null constant value"
- + ", name=" + name
- + ", field=" + field.name()
- + ", val=" + field.constantValueExpression());
- configProperty.setDefaultValue(field.constantValueExpression());
- } else {
- configProperty.setDefaultValue(field.constantValue().toString());
- }
- configProperty.setInferredType(field.type().simpleTypeName());
-
- if (name.equals("TEZ_AM_STAGING_DIR") &&
configProperty.getDefaultValue() != null) {
- String defaultValue = configProperty.getDefaultValue();
- defaultValue = defaultValue.replace(System.getProperty("user.name"),
"${user.name}");
- configProperty.setDefaultValue(defaultValue);
- }
-
- continue;
- }
-
- String name = field.name();
- if (!configProperties.containsKey(name)) {
- configProperties.put(name, new ConfigProperty());
- }
- ConfigProperty configProperty = configProperties.get(name);
- configProperty.setPropertyName(field.constantValue().toString());
-
- AnnotationDesc[] annotationDescs = field.annotations();
-
- for (AnnotationDesc annotationDesc : annotationDescs) {
-
- if (annotationDesc.annotationType().qualifiedTypeName().equals(
- Private.class.getCanonicalName())) {
- configProperty.setPrivate(true);
- }
- if (annotationDesc.annotationType().qualifiedTypeName().equals(
- Unstable.class.getCanonicalName())) {
- configProperty.setUnstable(true);
- }
- if (annotationDesc.annotationType().qualifiedTypeName().equals(
- Evolving.class.getCanonicalName())) {
- configProperty.setEvolving(true);
- }
- if (annotationDesc.annotationType().qualifiedTypeName().equals(
- ConfigurationProperty.class.getCanonicalName())) {
- configProperty.setValidConfigProp(true);
-
- for (ElementValuePair element : annotationDesc.elementValues()) {
- if (element.element().name().equals("type")) {
- configProperty.setType(stripQuotes(element.value().toString()));
- } else {
- logMessage("Unhandled annotation property: " +
element.element().name());
- }
- }
- }
-
- HtmlWriter writer = new HtmlWriter();
- try {
- writer.write(config);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- XmlWriter xmlWriter = new XmlWriter();
- try {
- xmlWriter.write(config);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
- }
- }
-
- private static String stripQuotes(String s) {
- if (s.charAt(0) == '"' && s.charAt(s.length() - 1) == '"') {
- return s.substring(1, s.length() - 1);
- }
- return s;
- }
-
- public static int optionLength(String option) {
- return Standard.optionLength(option);
- }
-
- public static boolean validOptions(String[][] options, DocErrorReporter
reporter) {
- return true;
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/package-info.java
deleted file mode 100644
index de1383847..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Custom Javadoc doclets for Tez (e.g. config documentation generator).
- */
-@Private
-package org.apache.tez.tools.javadoc.doclet;
-
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/Config.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/Config.java
deleted file mode 100644
index 0556a4e37..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/Config.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.model;
-
-import java.util.Map;
-import java.util.TreeMap;
-
-public class Config {
-
- private final String templateName;
- private final String configName;
- private Map<String, ConfigProperty> configProperties;
-
- public Config(String configName, String templateName) {
- this.configName = configName;
- this.templateName = templateName;
- this.setConfigProperties(new TreeMap<String, ConfigProperty>());
- }
-
- public String getTemplateName() {
- return templateName;
- }
-
- public String getConfigName() {
- return configName;
- }
-
- public Map<String, ConfigProperty> getConfigProperties() {
- return configProperties;
- }
-
- public void setConfigProperties(Map<String, ConfigProperty>
configProperties) {
- this.configProperties = configProperties;
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/ConfigProperty.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/ConfigProperty.java
deleted file mode 100644
index 27208de55..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/ConfigProperty.java
+++ /dev/null
@@ -1,115 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.model;
-
-public class ConfigProperty {
-
- private String propertyName;
- private String defaultValue;
- private String description;
- private String type = "string";
- private boolean isPrivate = false;
- private boolean isUnstable = false;
- private boolean isEvolving = false;
- private boolean isValidConfigProp = false;
- private String inferredType;
-
- public String getPropertyName() {
- return propertyName;
- }
-
- public void setPropertyName(String propertyName) {
- this.propertyName = propertyName;
- }
-
- public String getDefaultValue() {
- return defaultValue;
- }
-
- public void setDefaultValue(String defaultValue) {
- this.defaultValue = defaultValue;
- }
-
- public String getDescription() {
- return description;
- }
-
- public void setDescription(String description) {
- this.description = description;
- }
-
- public String getType() {
- return type;
- }
-
- public void setType(String type) {
- this.type = type;
- }
-
- public boolean isPrivate() {
- return isPrivate;
- }
-
- public void setPrivate(boolean aPrivate) {
- isPrivate = aPrivate;
- }
-
- public boolean isUnstable() {
- return isUnstable;
- }
-
- public void setUnstable(boolean unstable) {
- isUnstable = unstable;
- }
-
- public boolean isEvolving() {
- return isEvolving;
- }
-
- public void setEvolving(boolean evolving) {
- isEvolving = evolving;
- }
-
- public boolean isValidConfigProp() {
- return isValidConfigProp;
- }
-
- public void setValidConfigProp(boolean validConfigProp) {
- isValidConfigProp = validConfigProp;
- }
-
- public String getInferredType() {
- return inferredType;
- }
-
- public void setInferredType(String inferredType) {
- this.inferredType = inferredType;
- }
-
- @Override
- public String toString() {
- return "name=" + getPropertyName()
- + ", defaultValue=" + getDefaultValue()
- + ", description=" + getDescription()
- + ", type=" + getType()
- + ", inferredType=" + getInferredType()
- + ", private=" + isPrivate()
- + ", isConfigProp=" + isValidConfigProp();
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/HtmlWriter.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/HtmlWriter.java
deleted file mode 100644
index 548908c8e..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/HtmlWriter.java
+++ /dev/null
@@ -1,160 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.util;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.io.PrintWriter;
-
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-
-public class HtmlWriter extends Writer {
-
- private static final String DEFAULT_STYLESHEET = "default-stylesheet.css";
-
- public void write(Config config) throws IOException {
- PrintWriter out = null;
-
- if (config.getConfigName() == null || config.getConfigName().isEmpty()) {
- throw new RuntimeException("Config Name is null or empty");
- }
-
- try {
- File file = new File(config.getConfigName() + ".html");
- out = new PrintWriter(new OutputStreamWriter(new FileOutputStream(file),
"UTF-8"));
-
- out.println("<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
standalone=\"no\" ?>");
- out.println("<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0
Strict//EN\"");
- out.println("
\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\";>");
-
- out.println("<html xmlns=\"http://www.w3.org/1999/xhtml\";>");
-
- out.println("<head>");
- out.println("<meta http-equiv=\"Content-Type\" content=\"text/html;
charset=ISO-8859-1\" />");
- out.println("<title>" + config.getConfigName() + "</title>");
- out.println("</head>");
-
- out.println("<style>");
- out.println("table");
- out.println("{");
- out.println(" background-color: #000;");
- out.println(" border-spacing: 1px;");
- out.println(" margin: 0 auto 0 auto;");
- out.println("}");
- out.println("th");
- out.println("{");
- out.println(" background-color: #fff;");
- out.println(" padding: 5px;");
- out.println(" margin: 1px;");
- out.println("}");
- out.println("td");
- out.println("{");
- out.println(" background-color: #fff;");
- out.println(" padding: 2px;");
- out.println("}");
- out.println("table, th, td {");
- out.println(" border: 1px solid black;");
- out.println("}");
- out.println("tr.tr_private td");
- out.println("{");
- out.println(" background-color: #FF4500;");
- out.println(" padding: 2px;");
- out.println("}");
- out.println("tr.tr_evolve_unstable td");
- out.println("{");
- out.println(" background-color: #FFFFE0;");
- out.println(" padding: 2px;");
- out.println("}");
- out.println("th.th_private");
- out.println("{");
- out.println(" background-color: #FF4500;");
- out.println("}");
- out.println("th.th_evolve_unstable");
- out.println("{");
- out.println(" background-color: #FFFFE0;");
- out.println("}");
-
- out.println("</style>");
-
- out.println("<body>");
-
- out.println("<div id=\"wrapper\">");
- out.println("<div id=\"container\">");
-
- out.println("<h1>" + config.getConfigName() + "</h1>");
- out.println("<hr />");
-
- out.println("<table>");
- out.println("<tr>");
- out.println("<th>" + "Property Name" + "</th>");
- out.println("<th>" + "Default Value" + "</th>");
- out.println("<th>" + "Description" + "</th>");
- out.println("<th>" + "Type" + "</th>");
- // out.println("<th>" + "Valid Values" + "</th>");
- out.println("<th class=\"th_private\">" + "Is Private?" + "</th>");
- out.println("<th class=\"th_evolve_unstable\">" + "Is Unstable?" +
"</th>");
- out.println("<th class=\"th_evolve_unstable\">" + "Is Evolving?" +
"</th>");
- out.println("</tr>");
-
- for (ConfigProperty configProperty :
config.getConfigProperties().values()) {
- if (!isValidConfigProperty(configProperty)) {
- continue;
- }
-
- String altClass = "";
- if (configProperty.isPrivate()) {
- altClass = "class=\"tr_private\"";
- } else if (configProperty.isEvolving() || configProperty.isUnstable())
{
- altClass = "class=\"tr_evolve_unstable\"";
- }
-
- out.println("<tr " + altClass + ">");
- out.println("<td>" + configProperty.getPropertyName() + "</td>");
- out.println("<td>" + configProperty.getDefaultValue() + "</td>");
- out.println("<td>" + configProperty.getDescription() + "</td>");
- out.println("<td>" + configProperty.getType() + "</td>");
-
- out.println(
- "<td class=\"" + (configProperty.isPrivate() ? "td_private_true" :
"td_private_false")
- + "\">" + configProperty.isPrivate() + "</td>");
- out.println(
- "<td class=\"" + (configProperty.isEvolving() ? "td_evolve_true" :
"td_evolve_false") + "\">"
- + configProperty.isEvolving() + "</td>");
- out.println(
- "<td class=\"" + (configProperty.isUnstable() ? "td_unstable_true"
: "td_unstable_false") + "\">"
- + configProperty.isUnstable() + "</td>");
- out.println("</tr>");
- }
-
- out.println("</table>");
-
- out.println("</div>");
- out.println("</div>");
- out.println("</body>");
- out.println("</html>");
- } finally {
- if (out != null) {
- out.close();
- }
- }
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/Writer.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/Writer.java
deleted file mode 100644
index ca8ff80d9..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/Writer.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.util;
-
-import java.io.IOException;
-
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-
-public abstract class Writer {
-
- public abstract void write(Config config) throws IOException;
-
- public boolean isValidConfigProperty(ConfigProperty configProperty) {
- if (!configProperty.isValidConfigProp()) {
- return false;
- }
- if (configProperty.getPropertyName() == null ||
configProperty.getPropertyName().isEmpty()) {
- return false;
- }
- return true;
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/XmlWriter.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/XmlWriter.java
deleted file mode 100644
index f2586f7cd..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/XmlWriter.java
+++ /dev/null
@@ -1,107 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.tez.tools.javadoc.util;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.io.OutputStreamWriter;
-import java.io.PrintWriter;
-
-import org.apache.commons.lang3.StringEscapeUtils;
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-
-import com.google.common.io.ByteStreams;
-
-public class XmlWriter extends Writer {
-
- public void write(Config config) throws IOException {
- PrintWriter out = null;
-
- if (config.getConfigName() == null || config.getConfigName().isEmpty()) {
- throw new RuntimeException("Config Name is null or empty");
- }
-
- String fileName = config.getTemplateName() == null ||
- config.getTemplateName().isEmpty() ? config.getConfigName() :
config.getTemplateName();
- if (!fileName.endsWith(".xml")) {
- fileName += ".xml";
- }
-
- try {
- File file = new File(fileName);
- writeApacheHeader(file);
-
- out = new PrintWriter(new OutputStreamWriter(new FileOutputStream(file,
true), "UTF-8"));
-
- out.println("<?xml-stylesheet type=\"text/xsl\"
href=\"configuration.xsl\"?>");
- out.println();
- out.println("<!-- WARNING: THIS IS A GENERATED TEMPLATE PURELY FOR
DOCUMENTATION PURPOSES");
- out.println(" AND SHOULD NOT BE USED AS A CONFIGURATION FILE FOR TEZ
-->");
- out.println();
- out.println("<configuration>");
-
- for (ConfigProperty configProperty :
config.getConfigProperties().values()) {
- if (!isValidConfigProperty(configProperty)) {
- continue;
- }
- out.println();
- out.println(" <property>");
- out.println(" <name>" + configProperty.getPropertyName() +
"</name>");
- if (configProperty.getDefaultValue() != null &&
!configProperty.getDefaultValue().isEmpty()) {
- out.println(" <defaultValue>" + configProperty.getDefaultValue()
+ "</defaultValue>");
- }
- if (configProperty.getDescription() != null &&
!configProperty.getDescription().isEmpty()) {
- out.println(" <description>"
- + StringEscapeUtils.escapeXml(configProperty.getDescription()) +
"</description>");
- }
- if (configProperty.getType() != null &&
!configProperty.getType().isEmpty()) {
- out.println(" <type>" + configProperty.getType() + "</type>");
- }
- if (configProperty.isUnstable()) {
- out.println(" <unstable>true</unstable>");
- }
- if (configProperty.isEvolving()) {
- out.println(" <evolving>true</evolving>");
- }
- if (configProperty.isPrivate()) {
- out.println(" <private>true</private>");
- }
- out.println(" </property>");
- }
-
- out.println();
- out.println("</configuration>");
- } finally {
- if (out != null) {
- out.close();
- }
- }
- }
-
- private void writeApacheHeader(File file) throws IOException {
- try (InputStream in =
this.getClass().getClassLoader().getResourceAsStream("apache-licence.xml.header");
- OutputStream out = new FileOutputStream(file)) {
- ByteStreams.copy(in, out);
- }
- }
-}
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/package-info.java
deleted file mode 100644
index 1c2fed428..000000000
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Utility classes for the Tez Javadoc doclets.
- */
-@Private
-package org.apache.tez.tools.javadoc.util;
-
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
similarity index 96%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
index bd321475c..1691c43c9 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,30 +7,24 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.doclet;
-import com.sun.source.util.DocTrees;
-import jdk.javadoc.doclet.Doclet;
-import jdk.javadoc.doclet.DocletEnvironment;
-import jdk.javadoc.doclet.Reporter;
-import org.apache.hadoop.classification.InterfaceAudience.Private;
-import org.apache.hadoop.classification.InterfaceStability.Evolving;
-import org.apache.hadoop.classification.InterfaceStability.Unstable;
-import org.apache.tez.common.annotation.ConfigurationClass;
-import org.apache.tez.common.annotation.ConfigurationProperty;
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-import org.apache.tez.tools.javadoc.util.HtmlWriter;
-import org.apache.tez.tools.javadoc.util.XmlWriter;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.AnnotationMirror;
@@ -42,13 +36,22 @@ import javax.lang.model.element.Modifier;
import javax.lang.model.element.TypeElement;
import javax.lang.model.element.VariableElement;
import javax.tools.Diagnostic;
-import java.io.IOException;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.tez.common.annotation.ConfigurationClass;
+import org.apache.tez.common.annotation.ConfigurationProperty;
+import org.apache.tez.tools.javadoc.model.Config;
+import org.apache.tez.tools.javadoc.model.ConfigProperty;
+import org.apache.tez.tools.javadoc.util.HtmlWriter;
+import org.apache.tez.tools.javadoc.util.XmlWriter;
+
+import com.sun.source.util.DocTrees;
+
+import jdk.javadoc.doclet.Doclet;
+import jdk.javadoc.doclet.DocletEnvironment;
+import jdk.javadoc.doclet.Reporter;
public class ConfigStandardDoclet implements Doclet {
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/package-info.java
similarity index 99%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/package-info.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/package-info.java
index de1383847..7d2ec087a 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/doclet/package-info.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/doclet/package-info.java
@@ -22,4 +22,4 @@
@Private
package org.apache.tez.tools.javadoc.doclet;
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
+import org.apache.hadoop.classification.InterfaceAudience.Private;
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/Config.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/Config.java
similarity index 76%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/Config.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/Config.java
index d5121aaad..af36c4ed3 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/Config.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/Config.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,15 +7,15 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.model;
import java.util.Map;
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/ConfigProperty.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/ConfigProperty.java
similarity index 87%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/ConfigProperty.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/ConfigProperty.java
index 27208de55..653c50312 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/model/ConfigProperty.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/ConfigProperty.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,15 +7,15 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.model;
public class ConfigProperty {
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/package-info.java
similarity index 99%
rename from
tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/package-info.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/package-info.java
index 10fcee1f4..6b17b63d3 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/model/package-info.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/model/package-info.java
@@ -22,4 +22,4 @@
@Private
package org.apache.tez.tools.javadoc.model;
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
+import org.apache.hadoop.classification.InterfaceAudience.Private;
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/HtmlWriter.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/HtmlWriter.java
similarity index 93%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/HtmlWriter.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/HtmlWriter.java
index 9c512ee7f..a5b1dd295 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/HtmlWriter.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/HtmlWriter.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,23 +7,23 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.util;
-import org.apache.tez.tools.javadoc.model.Config;
-import org.apache.tez.tools.javadoc.model.ConfigProperty;
-
import java.io.*;
import java.nio.charset.StandardCharsets;
+import org.apache.tez.tools.javadoc.model.Config;
+import org.apache.tez.tools.javadoc.model.ConfigProperty;
+
public class HtmlWriter extends Writer {
private static final String DEFAULT_STYLESHEET = "default-stylesheet.css";
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/Writer.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/Writer.java
similarity index 71%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/Writer.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/Writer.java
index 523f8cae9..c45ffe4b0 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/Writer.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/Writer.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,22 +7,22 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.util;
+import java.io.IOException;
+
import org.apache.tez.tools.javadoc.model.Config;
import org.apache.tez.tools.javadoc.model.ConfigProperty;
-import java.io.IOException;
-
public abstract class Writer {
public abstract void write(Config config) throws IOException;
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/XmlWriter.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/XmlWriter.java
similarity index 90%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/XmlWriter.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/XmlWriter.java
index ba4b80119..e1d88e824 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/XmlWriter.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/XmlWriter.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -7,15 +7,15 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
-
package org.apache.tez.tools.javadoc.util;
import java.io.File;
diff --git
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/package-info.java
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/package-info.java
similarity index 99%
rename from
tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/package-info.java
rename to
tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/package-info.java
index 1c2fed428..5495c566e 100644
---
a/tez-tools/tez-javadoc-tools/src/main/java-17/org/apache/tez/tools/javadoc/util/package-info.java
+++
b/tez-tools/tez-javadoc-tools/src/main/java/org/apache/tez/tools/javadoc/util/package-info.java
@@ -22,4 +22,4 @@
@Private
package org.apache.tez.tools.javadoc.util;
-import org.apache.hadoop.classification.InterfaceAudience.Private;
\ No newline at end of file
+import org.apache.hadoop.classification.InterfaceAudience.Private;
