zhangbutao commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900139319
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to
run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive
from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the
hadoop_home finger of the env script in HIVE4 to the path where the high
version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related
dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure
the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual
directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop
3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 ->
apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution
can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come
with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
+ </property>
+ <property>
+ <name>tez.lib.uris.classpath</name> <!--only use tez self lib-->
+ <value>$PWD/tezlib/*,$PWD/tezlib/lib/*</value>
+ </property>
+
+ <property>
+ <name>tez.am.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
Review Comment:
Can remove the property?
If this configuration is required, please avoid using the hdp version.
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to
run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive
from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the
hadoop_home finger of the env script in HIVE4 to the path where the high
version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related
dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure
the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual
directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop
3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 ->
apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution
can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come
with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
Review Comment:
Please use a correct hdfs path format, such as
`${fs.defaultFS}/apps/apache-tez-0.10.4-bin.tar.gz` (Refer to
https://tez.apache.org/install.html)
BTW, should the apache-tez-0.10.4-bin.tar.gz contain all high hadoop version
(3.3.6) related dependencies?
You can add a description for the property.
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to
run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive
from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the
hadoop_home finger of the env script in HIVE4 to the path where the high
version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related
dependencies(not tez minimal tarball),
Review Comment:
You can add more about how to` Compile TEZ to get tez.tar.gz which contains
all hadoop related dependencies`. Such as what mvn command can be used to
compile?
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to
run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive
from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the
hadoop_home finger of the env script in HIVE4 to the path where the high
version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related
dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure
the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual
directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop
3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 ->
apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution
can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come
with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
+ </property>
+ <property>
+ <name>tez.lib.uris.classpath</name> <!--only use tez self lib-->
+ <value>$PWD/tezlib/*,$PWD/tezlib/lib/*</value>
+ </property>
+
+ <property>
+ <name>tez.am.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
+
+ <property>
+ <name>tez.task.launch.env</name><!--Example, replace with actual
value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
Review Comment:
Can remove the property?
If this configuration is required, please avoid using the hdp version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
