This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hive-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 94f4894 deploy: 25291cb5dd5a01175f3c2772bf81723353dee12b
94f4894 is described below
commit 94f48943b0dad854d6d48112b1c5f4c42fd6c4f8
Author: zhangbutao <[email protected]>
AuthorDate: Wed Jan 8 13:53:11 2025 +0000
deploy: 25291cb5dd5a01175f3c2772bf81723353dee12b
---
docs/latest/index.xml | 2 +-
.../manual-installation_283118363/index.html | 63 ++++++++++++++++++++++
index.xml | 2 +-
3 files changed, 65 insertions(+), 2 deletions(-)
diff --git a/docs/latest/index.xml b/docs/latest/index.xml
index fc4131a..d6e30c3 100644
--- a/docs/latest/index.xml
+++ b/docs/latest/index.xml
@@ -206,7 +206,7 @@ UDFs xpath, xpath_short, xpath_int, xpath_long,
xpath_float, xpath_double, xpath
Type Postfix Example TINYINT Y 100Y SMALLINT S 100S BIGINT L 100L String types
String literals can be expressed with either single quotes (') or double quotes
(&quot;).</description></item><item><title>Apache Hive :
LLAP</title><link>https://hive.apache.org/docs/latest/llap_62689557/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/llap_62689557/</guid><description>Apache
Hive : LLAP Overview Persistent Daemon Execution Engine Query Fr [...]
For configuration of LLAP, see the LLAP Section of Configuration Properties.
Overview Hive has become significantly faster thanks to various features and
improvements that were built by the community in recent years, including Tez
and Cost-based-optimization.</description></item><item><title>Apache Hive :
Locking</title><link>https://hive.apache.org/docs/latest/locking_27362050/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/locking_27362050/</guid><description>Apache
Hive : Locking Hive Concurrency Model Hive Co [...]
-Managed (Internal) External Introduction This document lists some of the
differences between the two but the fundamental difference is that Hive assumes
that it owns the data for managed tables. That means that the data, its
properties and data layout will and can only be changed via Hive command. The
data still lives in a normal file system and nothing is stopping you from
changing it without telling Hive about
it.</description></item><item><title>Apache Hive : Manual Installation</titl
[...]
+Managed (Internal) External Introduction This document lists some of the
differences between the two but the fundamental difference is that Hive assumes
that it owns the data for managed tables. That means that the data, its
properties and data layout will and can only be changed via Hive command. The
data still lives in a normal file system and nothing is stopping you from
changing it without telling Hive about
it.</description></item><item><title>Apache Hive : Manual Installation</titl
[...]
1. Map Join Optimization 1.1 Using Distributed Cache to Propagate Hashtable
File 1.2 Removing JDBM 1.3 Performance Evaluation 2. Converting Join into Map
Join Automatically 2.1 New Join Execution Flow 2.2 Resolving the Join Operation
at Run Time 2.3 Backup Task 2.4 Performance Evaluation 1. Map Join Optimization
1.1 Using Distributed Cache to Propagate Hashtable File Previously, when 2
large data tables need to do a join, there will be 2 different Mappers to sort
these tables based on th [...]
Materialized views support is introduced in Hive 3.0.0.
Introduction This page documents the work done for the supporting materialized
views in Apache Hive.</description></item><item><title>Apache Hive :
Materialized views in
Hive</title><link>https://hive.apache.org/docs/latest/materialized-views-in-hive_283118346/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/materialized-views-in-hive_283118346/</guid><description>Apache
Hive : Materialized views in Hive Objectives Traditionally, one of t [...]
diff --git a/docs/latest/manual-installation_283118363/index.html
b/docs/latest/manual-installation_283118363/index.html
index e2cbd2f..f13f4e2 100644
--- a/docs/latest/manual-installation_283118363/index.html
+++ b/docs/latest/manual-installation_283118363/index.html
@@ -131,6 +131,7 @@ ASF
<li><a href=#extra-hadoop-configurations-to-make-everything-working>Extra
hadoop configurations to make everything working</a></li>
<li><a href=#installing-hive-from-a-tarball>Installing Hive from a
Tarball</a></li>
<li><a href=#installing-from-source-code>Installing from Source Code</a></li>
+<li><a
href=#installing-with-old-version-hadoop(greater-than-or-equal-3.1.0)>Installing
with old version hadoop(greater than or equal 3.1.0)</a></li>
<li><a href=#next-steps>Next Steps</a></li>
<li><a href=#beeline-cli>Beeline CLI</a></li>
<li><a href=#hive-metastore>Hive Metastore</a></li>
@@ -354,6 +355,68 @@ insert into test values ('Hello, from Hive!');
</ul>
<p>That directory should contain all the files necessary to run Hive. You can
run it from there or copy it to a different location, if you prefer.</p>
<p>From now, you can follow the steps described in the section Installing Hive
from a Tarball</p>
+<h2 id=installing-with-old-version-hadoopgreater-than-or-equal-310>Installing
with old version hadoop(greater than or equal 3.1.0)</h2>
+<p>Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.</p>
+<p>The steps are as follows:</p>
+<p>1.Compile TEZ to get tez.tar.gz which contains all hadoop related
dependencies(not tez minimal tarball),
+run <code>mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true -Pdist
-Paws -Pazure</code>.
+For more detail,see:<code>https://tez.apache.org/install.html</code>.
+After compiling to get tez.tar.gz, users should set the following properties
in tez-site.xml:</p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-xml data-lang=xml> <span
style=color:#f92672><property></span>
+ <span style=color:#f92672><name></span>tez.lib.uris<span
style=color:#f92672></name></span><span
style=color:#75715e><!--Example, replace with actual hdfs path--></span>
+ <span
style=color:#f92672><value></span>/apps/apache-tez-0.10.4-bin.tar.gz<span
style=color:#f92672></value></span>
+ <span style=color:#f92672></property></span>
+ <span style=color:#f92672><property></span>
+ <span
style=color:#f92672><name></span>tez.lib.uris.classpath<span
style=color:#f92672></name></span> <span style=color:#75715e><!--only
use tez self lib,do not use any old version hadoop cluster's
lib--></span>
+ <span
style=color:#f92672><value></span>$PWD/tezlib/*,$PWD/tezlib/lib/*<span
style=color:#f92672></value></span>
+ <span style=color:#f92672></property></span>
+ <span style=color:#f92672><property></span>
+ <span
style=color:#f92672><name></span>tez.use.cluster.hadoop-libs<span
style=color:#f92672></name></span><span style=color:#75715e><!--only
use tez self lib,do not use any old version hadoop cluster's
lib--></span>
+ <span style=color:#f92672><value></span>false<span
style=color:#f92672></value></span>
+ <span style=color:#f92672></property></span>
+
+ <span style=color:#f92672><property></span>
+ <span style=color:#f92672><name></span>tez.am.launch.env<span
style=color:#f92672></name></span><span
style=color:#75715e><!--Example, replace with actual native-lib install
path.Reuse old version hadoop cluster's native lib is ok.--></span>
+ <span
style=color:#f92672><value></span>LD_LIBRARY_PATH=/usr/hadoop/3.1.0/hadoop/lib/native<span
style=color:#f92672></value></span>
+ <span style=color:#f92672><description></span>Users can set up
environment variables individually, including but not limited to: JAVA_HOME,
LD_LIBRARY_PATH.<span style=color:#f92672></description></span>
+ <span style=color:#f92672></property></span>
+
+ <span style=color:#f92672><property></span>
+ <span style=color:#f92672><name></span>tez.task.launch.env<span
style=color:#f92672></name></span><span
style=color:#75715e><!--Example, replace with actual native-lib install
path.Reuse old version hadoop cluster's native lib is ok.--></span>
+ <span
style=color:#f92672><value></span>LD_LIBRARY_PATH=/usr/hadoop/3.1.0/hadoop/lib/native<span
style=color:#f92672></value></span>
+ <span style=color:#f92672><description></span>Users can set up
environment variables individually, including but not limited to: JAVA_HOME,
LD_LIBRARY_PATH.<span style=color:#f92672></description></span>
+ <span style=color:#f92672></property></span>
+</code></pre></div><p>2.Upload tez to the specified HDFS path in
<code>tez.lib.uris</code>.(Please remember, do not use the minimal tarball for
installation.)</p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-shell data-lang=shell><span style=color:#75715e>## DO not
upload minimal tarball !!!</span>
+<span style=color:#f92672>[</span>hadoop@hive opt<span
style=color:#f92672>]</span><span style=color:#75715e># hdfs dfs -put
apache-tez-0.10.4-bin.tar.gz /apps/</span>
+</code></pre></div><p>3.Download the high version of the Hadoop package(Please
ensure that the HADOOP version on which TEZ depends is the same as the HADOOP
version you have downloaded.).Unzip HIVE, HADOOP, and TEZ all in the
installation path.</p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-shell data-lang=shell><span style=color:#75715e>## In this
example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0
cluster.users should install HIVE,HADOOP and TEZ into actual directories.</span>
+<span style=color:#f92672>[</span>hadoop@hive opt<span
style=color:#f92672>]</span><span style=color:#75715e># cd /opt</span>
+<span style=color:#f92672>[</span>hadoop@hive opt<span
style=color:#f92672>]</span><span style=color:#75715e># ll</span>
+drwxr-xr-x <span style=color:#ae81ff>11</span> hive hadoop <span
style=color:#ae81ff>4096</span> Nov <span style=color:#ae81ff>7</span> 13:59
apache-hive-4.0.1-bin
+drwxr-xr-x <span style=color:#ae81ff>3</span> hive hadoop <span
style=color:#ae81ff>4096</span> Nov <span style=color:#ae81ff>7</span> 13:59
apache-tez-0.10.4-bin
+drwxr-xr-x <span style=color:#ae81ff>10</span> hive hadoop <span
style=color:#ae81ff>4096</span> Nov <span style=color:#ae81ff>7</span> 13:59
hadoop-3.3.6
+lrwxrwxrwx <span style=color:#ae81ff>1</span> hive hadoop <span
style=color:#ae81ff>30</span> Nov <span style=color:#ae81ff>7</span> 13:59
hive-4.0.1 -> apache-hive-4.0.1-bin
+lrwxrwxrwx <span style=color:#ae81ff>1</span> hive hadoop <span
style=color:#ae81ff>21</span> Nov <span style=color:#ae81ff>7</span> 13:59 tez
-> apache-tez-0.10.4-bin
+</code></pre></div><p>edit <code>hive-env.sh</code></p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-shell data-lang=shell><span style=color:#75715e># Set
HADOOP_HOME to point to a specific hadoop install directory</span>
+HADOOP_HOME<span style=color:#f92672>=</span><span
style=color:#e6db74>${</span>HADOOP_HOME<span
style=color:#66d9ef>:-</span>/opt/hadoop-3.3.6<span style=color:#e6db74>}</span>
+export HIVE_HOME<span style=color:#f92672>=</span><span
style=color:#e6db74>${</span>HIVE_HOME<span
style=color:#66d9ef>:-</span>/opt/hive-4.0.1<span style=color:#e6db74>}</span>
+export TEZ_HOME<span style=color:#f92672>=</span>/opt/tez
+</code></pre></div><p>Copy old version hadoop conf into hadoop3.3.6+:</p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-shell data-lang=shell>cp /usr/hadoop/3.1.0/hadoop/conf/*
/opt/hadoop3.3.6/conf/
+</code></pre></div><p>Put <code>tez-site.xml</code> into hive conf dir:</p>
+<div class=highlight><pre tabindex=0
style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code
class=language-shell data-lang=shell>mv tez-site.xml /opt/hive-4.0.1/conf/
+</code></pre></div><p>After completing the above steps, users should be able
to start the HMS service and HS2 service normally, and submit TEZ computing
tasks without any issues.</p>
+<p>Through the above steps, we can run Hive4+tez in any Hadoop3 environment.
Users do not need to upgrade the cluster’s original hive/hadoop/tez.</p>
<h2 id=next-steps>Next Steps</h2>
<p>You can begin using Hive as soon as it is installed, it should be work on
you computer. There are some extra information in the following sections.</p>
<h2 id=beeline-cli>Beeline CLI</h2>
diff --git a/index.xml b/index.xml
index 6c2ad82..ad3ace9 100644
--- a/index.xml
+++ b/index.xml
@@ -207,7 +207,7 @@ UDFs xpath, xpath_short, xpath_int, xpath_long,
xpath_float, xpath_double, xpath
Type Postfix Example TINYINT Y 100Y SMALLINT S 100S BIGINT L 100L String types
String literals can be expressed with either single quotes (') or double quotes
(&quot;).</description></item><item><title>Apache Hive :
LLAP</title><link>https://hive.apache.org/docs/latest/llap_62689557/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/llap_62689557/</guid><description>Apache
Hive : LLAP Overview Persistent Daemon Execution Engine Query Fr [...]
For configuration of LLAP, see the LLAP Section of Configuration Properties.
Overview Hive has become significantly faster thanks to various features and
improvements that were built by the community in recent years, including Tez
and Cost-based-optimization.</description></item><item><title>Apache Hive :
Locking</title><link>https://hive.apache.org/docs/latest/locking_27362050/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/locking_27362050/</guid><description>Apache
Hive : Locking Hive Concurrency Model Hive Co [...]
-Managed (Internal) External Introduction This document lists some of the
differences between the two but the fundamental difference is that Hive assumes
that it owns the data for managed tables. That means that the data, its
properties and data layout will and can only be changed via Hive command. The
data still lives in a normal file system and nothing is stopping you from
changing it without telling Hive about
it.</description></item><item><title>Apache Hive : Manual Installation</titl
[...]
+Managed (Internal) External Introduction This document lists some of the
differences between the two but the fundamental difference is that Hive assumes
that it owns the data for managed tables. That means that the data, its
properties and data layout will and can only be changed via Hive command. The
data still lives in a normal file system and nothing is stopping you from
changing it without telling Hive about
it.</description></item><item><title>Apache Hive : Manual Installation</titl
[...]
1. Map Join Optimization 1.1 Using Distributed Cache to Propagate Hashtable
File 1.2 Removing JDBM 1.3 Performance Evaluation 2. Converting Join into Map
Join Automatically 2.1 New Join Execution Flow 2.2 Resolving the Join Operation
at Run Time 2.3 Backup Task 2.4 Performance Evaluation 1. Map Join Optimization
1.1 Using Distributed Cache to Propagate Hashtable File Previously, when 2
large data tables need to do a join, there will be 2 different Mappers to sort
these tables based on th [...]
Materialized views support is introduced in Hive 3.0.0.
Introduction This page documents the work done for the supporting materialized
views in Apache Hive.</description></item><item><title>Apache Hive :
Materialized views in
Hive</title><link>https://hive.apache.org/docs/latest/materialized-views-in-hive_283118346/</link><pubDate>Thu,
12 Dec 2024 00:00:00
+0000</pubDate><guid>https://hive.apache.org/docs/latest/materialized-views-in-hive_283118346/</guid><description>Apache
Hive : Materialized views in Hive Objectives Traditionally, one of t [...]
