Author: stoty
Date: Fri Dec 1 17:02:03 2023
New Revision: 1914267
URL: http://svn.apache.org/viewvc?rev=1914267&view=rev
Log:
Update Hive connector page, and document webpage building quirks
Modified:
phoenix/site/publish/building_website.html
phoenix/site/publish/classpath_and_url.html
phoenix/site/publish/hive_storage_handler.html
phoenix/site/publish/language/datatypes.html
phoenix/site/publish/language/functions.html
phoenix/site/publish/language/index.html
phoenix/site/source/src/site/markdown/building_website.md
phoenix/site/source/src/site/markdown/hive_storage_handler.md
Modified: phoenix/site/publish/building_website.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/building_website.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/building_website.html (original)
+++ phoenix/site/publish/building_website.html Fri Dec 1 17:02:03 2023
@@ -1,14 +1,14 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
- <title>Building Phoenix Project Web Site | Apache
Phoenix</title>
+ <title>Prerequisites | Apache Phoenix</title>
<meta name="viewport" content="width=device-width,
initial-scale=1.0" />
<meta name="description" content="" />
<meta http-equiv="content-language" content="en" />
@@ -168,8 +168,14 @@
<div class="span12">
<div class="body-content">
<div class="page-header">
- <h1>Building Phoenix Project Web Site</h1>
+ <h1>Prerequisites</h1>
</div>
+<p>The Website build script has severely bit-rotted see <a
class="externalLink"
href="https://issues.apache.org/jira/browse/PHOENIX-6867";>PHOENIX-6867</a></p>
+<ul>
+ <li>The maven site.xml must have proxy set for the
<tt>http://conjars.org</tt> org repo</li>
+ <li>The mvn command must be configured to use Java 1.8. Newer versions
wonât work.</li>
+</ul>
+<h1>Building Phoenix Project Web Site</h1>
<ol style="list-style-type: decimal">
<li>Make a local copy of source markdown files and html web pages</li>
</ol>
Modified: phoenix/site/publish/classpath_and_url.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/classpath_and_url.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/classpath_and_url.html (original)
+++ phoenix/site/publish/classpath_and_url.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -263,7 +263,7 @@
<div class="section">
<h2 id="Notes">Notes</h2>
<p>Support for <i>master</i> and <i>RPC</i> registries is only available in
Phoenix 5.1.4 and later and 5.2.0 and later. Earlier version only support the
jdbc:phoenix: protocol variant implementing the the original HBase Zookeeper
Connection Registry.</p>
- <p>Support for the registry variants is only available for the HBase versions
that support them. Phoenix will throw an error if a variant that the HBase
version doesnât support is speci.</p>
+ <p>Support for the registry variants is only available for the HBase versions
that support them. Phoenix will throw an error if a variant that the HBase
client version doesnât support is specified.</p>
<p>Phoenix 5.2 also supports Hight Availability connections. Documentation
for that is only available in the <a class="externalLink"
href="https://issues.apache.org/jira/browse/PHOENIX-6491";>JIRA ticket</a></p>
</div>
</div>
Modified: phoenix/site/publish/hive_storage_handler.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/hive_storage_handler.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/hive_storage_handler.html (original)
+++ phoenix/site/publish/hive_storage_handler.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -173,54 +173,61 @@
<p>The Apache Phoenix Storage Handler is a plugin that enables Apache Hive
access to Phoenix tables from the Apache Hive command line using HiveQL.</p>
<div class="section">
<h2 id="Prerequisites">Prerequisites</h2>
+ <p>This document describes the plugin available in the phoenix-connectors
source repo, as of December 2023.</p>
<ul>
- <li>Phoenix 4.8.0+</li>
- <li>Hive 1.2.1+</li>
+ <li>Phoenix 5.1.0+</li>
+ <li>Hive 3.1.2+</li>
+ <li>phoenix-connectors 6.0.0-SNAPSHOT</li>
</ul>
+ <p>The Phoenix Storage handler currently only supports Hive 3.1. It has only
been tested with Hive Hive 3.1.3, and Phoenix 5.1.3.</p>
+ <p>A variant for Hive 4 is planned to be provided after Hive 4.0.0 has been
released</p>
</div>
<div class="section">
- <h2 id="Hive_Setup">Hive Setup</h2>
- <p>Make phoenix-version-hive.jar available for Hive:</p>
- <p>Step 1: Add to hive-env.sh:</p>
+ <h2 id="Building">Building</h2>
+ <p>The Phoenix Storage Handler used to be part of the main Phoenix repo, but
it has been refactored into the separate phoenix-connectors repo after the
release of Phoenix 5.0. At the time of writing there is no released version of
the connectors project, it must be built from the git source repository
HEAD.</p>
+ <p>Official releases will be available at the <a
href="download.html">Download</a> page.</p>
+ <p>Check the value of the hbase.version property in the root pom.xml If it is
older than HBase 2.5.0, then you need to rebuild HBase locally as described in
the BUILDING.md file in the main phoenix repository.</p>
+ <p>Check out the HEAD version of the <a class="externalLink"
href="https://github.com/apache/phoenix-connectors";>https://github.com/apache/phoenix-connectors</a>
repo. Build it with the <tt> mvn clean package </tt> command. The binary
distribution will be created in phoenix5-connectors-assembly/target
direcory.</p>
+ <p>The driver is built for Hbase 2.4.x by default.</p>
+ <p>To build it for other Hbase versions the <i>hbase.version</i>,
<i>hbase.compat.version</i>, <i>hadoop.version</i>, <i>zookeeper.version</i>
and <i>hbase-thirdparty-version</i> properties must be to the versions used by
HBase. hbase.compat.version is the matching hbase-compat-module in Phoenix, the
other versions can be copied from the root pom.xml from the sources of the
HBase release.</p>
+ <p>Fore example, to build with HBase 2.1.10 (re-built with
-Dhadoop.profile=3.0), use the following command:</p>
<div class="source">
- <pre>HIVE_AUX_JARS_PATH=<path to jar>
+ <pre>mvn clean package -Dhbase.version=2.1.10 -Dhbase.compat.version=2.1.6
-Dhadoop.version=3.0.3 -Dzookeeper.version=3.4.10
-Dhbase-thirdparty-version=2.1.0
</pre>
</div>
- <p>Step 2: Add a property to hive-site.xml so that Hive MapReduce jobs can
use the .jar:</p>
+</div>
+<div class="section">
+ <h2 id="Preparing_Hive_3">Preparing Hive 3</h2>
+ <p>Hive 3.1 ships with Hbase 2.0 beta, which is incompatible with Phoenix. To
use the Phoenix Storage handler HBase and its dependencies have to be removed
from Hive.</p>
+ <p>To remove the shipped HBase 2.0 beta perform the following on each Hive
node:</p>
+ <ol style="list-style-type: decimal">
+ <li> <p>Create a backup of the Hive /lib directory</p></li>
+ <li> <p>Remove the HBase dependencies from the /lib directory:</p></li>
+ </ol>
<div class="source">
- <pre><property>
- <name>hive.aux.jars.path</name>
- <value>file://<path></value>
-</property>
+ <pre>mkdir ../lib-removed
+mv hbase-* javax.inject* jcodings-* jersey-* joni-* osgi-resource-locator-*
../lib-removed/
</pre>
</div>
+ <p>Even though Hive ships with HBase jars, it includes a mechanism to load
the <tt>hbase mapredcp</tt> jars automatically.</p>
+ <p>To ensure that Hive finds and uses the correct HBase JARs and
hbase-site.xml, set the following environment variables in hive-env.sh on each
node, or make sure that they are set propery in the enviroment:</p>
+ <p>HBASE_HOME: The root directory of the HBase installation. HBASE_CONF_DIR:
The directory where hbase-site.xml resides. Defaults to /etc/hbase/conf, or if
it doesnât exist, to $HBASE_HOME/conf. HBASE_BIN_DIR: The directory which
holds the <tt>hbase</tt> command. Defaults to $HBASE_HOME/bin.</p>
+ <p>It is assumed that the HADOOP variables are already set correctly, and the
HBase libraries and up-date hbase-site.xml are available on each Hive node.</p>
</div>
<div class="section">
- <h2 id="Table_Creation_and_Deletion">Table Creation and Deletion</h2>
- <p>The Phoenix Storage Handler supports both INTERNAL and EXTERNAL Hive
tables.</p>
- <div class="section">
- <h3 id="Create_INTERNAL_Table">Create INTERNAL Table</h3>
- <p>For INTERNAL tables, Hive manages the lifecycle of the table and data.
When a Hive table is created, a corresponding Phoenix table is also created.
Once the Hive table is dropped, the Phoenix table is also deleted.</p>
- <div class="source">
- <pre> create table phoenix_table (
- s1 string,
- i1 int,
- f1 float,
- d1 double
- )
- STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
- TBLPROPERTIES (
- "phoenix.table.name" = "phoenix_table",
- "phoenix.zookeeper.quorum" = "localhost",
- "phoenix.zookeeper.znode.parent" = "/hbase",
- "phoenix.zookeeper.client.port" = "2181",
- "phoenix.rowkeys" = "s1, i1",
- "phoenix.column.mapping" = "s1:s1, i1:i1, f1:f1,
d1:d1",
- "phoenix.table.options" = "SALT_BUCKETS=10,
DATA_BLOCK_ENCODING='DIFF'"
- );
+ <h2 id="Hive_Setup">Hive Setup</h2>
+ <p>It is necessary to make the
<tt>phoenix5-hive-shaded-6.0.0-SNAPSHOT-shaded.jar</tt> available for every
hive component. There are many way to achiveve this, one of the simpler ones is
to use the <tt>HIVE_AUX_JARS_PATH</tt> environment variable.</p>
+ <p>If hive-env.sh already sets HIVE_AUX_JARS_PATH, then copy the connector
JAR there. Otherwise, create a world-readble directory on the system, and copy
the connector JAR there. then add a</p>
+ <div class="source">
+ <pre>HIVE_AUX_JARS_PATH=<PATH TO DIRECTORY>
</pre>
- </div>
</div>
+ <p>line to hive-env.sh</p>
+ <p>This must be performed on every Hive node.</p>
+</div>
+<div class="section">
+ <h2 id="Table_Creation_and_Deletion">Table Creation and Deletion</h2>
+ <p>The Phoenix Storage Handler supports only EXTERNAL Hive tables.</p>
<div class="section">
<h3 id="Create_EXTERNAL_Table">Create EXTERNAL Table</h3>
<p>For EXTERNAL tables, Hive works with an existing Phoenix table and
manages only Hive metadata. Dropping an EXTERNAL table from Hive deletes only
Hive metadata but does not delete the Phoenix table.</p>
@@ -242,6 +249,7 @@ TBLPROPERTIES (
);
</pre>
</div>
+ <p>Specifying the </p>
</div>
<div class="section">
<h3 id="Properties">Properties</h3>
@@ -276,6 +284,7 @@ TBLPROPERTIES (
<li>Mappings between column names for Hive and Phoenix. See <a
href="#Limitations">Limitations</a> for details.</li>
</ul></li>
</ol>
+ <p>The phoenix.zookeper properties are optional, if not specified then the
the values from hbase-site.xml will be used.</p>
</div>
</div>
<div class="section">
@@ -326,6 +335,7 @@ TBLPROPERTIES (
</tr>
</tbody>
</table>
+ <p>Disabling WAL can lead to data loss.</p>
</div>
<div class="section">
<h3 id="Query_Data">Query Data</h3>
Modified: phoenix/site/publish/language/datatypes.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/language/datatypes.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/datatypes.html (original)
+++ phoenix/site/publish/language/datatypes.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/publish/language/functions.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/language/functions.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/functions.html (original)
+++ phoenix/site/publish/language/functions.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/publish/language/index.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/language/index.html?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/publish/language/index.html (original)
+++ phoenix/site/publish/language/index.html Fri Dec 1 17:02:03 2023
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2023-11-06
+ Generated by Apache Maven Doxia at 2023-12-01
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
Modified: phoenix/site/source/src/site/markdown/building_website.md
URL:
http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/building_website.md?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/building_website.md (original)
+++ phoenix/site/source/src/site/markdown/building_website.md Fri Dec 1
17:02:03 2023
@@ -1,3 +1,10 @@
+# Prerequisites
+
+The Website build script has severely bit-rotted see
[PHOENIX-6867](https://issues.apache.org/jira/browse/PHOENIX-6867)
+
+ - The maven site.xml must have proxy set for the `http://conjars.org` org repo
+ - The mvn command must be configured to use Java 1.8. Newer versions won't
work.
+
# Building Phoenix Project Web Site
1. Make a local copy of source markdown files and html web pages
Modified: phoenix/site/source/src/site/markdown/hive_storage_handler.md
URL:
http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/hive_storage_handler.md?rev=1914267&r1=1914266&r2=1914267&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/hive_storage_handler.md (original)
+++ phoenix/site/source/src/site/markdown/hive_storage_handler.md Fri Dec 1
17:02:03 2023
@@ -2,55 +2,96 @@
The Apache Phoenix Storage Handler is a plugin that enables Apache Hive access
to Phoenix tables from the Apache Hive command line using HiveQL.
+
## Prerequisites
-* Phoenix 4.8.0+
-* Hive 1.2.1+
+This document describes the plugin available in the phoenix-connectors source
repo, as of December 2023.
+
+* Phoenix 5.1.0+
+* Hive 3.1.2+
+* phoenix-connectors 6.0.0-SNAPSHOT
+
+The Phoenix Storage handler currently only supports Hive 3.1.
+It has only been tested with Hive Hive 3.1.3, and Phoenix 5.1.3.
+
+A variant for Hive 4 is planned to be provided after Hive 4.0.0 has been
released
-## Hive Setup
-Make phoenix-version-hive.jar available for Hive:
+## Building
-Step 1: Add to hive-env.sh:
+The Phoenix Storage Handler used to be part of the main Phoenix repo, but it
has been refactored into the separate phoenix-connectors repo after the release
of Phoenix 5.0.
+At the time of writing there is no released version of the connectors project,
it must be built from the git source repository HEAD.
+Official releases will be available at the [Download](download.html) page.
+
+Check the value of the hbase.version property in the root pom.xml
+If it is older than HBase 2.5.0, then you need to rebuild HBase locally as
described
+in the BUILDING.md file in the main phoenix repository.
+
+Check out the HEAD version of the https://github.com/apache/phoenix-connectors
repo.
+Build it with the
```
-HIVE_AUX_JARS_PATH=<path to jar>
+mvn clean package
```
+command.
+The binary distribution will be created in phoenix5-connectors-assembly/target
direcory.
+
+The driver is built for Hbase 2.4.x by default.
+
+To build it for other Hbase versions the *hbase.version*,
*hbase.compat.version*, *hadoop.version*, *zookeeper.version* and
*hbase-thirdparty-version* properties must be to the versions used by HBase.
hbase.compat.version is the matching hbase-compat-module in Phoenix, the other
versions can be copied from the root pom.xml from the sources of the HBase
release.
-Step 2: Add a property to hive-site.xml so that Hive MapReduce jobs can use
the .jar:
+Fore example, to build with HBase 2.1.10 (re-built with -Dhadoop.profile=3.0),
use the following command:
```
-<property>
- <name>hive.aux.jars.path</name>
- <value>file://<path></value>
-</property>
+mvn clean package -Dhbase.version=2.1.10 -Dhbase.compat.version=2.1.6
-Dhadoop.version=3.0.3 -Dzookeeper.version=3.4.10
-Dhbase-thirdparty-version=2.1.0
```
-## Table Creation and Deletion
-The Phoenix Storage Handler supports both INTERNAL and EXTERNAL Hive tables.
-### Create INTERNAL Table
-For INTERNAL tables, Hive manages the lifecycle of the table and data. When a
Hive table is created, a corresponding Phoenix table is also created.
-Once the Hive table is dropped, the Phoenix table is also deleted.
+## Preparing Hive 3
-```sql
- create table phoenix_table (
- s1 string,
- i1 int,
- f1 float,
- d1 double
- )
- STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
- TBLPROPERTIES (
- "phoenix.table.name" = "phoenix_table",
- "phoenix.zookeeper.quorum" = "localhost",
- "phoenix.zookeeper.znode.parent" = "/hbase",
- "phoenix.zookeeper.client.port" = "2181",
- "phoenix.rowkeys" = "s1, i1",
- "phoenix.column.mapping" = "s1:s1, i1:i1, f1:f1, d1:d1",
- "phoenix.table.options" = "SALT_BUCKETS=10,
DATA_BLOCK_ENCODING='DIFF'"
- );
+Hive 3.1 ships with Hbase 2.0 beta, which is incompatible with Phoenix.
+To use the Phoenix Storage handler HBase and its dependencies have to be
removed from Hive.
+
+To remove the shipped HBase 2.0 beta perform the following on each Hive node:
+
+1. Create a backup of the Hive /lib directory
+
+2. Remove the HBase dependencies from the /lib directory:
+
+```
+mkdir ../lib-removed
+mv hbase-* javax.inject* jcodings-* jersey-* joni-* osgi-resource-locator-*
../lib-removed/
+```
+
+Even though Hive ships with HBase jars, it includes a mechanism to load the
`hbase mapredcp` jars automatically.
+
+To ensure that Hive finds and uses the correct HBase JARs and hbase-site.xml,
set the following
+environment variables in hive-env.sh on each node, or make sure that they are
set propery in the enviroment:
+
+HBASE_HOME: The root directory of the HBase installation.
+HBASE_CONF_DIR: The directory where hbase-site.xml resides. Defaults to
/etc/hbase/conf, or if it doesn't exist, to $HBASE_HOME/conf.
+HBASE_BIN_DIR: The directory which holds the `hbase` command. Defaults to
$HBASE_HOME/bin.
+
+It is assumed that the HADOOP variables are already set correctly, and the
HBase libraries and up-date hbase-site.xml are available on each Hive node.
+
+
+## Hive Setup
+
+It is necessary to make the `phoenix5-hive-shaded-6.0.0-SNAPSHOT-shaded.jar`
available for every hive component.
+There are many way to achiveve this, one of the simpler ones is to use the
`HIVE_AUX_JARS_PATH` environment variable.
+
+If hive-env.sh already sets HIVE_AUX_JARS_PATH, then copy the connector JAR
there.
+Otherwise, create a world-readble directory on the system, and copy the
connector JAR there. then add a
+
+```
+HIVE_AUX_JARS_PATH=<PATH TO DIRECTORY>
```
+line to hive-env.sh
+
+This must be performed on every Hive node.
+
+## Table Creation and Deletion
+The Phoenix Storage Handler supports only EXTERNAL Hive tables.
### Create EXTERNAL Table
For EXTERNAL tables, Hive works with an existing Phoenix table and manages
only Hive metadata. Dropping an EXTERNAL table from Hive deletes only Hive
metadata but does not delete the Phoenix table.
@@ -73,6 +114,7 @@ TBLPROPERTIES (
);
```
+Specifying the
### Properties
1. phoenix.table.name
@@ -93,7 +135,7 @@ TBLPROPERTIES (
6. phoenix.column.mapping
* Mappings between column names for Hive and Phoenix. See
[Limitations](#Limitations) for details.
-
+The phoenix.zookeper properties are optional, if not specified then the the
values from hbase-site.xml will be used.
## Data Ingestion, Deletions, and Updates
Data ingestion can be done by all ways that Hive and Phoenix support:
@@ -126,6 +168,8 @@ phoenix.upsert.batch.size | 1000 | Batch
[phoenix-table-name].disable.wal | false | Temporarily sets the table
attribute `DISABLE_WAL` to `true`. Sometimes used to improve performance
[phoenix-table-name].auto.flush | false | When WAL is disabled and if this
value is `true`, then MemStore is flushed to an HFile.
+Disabling WAL can lead to data loss.
+
### Query Data
You can use HiveQL for querying data in a Phoenix table. A Hive query on a
single table can be as fast as running the query in the Phoenix CLI with the
following property settings: `hive.fetch.task.conversion=more` and
`hive.exec.parallel=true`
@@ -145,3 +189,4 @@ split.by.stats | false | If true, mapper
## Resources <a id="Resources"></a>
* [PHOENIX-2743] (https://issues.apache.org/jira/browse/PHOENIX-2743) :
Implementation, accepted by Apache Phoenix community. Original pull request
contains modification for Hive classes.
* [PHOENIX-331] (https://issues.apache.org/jira/browse/PHOENIX-331) : An
outdated implementation with support of Hive 0.98.
+
