Author: rajeshbabu
Date: Fri Aug 12 13:12:51 2016
New Revision: 1756181
URL: http://svn.apache.org/viewvc?rev=1756181&view=rev
Log:
Updating docs and release notes for new local index implementation(Rajeshbabu)
Modified:
phoenix/site/publish/release_notes.html
phoenix/site/publish/secondary_indexing.html
phoenix/site/source/src/site/markdown/release_notes.md
phoenix/site/source/src/site/markdown/secondary_indexing.md
Modified: phoenix/site/publish/release_notes.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/release_notes.html?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/publish/release_notes.html (original)
+++ phoenix/site/publish/release_notes.html Fri Aug 12 13:12:51 2016
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2016-08-10
+ Generated by Apache Maven Doxia at 2016-08-11
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -155,6 +155,22 @@
<h3 id="Phoenix-4.8.0_Release_Notes"><u>Phoenix-4.8.0 Release Notes</u></h3>
<p><a class="externalLink"
href="https://issues.apache.org/jira/browse/PHOENIX-3164";>PHOENIX-3164</a> is a
relatively serious bug that affects the <a class="externalLink"
href="http://phoenix.apache.org/server.html";>Phoenix Query Server</a> deployed
with âsecurity enabledâ (Kerberos or Active Directory). Due to another
late-game change in the 4.8.0 release as well as an issue with the use of
Hadoopâs UserGroupInformation class, every âclient sessionâ to the
Phoenix Query Server with security enabled will result in a new instance of the
Phoenix JDBC driver <tt>PhoenixConnection</tt> (and other related classes).
This ultimately results in a new connection to ZooKeeper for each âclient
sessionâ.</p>
<p>Within a short amount of time of active use with the Phoenix Query Server
creating a new ZooKeeper connection for each âclient sessionâ, the number
of ZooKeeper connections will have grown rapidly likely triggering
ZooKeeperâs built-in denial of service protection (<a class="externalLink"
href="https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html";>maxClientCnxns</a>).
This will cause all future connections to ZooKeeper by the host running the
Phoenix Query Server to be dropped. This would prevent all HBase client API
calls which need to access ZooKeeper from completing.</p>
+ <p>As part of <a class="externalLink"
href="https://issues.apache.org/jira/browse/PHOENIX-1734";>PHOENIX-1734</a> we
have changed the local index implementation to store index data in the separate
column families in the same data table. So while upgrading the phoenix at
server we need to remove below local index related configurations from
<tt>hbase-site.xml</tt> and run upgrade steps mentioned <a
href="secondary_indexing.html#Upgrading_Local_Indexes_created_before_4.8.0">here</a></p>
+ <div class="source">
+ <pre><property>
+ <name>hbase.master.loadbalancer.class</name>
+
<value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
+</property>
+<property>
+ <name>hbase.coprocessor.master.classes</name>
+
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
+</property>
+<property>
+ <name>hbase.coprocessor.regionserver.classes</name>
+
<value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
+</property>
+</pre>
+ </div>
</div>
<div class="section">
<h3 id="Phoenix-4.5.0_Release_Notes"><u>Phoenix-4.5.0 Release Notes</u></h3>
Modified: phoenix/site/publish/secondary_indexing.html
URL:
http://svn.apache.org/viewvc/phoenix/site/publish/secondary_indexing.html?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/publish/secondary_indexing.html (original)
+++ phoenix/site/publish/secondary_indexing.html Fri Aug 12 13:12:51 2016
@@ -1,7 +1,7 @@
<!DOCTYPE html>
<!--
- Generated by Apache Maven Doxia at 2016-08-11
+ Generated by Apache Maven Doxia at 2016-08-12
Rendered using Reflow Maven Skin 1.1.0
(http://andriusvelykis.github.io/reflow-maven-skin)
-->
<html xml:lang="en" lang="en">
@@ -180,7 +180,7 @@
</div>
<div class="section">
<h2 id="Local_Indexes">Local Indexes</h2>
- <p>Local indexing targets <i>write heavy</i>, <i>space constrained</i> use
cases. Just like with global indexes, Phoenix will automatically select whether
or not to use a local index at query-time. With local indexes, index data and
table data co-reside on same server preventing any network overhead during
writes. Local indexes can be used even when the query isnât fully covered
(i.e. Phoenix automatically retrieve the columns not in the index through point
gets against the data table). Unlike global indexes, all local indexes of a
table are stored in a single, separate shared table. At read time when the
local index is used, every region must be examined for the data as the exact
region location of index data cannot be predetermined. Thus some overhead
occurs at read-time.</p>
+ <p>Local indexing targets <i>write heavy</i>, <i>space constrained</i> use
cases. Just like with global indexes, Phoenix will automatically select whether
or not to use a local index at query-time. With local indexes, index data and
table data co-reside on same server preventing any network overhead during
writes. Local indexes can be used even when the query isnât fully covered
(i.e. Phoenix automatically retrieve the columns not in the index through point
gets against the data table). Unlike global indexes, all local indexes of a
table are stored in a single, separate shared table prior to 4.8.0 version.
From 4.8.0 onwards we are storing all local index data in the separate shadow
column families in the same data table. At read time when the local index is
used, every region must be examined for the data as the exact region location
of index data cannot be predetermined. Thus some overhead occurs at
read-time.</p>
</div>
<div class="section">
<h2 id="Index_Population">Index Population</h2>
@@ -350,8 +350,7 @@ CREATE LOCAL INDEX my_index ON my_table
</pre>
</div>
<p>The above properties prevent deadlocks from occurring during index
maintenance for global indexes (HBase 0.98.4+ and Phoenix 4.3.1+ only) by
ensuring index updates are processed with a higher priority than data updates.
It also prevents deadlocks by ensuring metadata rpc calls are processed with a
higher priority than data rpc calls.</p>
- <p>Local indexing also requires special configurations in the master to
ensure data table and local index regions co-location.</p>
- <p>You will need to add the following parameters to <tt>hbase-site.xml</tt>
on the master:</p>
+ <p>From Phoenix 4.8.0 onward, no configuration changes are required to use
local indexing. In Phoenix 4.7 and below, the following configuration changes
are required to the server-side hbase-site.xml on the master and regions server
nodes:</p>
<div class="source">
<pre><property>
<name>hbase.master.loadbalancer.class</name>
@@ -361,16 +360,25 @@ CREATE LOCAL INDEX my_index ON my_table
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
-</pre>
- </div>
- <p>To support local index regions merge on data regions merge you will need
to add the following parameter to <tt>hbase-site.xml</tt> in all the region
servers and restart. (Itâs applicable for Phoenix 4.3+ versions)</p>
- <div class="source">
- <pre><property>
+<property>
<name>hbase.coprocessor.regionserver.classes</name>
<value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
</property>
</pre>
</div>
+ <div class="section">
+ <h3 id="Upgrading_Local_Indexes_created_before_4.8.0">Upgrading Local
Indexes created before 4.8.0</h3>
+ <p>While upgrading the Phoenix to 4.8.0+ version at server remove above
three local indexing related configurations from <tt>hbase-site.xml</tt> if
present. From client we are supporting both online(while initializing the
connection from phoenix client of 4.8.0+ versions) and offline(using psql tool)
upgrade of local indexes created before 4.8.0. As part of upgrade we recreate
the local indexes in ASYNC mode. After upgrade user need to build the indexes
using <a class="externalLink"
href="http://phoenix.apache.org/secondary_indexing.html#Index_Population";>IndexTool</a></p>
+ <p>Following client side configuration used in the upgrade. </p>
+ <ol style="list-style-type: decimal">
+ <li><tt>phoenix.client.localIndexUpgrade</tt>
+ <ul>
+ <li>The value of it is true means online upgrade and false means offline
upgrade.</li>
+ <li><b>Default: true</b></li>
+ </ul></li>
+ </ol>
+ <p>Command to run offline upgrade using psql tool <tt>$ psql [zookeeper]
-l</tt></p>
+ </div>
</div>
<div class="section">
<h2 id="Tuning">Tuning</h2>
Modified: phoenix/site/source/src/site/markdown/release_notes.md
URL:
http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/release_notes.md?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/release_notes.md (original)
+++ phoenix/site/source/src/site/markdown/release_notes.md Fri Aug 12 13:12:51
2016
@@ -22,6 +22,27 @@ will cause all future connections to Zoo
be dropped. This would prevent all HBase client API calls which need to access
ZooKeeper
from completing.
+As part of [PHOENIX-1734](https://issues.apache.org/jira/browse/PHOENIX-1734)
we have changed
+the local index implementation to store index data in the separate column
families in the same
+ data table. So while upgrading the phoenix at server we need to remove below
local index
+related configurations from `hbase-site.xml` and run upgrade steps mentioned
+[here](secondary_indexing.html#Upgrading_Local_Indexes_created_before_4.8.0)
+
+```
+<property>
+ <name>hbase.master.loadbalancer.class</name>
+ <value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
+</property>
+<property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
+</property>
+<property>
+ <name>hbase.coprocessor.regionserver.classes</name>
+ <value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
+</property>
+```
+
###<u>Phoenix-4.5.0 Release Notes</u>
Both [PHOENIX-2067](https://issues.apache.org/jira/browse/PHOENIX-2067) and
[PHOENIX-2120](https://issues.apache.org/jira/browse/PHOENIX-2120) cause rows
to not be ordered
Modified: phoenix/site/source/src/site/markdown/secondary_indexing.md
URL:
http://svn.apache.org/viewvc/phoenix/site/source/src/site/markdown/secondary_indexing.md?rev=1756181&r1=1756180&r2=1756181&view=diff
==============================================================================
--- phoenix/site/source/src/site/markdown/secondary_indexing.md (original)
+++ phoenix/site/source/src/site/markdown/secondary_indexing.md Fri Aug 12
13:12:51 2016
@@ -37,7 +37,7 @@ Each are useful in different scenarios a
Global indexing targets _read heavy_ uses cases. With global indexes, all the
performance penalties for indexes occur at write time. We intercept the data
table updates on write ([DELETE](language/index.html#delete), [UPSERT
VALUES](language/index.html#upsert_values) and [UPSERT
SELECT](language/index.html#upsert_select)), build the index update and then
sent any necessary updates to all interested index tables. At read time,
Phoenix will select the index table to use that will produce the fastest query
time and directly scan it just like any other HBase table. By default, unless
hinted, an index will not be used for a query that references a column that
isn't part of the index.
## Local Indexes
-Local indexing targets _write heavy_, _space constrained_ use cases. Just like
with global indexes, Phoenix will automatically select whether or not to use a
local index at query-time. With local indexes, index data and table data
co-reside on same server preventing any network overhead during writes. Local
indexes can be used even when the query isn't fully covered (i.e. Phoenix
automatically retrieve the columns not in the index through point gets against
the data table). Unlike global indexes, all local indexes of a table are stored
in a single, separate shared table. At read time when the local index is used,
every region must be examined for the data as the exact region location of
index data cannot be predetermined. Thus some overhead occurs at read-time.
+Local indexing targets _write heavy_, _space constrained_ use cases. Just like
with global indexes, Phoenix will automatically select whether or not to use a
local index at query-time. With local indexes, index data and table data
co-reside on same server preventing any network overhead during writes. Local
indexes can be used even when the query isn't fully covered (i.e. Phoenix
automatically retrieve the columns not in the index through point gets against
the data table). Unlike global indexes, all local indexes of a table are stored
in a single, separate shared table prior to 4.8.0 version. From 4.8.0 onwards
we are storing all local index data in the separate shadow column families in
the same data table. At read time when the local index is used, every region
must be examined for the data as the exact region location of index data cannot
be predetermined. Thus some overhead occurs at read-time.
## Index Population
By default, when an index is created, it is populated synchronously during the
CREATE INDEX call. This may not be feasible depending on the current size of
the data table. As of 4.5, initially population of an index may be done
asynchronously by including the ASYNC keyword in the index creation DDL
statement:
@@ -232,9 +232,7 @@ The above property enables custom WAL ed
The above properties prevent deadlocks from occurring during index maintenance
for global indexes (HBase 0.98.4+ and Phoenix 4.3.1+ only) by ensuring index
updates are processed with a higher priority than data updates. It also
prevents deadlocks by ensuring metadata rpc calls are processed with a higher
priority than data rpc calls.
-Local indexing also requires special configurations in the master to ensure
data table and local index regions co-location.
-
-You will need to add the following parameters to `hbase-site.xml` on the
master:
+From Phoenix 4.8.0 onward, no configuration changes are required to use local
indexing. In Phoenix 4.7 and below, the following configuration changes are
required to the server-side hbase-site.xml on the master and regions server
nodes:
```
<property>
@@ -245,16 +243,22 @@ You will need to add the following param
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
-```
-
-To support local index regions merge on data regions merge you will need to
add the following parameter to `hbase-site.xml` in all the region servers and
restart. (Itâs applicable for Phoenix 4.3+ versions)
-
-```
<property>
<name>hbase.coprocessor.regionserver.classes</name>
<value>org.apache.hadoop.hbase.regionserver.LocalIndexMerger</value>
</property>
```
+### Upgrading Local Indexes created before 4.8.0
+While upgrading the Phoenix to 4.8.0+ version at server remove above three
local indexing related configurations from `hbase-site.xml` if present. From
client we are supporting both online(while initializing the connection from
phoenix client of 4.8.0+ versions) and offline(using psql tool) upgrade of
local indexes created before 4.8.0. As part of upgrade we recreate the local
indexes in ASYNC mode. After upgrade user need to build the indexes using
[IndexTool](http://phoenix.apache.org/secondary_indexing.html#Index_Population)
+
+Following client side configuration used in the upgrade.
+
+1. <code>phoenix.client.localIndexUpgrade</code>
+ * The value of it is true means online upgrade and false means offline
upgrade.
+ * **Default: true**
+
+Command to run offline upgrade using psql tool
+`$ psql [zookeeper] -l`
## Tuning
Out the box, indexing is pretty fast. However, to optimize for your particular
environment and workload, there are several properties you can tune.
