Author: stack
Date: Thu Aug 18 07:34:11 2011
New Revision: 1159069
URL: http://svn.apache.org/viewvc?rev=1159069&view=rev
Log:
HBASE-4176 Exposing HBase Filters to the Thrift API
Modified:
hbase/trunk/src/docbkx/book.xml
Modified: hbase/trunk/src/docbkx/book.xml
URL:
http://svn.apache.org/viewvc/hbase/trunk/src/docbkx/book.xml?rev=1159069&r1=1159068&r2=1159069&view=diff
==============================================================================
--- hbase/trunk/src/docbkx/book.xml (original)
+++ hbase/trunk/src/docbkx/book.xml Thu Aug 18 07:34:11 2011
@@ -914,6 +914,367 @@ HTable table2 = new HTable(conf2, "myTab
</section>
</section>
+ <section xml:id="client.filter-language"><title>Filter
Language</title>
+ <section xml:id="use-case"><title>Use Case</title>
+ <para>This allows the user to perform server-side filtering
when accessing HBase over Thrift. The user specifies a filter via a string. The
string is parsed on the server to construct the filter</para>
+ </section>
+
+ <section xml:id="general-syntax"><title>General Filter String
Syntax</title>
+ <para>A simple filter expression is expressed as:
<code>âFilterName (argument, argument, ... , argument)â</code></para>
+ <para>You must specify the name of the filter followed by the
argument list in parenthesis. Commas separate the individual arguments</para>
+ <para>If the argument represents a string, it should be
enclosed in single quotes.</para>
+ <para>If it represents a boolean, an integer or a comparison
operator like <,
+ >, != etc. it should not be enclosed in quotes</para>
+ <para>The filter name must be one word. All ASCII characters
are allowed except for whitespace, single quotes and parenthesis.</para>
+ <para>The filterâs arguments can contain any ASCII character.
<code>If single quotes are present in the argument, they must be escaped by a
+ preceding single quote</code></para>
+ </section>
+
+ <section xml:id="compound-filters-and-operators"><title>Compound
Filters and Operators</title>
+ <para>Currently, two binary operators â AND/OR and two unary
operators â WHILE/SKIP are supported.</para>
+ <para>Note: the operators are all in uppercase</para>
+ <para><emphasis role="bold">AND</emphasis> â as the name
suggests, if this
+ operator is used, the key-value must pass both the
filters</para>
+ <para><emphasis role="bold">OR</emphasis> â as the name
suggests, if this operator
+ is used, the key-value must pass at least one of the
filters</para>
+ <para><emphasis role="bold">SKIP</emphasis> â For a
particular row, if any of the
+ key-values donât pass the filter condition, the entire row
is skipped</para>
+ <para><emphasis role="bold">WHILE</emphasis> - For a particular
row, it continues
+ to emit key-values until a key-value is reached that fails
the filter condition</para>
+ <para><emphasis role="bold">Compound Filters:</emphasis> Using
these operators, a
+ hierarchy of filters can be created. For example:
<code>â(Filter1 AND Filter2) OR (Filter3 AND Filter4)â</code></para>
+ </section>
+
+ <section xml:id="order-of-evaluation"><title>Order of
Evaluation</title>
+ <para>Parenthesis have the highest precedence. The SKIP and
WHILE operators are next and have the same precedence.The AND operator has the
next highest precedence followed by the OR operator.</para>
+ <para>For example:</para>
+ <para>A filter string of the form:<code>âFilter1 AND Filter2
OR Filter3â</code>
+ will be evaluated as:<code>â(Filter1 AND Filter2) OR
Filter3â</code></para>
+ <para>A filter string of the form:<code>âFilter1 AND SKIP
Filter2 OR Filter3â</code>
+ will be evaluated as:<code>â(Filter1 AND (SKIP Filter2)) OR
Filter3â</code></para>
+ </section>
+
+ <section xml:id="compare-operator"><title>Compare Operator</title>
+ <para>A compare operator can be any of the following:</para>
+ <orderedlist>
+ <listitem>
+ <para>LESS (<)</para>
+ </listitem>
+ <listitem>
+ <para>LESS_OR_EQUAL (<=)</para>
+ </listitem>
+ <listitem>
+ <para>EQUAL (=)</para>
+ </listitem>
+ <listitem>
+ <para>NOT_EQUAL (!=)</para>
+ </listitem>
+ <listitem>
+ <para>GREATER_OR_EQUAL (>=)</para>
+ </listitem>
+ <listitem>
+ <para>GREATER (>)</para>
+ </listitem>
+ <listitem>
+ <para>NO_OP (no operation)</para>
+ </listitem>
+ </orderedlist>
+ <para>The client should use the symbols (<, <=, =, !=, >,
>=) to express
+ compare operators.</para>
+ </section>
+
+ <section xml:id="comparator"><title>Comparator</title>
+ <para>A comparator can be any of the following:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">BinaryComparator</emphasis> -
This
+ lexicographically compares against the specified byte
array using
+ Bytes.compareTo(byte[], byte[])</para>
+ </listitem>
+ <listitem>
+ <para><emphasis
role="bold">BinaryPrefixComparator</emphasis> - This
+ lexicographically compares against a specified byte
array. It only compares up to
+ the length of this byte array.</para>
+ </listitem>
+ <listitem>
+ <para><emphasis
role="bold">RegexStringComparator</emphasis> - This compares
+ against the specified byte array using the given regular
expression. Only EQUAL
+ and NOT_EQUAL comparisons are valid with this
comparator</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">SubStringComparator</emphasis>
- This tests if
+ the given substring appears in a specified byte array.
The comparison is case
+ insensitive. Only EQUAL and NOT_EQUAL comparisons are
valid with this
+ comparator</para>
+ </listitem>
+ </orderedlist>
+ <para>The general syntax of a comparator is:<code>
ComparatorType:ComparatorValue</code></para>
+ <para>The ComparatorType for the various comparators is as
follows:</para>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold">BinaryComparator</emphasis> -
binary</para>
+ </listitem>
+ <listitem>
+ <para><emphasis
role="bold">BinaryPrefixComparator</emphasis> - binaryprefix</para>
+ </listitem>
+ <listitem>
+ <para><emphasis
role="bold">RegexStringComparator</emphasis> - regexstring</para>
+ </listitem>
+ <listitem>
+ <para><emphasis role="bold">SubStringComparator</emphasis>
- substring</para>
+ </listitem>
+ </orderedlist>
+ <para>The ComparatorValue can be any value.</para>
+ <para>Example1:<code> >, 'binary:abc' </code>will match
everything that is lexicographically greater than "abc" </para>
+ <para>Example2:<code> =, 'binaryprefix:abc' </code>will match
everything whose first 3 characters are lexicographically equal to "abc"</para>
+ <para>Example3:<code> !=, 'regexstring:ab*yz' </code>will match
everything that doesn't begin with "ab" and ends with "yz"</para>
+ <para>Example4:<code> =, 'substring:abc123' </code>will match
everything that begins with the substring "abc123"</para>
+ </section>
+
+ <section xml:id="example PHP Client Program"><title>Example PHP
Client Program that uses the Filter Language</title>
+ <programlisting>
+<? $_SERVER['PHP_ROOT'] = realpath(dirname(__FILE__).'/..');
+ require_once $_SERVER['PHP_ROOT'].'/flib/__flib.php';
+ flib_init(FLIB_CONTEXT_SCRIPT);
+ require_module('storage/hbase');
+ $hbase = new HBase('<server_name_running_thrift_server>', <port on
which thrift server is running>);
+ $hbase->open();
+ $client = $hbase->getClient();
+ $result = $client->scannerOpenWithFilterString('table_name',
"(PrefixFilter ('row2') AND (QualifierFilter (>=, 'binary:xyz'))) AND
(TimestampsFilter ( 123, 456))");
+ $to_print = $client->scannerGetList($result,1);
+ while ($to_print) {
+ print_r($to_print);
+ $to_print = $client->scannerGetList($result,1);
+ }
+ $client->scannerClose($result);
+?>
+ </programlisting>
+ </section>
+
+ <section xml:id="example-filter-strings"><title>Example Filter
Strings</title>
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para><code>âPrefixFilter (âRowâ) AND PageFilter (1) AND
FirstKeyOnlyFilter ()â</code> will return all key-value pairs that match the
following conditions:</para>
+ <para>1) The row containing the key-value should have prefix
âRowâ </para>
+ <para>2) The key-value must be located in the first row of the
table </para>
+ <para>3) The key-value pair must be the first key-value in the
row </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <orderedlist>
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para><code>â(RowFilter (=, âbinary:Row 1â) AND
TimeStampsFilter (74689, 89734)) OR
+ ColumnRangeFilter (âabcâ, true, âxyzâ,
false))â</code> will return all key-value pairs that match both the following
conditions:</para>
+ <para>1) The key-value is in a row having row key âRow 1â
</para>
+ <para>2) The key-value must have a timestamp of either 74689
or 89734.</para>
+ <para>Or it must match the following condition:</para>
+ <para>1) The key-value pair must be in a column that is
lexicographically >= abc and < xyz </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </orderedlist>
+
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para><code>âSKIP ValueFilter (0)â</code> will skip the
entire row if any of the values in the row is not 0</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
+ <section xml:id="Individual Filter Syntax"><title>Individual Filter
Syntax</title>
+ <orderedlist>
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">KeyOnlyFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
doesnât take any
+ arguments. It returns only the key component of each key-value.
</para>
+ <para><emphasis role="bold">Syntax:</emphasis> KeyOnlyFilter ()
</para>
+ <para><emphasis role="bold">Example:</emphasis> "KeyOnlyFilter
()"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">FirstKeyOnlyFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
doesnât take any
+ arguments. It returns only the first key-value from each row.
</para>
+ <para><emphasis role="bold">Syntax:</emphasis> FirstKeyOnlyFilter
() </para>
+ <para><emphasis role="bold">Example:</emphasis>
"FirstKeyOnlyFilter ()" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">PrefixFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes one argument â a prefix of a
+ row key. It returns only those key-values present in a row that
starts with the
+ specified row prefix</para>
+ <para><emphasis role="bold">Syntax:</emphasis> PrefixFilter
(â<row_prefix>â) </para>
+ <para><emphasis role="bold">Example:</emphasis> "PrefixFilter
(âRowâ)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis role="underline">
+ ColumnPrefixFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes one argument
+ â a column prefix. It returns only those key-values present in
a column that starts
+ with the specified column prefix. The column prefix must be of
the form: <code>âqualifierâ </code></para>
+ <para><emphasis
role="bold">Syntax:</emphasis>ColumnPrefixFilter(â<column_prefix>â)</para>
+ <para><emphasis role="bold">Example:</emphasis>
"ColumnPrefixFilter(âColâ)"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="underline"><emphasis
role="bold">MultipleColumnPrefixFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a list of
+ column prefixes. It returns key-values that are present in a
column that starts with
+ any of the specified column prefixes. Each of the column
prefixes must be of the form: <code>âqualifierâ</code></para>
+ <para><emphasis
role="bold">Syntax:</emphasis>MultipleColumnPrefixFilter(â<column_prefix>â,
â<column_prefix>â, â¦, â<column_prefix>â)</para>
+ <para><emphasis role="bold">Example:</emphasis>
"MultipleColumnPrefixFilter(âCol1â, âCol2â)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">ColumnCountGetFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes one argument
+ â a limit. It returns the first limit number of columns in the
table</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
ColumnCountGetFilter (â<limit>â)</para>
+ <para><emphasis role="bold">Example:</emphasis>
"ColumnCountGetFilter (4)"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">PageFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes one argument
+ â a page size. It returns page size number of rows from the
table. </para>
+ <para><emphasis role="bold">Syntax:</emphasis> PageFilter
(â<page_size>â)</para>
+ <para><emphasis role="bold">Example:</emphasis> "PageFilter (2)"
</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">ColumnPaginationFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes two
+ arguments â a limit and offset. It returns limit number of
columns after offset number
+ of columns. It does this for all the rows</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
ColumnPaginationFilter(â<limit>â, â<offest>â) </para>
+ <para><emphasis role="bold">Example:</emphasis>
"ColumnPaginationFilter (3, 5)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">InclusiveStopFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes one argument
+ â a row key on which to stop scanning. It returns all
key-values present in rows up to
+ and including the specified row</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
InclusiveStopFilter(â<stop_row_key>â) </para>
+ <para><emphasis role="bold">Example:</emphasis>
"InclusiveStopFilter ('Row2')" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">TimeStampsFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a list of
+ timestamps. It returns those key-values whose timestamps matches
any of the specified
+ timestamps</para>
+ <para> <emphasis role="bold">Syntax:</emphasis> TimeStampsFilter
(<timestamp>, <timestamp>, ... ,<timestamp>) </para>
+ <para> <emphasis role="bold">Example:</emphasis> "TimeStampsFilter
(5985489, 48895495, 58489845945)"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">RowFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a compare
+ operator and a comparator. It compares each row key with the
comparator using the
+ compare operator and if the comparison returns true, it returns
all the key-values in
+ that row</para>
+ <para><emphasis role="bold">Syntax:</emphasis> RowFilter
(<compareOp>, â<row_comparator>â) </para>
+ <para><emphasis role="bold">Example: </emphasis>"RowFilter (<=,
âxyz)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis role="underline">Family
Filter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a compare
+ operator and a comparator. It compares each qualifier name with
the comparator using
+ the compare operator and if the comparison returns true, it
returns all the key-values
+ in that column</para>
+ <para><emphasis role="bold">Syntax:</emphasis> QualifierFilter
(<compareOp>, â<qualifier_comparator>â) </para>
+ <para><emphasis role="bold">Example:</emphasis> "QualifierFilter
(=, âColumn1â)"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">QualifierFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a compare
+ operator and a comparator. It compares each qualifier name with
the comparator using
+ the compare operator and if the comparison returns true, it
returns all the key-values
+ in that column</para>
+ <para><emphasis role="bold">Syntax:</emphasis> QualifierFilter
(<compareOp>,â<qualifier_comparator>â) </para>
+ <para><emphasis role="bold">Example:</emphasis> "QualifierFilter
(=,âColumn1â)"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">ValueFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a compare operator and a
+ comparator. It compares each value with the comparator using the
compare operator and
+ if the comparison returns true, it returns that key-value</para>
+ <para><emphasis role="bold">Syntax:</emphasis> ValueFilter
(<compareOp>,â<value_comparator>â) </para>
+ <para><emphasis role="bold">Example:</emphasis> "ValueFilter (!=,
âValueâ)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">DependentColumnFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes two arguments â a family
+ and a qualifier. It tries to locate this column in each row and
returns all key-values
+ in that row that have the same timestamp. If the row doesnât
contain the specified
+ column â none of the key-values in that row will be
returned.</para>
+ <para>The filter can also take an optional boolean argument â
dropDependentColumn. If set to true, the column we were depending on doesnât
get returned.</para>
+ <para>The filter can also take two more additional optional
arguments â a compare operator and a value comparator, which are further
checks in addition to the family and qualifier. If the dependent column is
found, its value should also pass the value check and then only is its
timestamp taken into consideration</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
DependentColumnFilter (â<family>â, â<qualifier>â, <boolean>,
<compare operator>, â<value comparatorâ)</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
DependentColumnFilter (â<family>â, â<qualifier>â, <boolean>)
</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
DependentColumnFilter (â<family>â, â<qualifier>â) </para>
+ <para><emphasis role="bold">Example:</emphasis>
"DependentColumnFilter (âconfâ, âblacklistâ, false, >=, âzebraâ)"
</para>
+ <para><emphasis role="bold">Example:</emphasis>
"DependentColumnFilter (âconfâ, 'blacklist', true)"</para>
+ <para><emphasis role="bold">Example:</emphasis>
"DependentColumnFilter (âconfâ, 'blacklist')"</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">SingleColumnValueFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes a column family, a
+ qualifier, a compare operator and a comparator. If the specified
column is not found â
+ all the columns of that row will be emitted. If the column is
found and the comparison
+ with the comparator returns true, all the columns of the row
will be emitted. If the
+ condition fails, the row will not be emitted. </para>
+ <para>This filter also takes two additional optional boolean
arguments â filterIfColumnMissing and setLatestVersionOnly</para>
+ <para>If the filterIfColumnMissing flag is set to true the columns
of the row will not be emitted if the specified column to check is not found in
the row. The default value is false.</para>
+ <para>If the setLatestVersionOnly flag is set to false, it will
test previous versions (timestamps) too. The default value is true.</para>
+ <para>These flags are optional and if you must set neither or
both</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
SingleColumnValueFilter(<compare operator>, â<comparator>â,
â<family>â, â<qualifier>â,<filterIfColumnMissing_boolean>,
<latest_version_boolean>) </para>
+ <para><emphasis role="bold">Syntax:</emphasis>
SingleColumnValueFilter(<compare operator>, â<comparator>â,
â<family>â, â<qualifier>) </para>
+ <para><emphasis role="bold">Example:</emphasis>
"SingleColumnValueFilter (<=, âabcâ,âFamilyAâ, âColumn1â, true,
false)" </para>
+ <para><emphasis role="bold">Example:</emphasis>
"SingleColumnValueFilter (<=, âabcâ,âFamilyAâ, âColumn1â)"
</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">SingleColumnValueExcludeFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter
takes the same arguments and
+ behaves same as SingleColumnValueFilter â however, if the
column is found and the
+ condition passes, all the columns of the row will be emitted
except for the tested
+ column value. </para>
+ <para><emphasis role="bold">Syntax:</emphasis>
SingleColumnValueExcludeFilter(<compare operator>, '<comparator>',
'<family>', '<qualifier>',<latest_version_boolean>,
<filterIfColumnMissing_boolean>)</para>
+ <para><emphasis role="bold">Syntax:</emphasis>
SingleColumnValueExcludeFilter(<compare operator>, '<comparator>',
'<family>', '<qualifier>') </para>
+ <para><emphasis role="bold">Example:</emphasis>
"SingleColumnValueExcludeFilter (â<=â, âabcâ,âFamilyAâ,
âColumn1â, âfalseâ, âtrueâ)"</para>
+ <para><emphasis role="bold">Example:</emphasis>
"SingleColumnValueExcludeFilter (â<=â, âabcâ, âFamilyAâ,
âColumn1â)" </para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="bold"><emphasis
role="underline">ColumnRangeFilter</emphasis></emphasis></para>
+ <para><emphasis role="bold">Description:</emphasis> This filter is
used for selecting only those
+ keys with columns that are between minColumn and maxColumn. It
also takes two boolean
+ variables to indicate whether to include the minColumn and
maxColumn or not.</para>
+ <para>If you donât want to set the minColumn or the maxColumn
â you can pass in an empty argument.</para>
+ <para><emphasis role="bold">Syntax:</emphasis> ColumnRangeFilter
(â<minColumn>â, <minColumnInclusive_bool>, â<maxColumn>â,
<maxColumnInclusive_bool>)</para>
+ <para><emphasis role="bold">Example:</emphasis> "ColumnRangeFilter
(âabcâ, true, âxyzâ, false)"</para>
+ </listitem>
+
+ </orderedlist>
+ </section>
+ </section>
+
+
<section xml:id="daemons">
<title>Daemons</title>
<section xml:id="master"><title>Master</title>
