http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/1.3/user_manual/Shell_Commands.html ---------------------------------------------------------------------- diff --git a/1.3/user_manual/Shell_Commands.html b/1.3/user_manual/Shell_Commands.html new file mode 100644 index 0000000..1856c09 --- /dev/null +++ b/1.3/user_manual/Shell_Commands.html @@ -0,0 +1,827 @@ +<!DOCTYPE html> +<html lang="en"> +<head> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<meta charset="utf-8"> +<meta http-equiv="X-UA-Compatible" content="IE=edge"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css"; rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous"> +<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"> +<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css";> +<link href="/css/accumulo.css" rel="stylesheet" type="text/css"> + +<title>User Manual: Shell Commands</title> + +<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js";></script> +<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"; integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> +<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js";></script> +<script> + // show location of canonical site if not currently on the canonical site + $(function() { + var host = window.location.host; + if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') { + $('#non-canonical').show(); + } + }); + + $(function() { + // decorate section headers with anchors + return $("h2, h3, h4, h5, h6").each(function(i, el) { + var $el, icon, id; + $el = $(el); + id = $el.attr('id'); + icon = '<i class="fa fa-link"></i>'; + if (id) { + return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon)); + } + }); + }); + + // configure Google Analytics + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + if (ga.hasOwnProperty('loaded') && ga.loaded === true) { + ga('create', 'UA-50934829-1', 'apache.org'); + ga('send', 'pageview'); + } +</script> + +</head> +<body style="padding-top: 100px"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"/></a> + </div> + <div class="collapse navbar-collapse" id="navbar-items"> + <ul class="nav navbar-nav"> + <li class="nav-link"><a href="/downloads">Download</a></li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/release/accumulo-1.8.0/">1.8.0 (Latest)</a></li> + <li><a href="/release/accumulo-1.7.2/">1.7.2</a></li> + <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li> + <li><a href="/release/">Archive</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/user-manual/">User Manuals</a></li> + <li><a href="/javadocs/">Javadocs</a></li> + <li><a href="/examples/">Examples</a></li> + <li><a href="/notable_features">Features</a></li> + <li><a href="/screenshots">Screenshots</a></li> + <li><a href="/papers">Papers & Presentations</a></li> + <li><a href="/glossary">Glossary</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/get_involved">Get Involved</a></li> + <li><a href="/mailing_list">Mailing Lists</a></li> + <li><a href="/people">People</a></li> + <li><a href="/news">News Archive</a></li> + <li><a href="/projects">Community Projects</a></li> + <li><a href="/thanks">Thanks</a></li> + <li class="divider"></li> + <li class="dropdown-header">Governance</li> + <li><a href="/bylaws">Bylaws</a></li> + <li><a href="/governance/consensusBuilding">Consensus Building</a></li> + <li><a href="/governance/lazyConsensus">Lazy Consensus</a></li> + <li><a href="/governance/releasing">Releasing</a></li> + <li><a href="/governance/voting">Voting</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Development<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://issues.apache.org/jira/browse/ACCUMULO";>Issue Tracker <i class="fa fa-external-link"></i></a></li> + <li><a href="https://github.com/apache/accumulo/pulls";>Pull Requests <i class="fa fa-external-link"></i></a></li> + <li><a href="https://builds.apache.org/view/A/view/Accumulo";>Jenkins Builds <i class="fa fa-external-link"></i></a></li> + <li><a href="https://travis-ci.org/apache/accumulo";>TravisCI Builds <i class="fa fa-external-link"></i></a></li> + <li class="divider"></li> + <li class="dropdown-header">Guides</li> + <li><a href="/source">Source & Guide</a></li> + <li><a href="/git">Git Workflow</a></li> + <li><a href="/versioning">Versioning</a></li> + <li><a href="/contrib">Contrib Projects</a></li> + <li><a href="/rb">Review Board</a></li> + <li><a href="/releasing">Making Releases</a></li> + <li><a href="/verifying_releases">Verifying Releases</a></li> + </ul> + </li> + </ul> + <ul class="nav navbar-nav navbar-right"> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org";>Apache Homepage <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/licenses/LICENSE-2.0";>License <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/sponsorship";>Sponsorship <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/security";>Security <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/thanks";>Thanks <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct <i class="fa fa-external-link"></i></a></li> + </ul> + </li> + </ul> + </div> + </div> +</nav> + + + <div class="container"> + <div class="row"> + <div class="col-md-12"> + + <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;"> + Visit the official site at: <a href="https://accumulo.apache.org";>https://accumulo.apache.org</a> + </div> + <div id="content"> + + <h1 class="title">User Manual: Shell Commands</h1> + + <p>** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Administration.html">Administration</a> ** <a href="Contents.html">Contents</a>**</p> + +<h2 id="a-idshellcommandsa-shell-commands"><a id="Shell_Commands"></a> Shell Commands</h2> + +<p><strong>?</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: ? [ <command> <command> ] [-?] [-np] +description: provides information about the available commands + -?,-help display this help + -np,-no-pagination disables pagination of output +</code></pre> +</div> + +<p><strong>about</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: about [-?] [-v] +description: displays information about this program + -?,-help display this help + -v,-verbose displays details session information +</code></pre> +</div> + +<p><strong>addsplits</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: addsplits [<split> <split> ] [-?] [-b64] [-sf <filename>] -t <tableName> +description: add split points to an existing table + -?,-help display this help + -b64,-base64encoded decode encoded split points + -sf,-splits-file <filename> file with newline separated list of rows to add + to table + -t,-table <tableName> name of a table to add split points to +</code></pre> +</div> + +<p><strong>authenticate</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: authenticate <username> [-?] +description: verifies a user's credentials + -?,-help display this help +</code></pre> +</div> + +<p><strong>bye</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: bye [-?] +description: exits the shell + -?,-help display this help +</code></pre> +</div> + +<p><strong>classpath</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: classpath [-?] +description: lists the current files on the classpath + -?,-help display this help +</code></pre> +</div> + +<p><strong>clear</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: clear [-?] +description: clears the screen + -?,-help display this help +</code></pre> +</div> + +<p><strong>cls</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: cls [-?] +description: clears the screen + -?,-help display this help +</code></pre> +</div> + +<p><strong>compact</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: compact [-?] [-override] -p <pattern> | -t <tableName> +description: sets all tablets for a table to major compact as soon as possible + (based on current time) + -?,-help display this help + -override override a future scheduled compaction + -p,-pattern <pattern> regex pattern of table names to flush + -t,-table <tableName> name of a table to flush +</code></pre> +</div> + +<p><strong>config</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: config [-?] [-d <property> | -f <string> | -s <property=value>] [-np] + [-t <table>] +description: prints system properties and table specific properties + -?,-help display this help + -d,-delete <property> delete a per-table property + -f,-filter <string> show only properties that contain this string + -np,-no-pagination disables pagination of output + -s,-set <property=value> set a per-table property + -t,-table <table> display/set/delete properties for specified table +</code></pre> +</div> + +<p><strong>createtable</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: createtable <tableName> [-?] [-a + <<columnfamily>[:<columnqualifier>]=<aggregation_class>>] [-b64] + [-cc <table>] [-cs <table> | -sf <filename>] [-ndi] [-tl | -tm] +description: creates a new table, with optional aggregators and optionally + pre-split + -?,-help display this help + -a,-aggregator <<columnfamily>[:<columnqualifier>]=<aggregation_class>> + comma separated column=aggregator + -b64,-base64encoded decode encoded split points + -cc,-copy-config <table> table to copy configuration from + -cs,-copy-splits <table> table to copy current splits from + -ndi,-no-default-iterators prevents creation of the normal default iterator + set + -sf,-splits-file <filename> file with newline separated list of rows to + create a pre-split table + -tl,-time-logical use logical time + -tm,-time-millis use time in milliseconds +</code></pre> +</div> + +<p><strong>createuser</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: createuser <username> [-?] [-s <comma-separated-authorizations>] +description: creates a new user + -?,-help display this help + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations +</code></pre> +</div> + +<p><strong>debug</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: debug [ on | off ] [-?] +description: turns debug logging on or off + -?,-help display this help +</code></pre> +</div> + +<p><strong>delete</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: delete <row> <colfamily> <colqualifier> [-?] [-l <expression>] [-t + <timestamp>] +description: deletes a record from a table + -?,-help display this help + -l,-authorization-label <expression> formatted authorization label expression + -t,-timestamp <timestamp> timestamp to use for insert +</code></pre> +</div> + +<p><strong>deleteiter</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: deleteiter [-?] [-majc] [-minc] -n <itername> [-scan] [-t <table>] +description: deletes a table-specific iterator + -?,-help display this help + -majc,-major-compaction applied at major compaction + -minc,-minor-compaction applied at minor compaction + -n,-name <itername> iterator to delete + -scan,-scan-time applied at scan time + -t,-table <table> tableName +</code></pre> +</div> + +<p><strong>deletemany</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: deletemany [-?] [-b <start-row>] [-c + <<columnfamily>[:<columnqualifier>]>] [-e <end-row>] [-f] [-np] + [-s <comma-separated-authorizations>] [-st] +description: scans a table and deletes the resulting records + -?,-help display this help + -b,-begin-row <start-row> begin row (inclusive) + -c,-columns <<columnfamily>[:<columnqualifier>]> comma-separated columns + -e,-end-row <end-row> end row (inclusive) + -f,-force forces deletion without prompting + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + (all user auths are used if this argument is not specified) + -st,-show-timestamps enables displaying timestamps +</code></pre> +</div> + +<p><strong>deletescaniter</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: deletescaniter [-?] [-a] [-n <itername>] [-t <table>] +description: deletes a table-specific scan iterator so it is no longer used + during this shell session + -?,-help display this help + -a,-all delete all for tableName + -n,-name <itername> iterator to delete + -t,-table <table> tableName +</code></pre> +</div> + +<p><strong>deletetable</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: deletetable <tableName> [-?] +description: deletes a table + -?,-help display this help +</code></pre> +</div> + +<p><strong>deleteuser</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: deleteuser <username> [-?] +description: deletes a user + -?,-help display this help +</code></pre> +</div> + +<p><strong>droptable</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: droptable <tableName> [-?] +description: deletes a table + -?,-help display this help +</code></pre> +</div> + +<p><strong>dropuser</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: dropuser <username> [-?] +description: deletes a user + -?,-help display this help +</code></pre> +</div> + +<p><strong>egrep</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: egrep <regex> <regex> [-?] [-b <start-row>] [-c + <<columnfamily>[:<columnqualifier>]>] [-e <end-row>] [-np] [-s + <comma-separated-authorizations>] [-st] [-t <arg>] +description: egreps a table in parallel on the server side (uses java regex) + -?,-help display this help + -b,-begin-row <start-row> begin row (inclusive) + -c,-columns <<columnfamily>[:<columnqualifier>]> comma-separated columns + -e,-end-row <end-row> end row (inclusive) + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + (all user auths are used if this argument is not specified) + -st,-show-timestamps enables displaying timestamps + -t,-num-threads <arg> num threads +</code></pre> +</div> + +<p><strong>execfile</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: execfile [-?] [-v] +description: specifies a file containing accumulo commands to execute + -?,-help display this help + -v,-verbose displays command prompt as commands are executed +</code></pre> +</div> + +<p><strong>exit</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: exit [-?] +description: exits the shell + -?,-help display this help +</code></pre> +</div> + +<p><strong>flush</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: flush [-?] -p <pattern> | -t <tableName> +description: makes a best effort to flush tables from memory to disk + -?,-help display this help + -p,-pattern <pattern> regex pattern of table names to flush + -t,-table <tableName> name of a table to flush +</code></pre> +</div> + +<p><strong>formatter</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: formatter [-?] -f <className> | -l | -r +description: specifies a formatter to use for displaying database entries + -?,-help display this help + -f,-formatter <className> fully qualified name of formatter class to use + -l,-list display the current formatter + -r,-reset reset to default formatter +</code></pre> +</div> + +<p><strong>getauths</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: getauths [-?] [-u <user>] +description: displays the maximum scan authorizations for a user + -?,-help display this help + -u,-user <user> user to operate on +</code></pre> +</div> + +<p><strong>getgroups</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: getgroups [-?] -t <table> +description: gets the locality groups for a given table + -?,-help display this help + -t,-table <table> get locality groups for specified table +</code></pre> +</div> + +<p><strong>getsplits</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: getsplits [-?] [-b64] [-m <num>] [-o <file>] [-v] +description: retrieves the current split points for tablets in the current table + -?,-help display this help + -b64,-base64encoded encode the split points + -m,-max <num> specifies the maximum number of splits to create + -o,-output <file> specifies a local file to write the splits to + -v,-verbose print out the tablet information with start/end rows +</code></pre> +</div> + +<p><strong>grant</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: grant <permission> [-?] -p <pattern> | -s | -t <table> -u <username> +description: grants system or table permissions for a user + -?,-help display this help + -p,-pattern <pattern> regex pattern of tables to grant permissions on + -s,-system grant a system permission + -t,-table <table> grant a table permission on this table + -u,-user <username> user to operate on +</code></pre> +</div> + +<p><strong>grep</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: grep <term> <term> [-?] [-b <start-row>] [-c + <<columnfamily>[:<columnqualifier>]>] [-e <end-row>] [-np] [-s + <comma-separated-authorizations>] [-st] [-t <arg>] +description: searches a table for a substring, in parallel, on the server side + -?,-help display this help + -b,-begin-row <start-row> begin row (inclusive) + -c,-columns <<columnfamily>[:<columnqualifier>]> comma-separated columns + -e,-end-row <end-row> end row (inclusive) + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + (all user auths are used if this argument is not specified) + -st,-show-timestamps enables displaying timestamps + -t,-num-threads <arg> num threads +</code></pre> +</div> + +<p><strong>help</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: help [ <command> <command> ] [-?] [-np] +description: provides information about the available commands + -?,-help display this help + -np,-no-pagination disables pagination of output +</code></pre> +</div> + +<p><strong>importdirectory</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: importdirectory <directory> <failureDirectory> [-?] [-a <num>] [-f <num>] + [-g] [-v] +description: bulk imports an entire directory of data files to the current table + -?,-help display this help + -a,-numAssignThreads <num> number of assign threads for import (default: 20) + -f,-numFileThreads <num> number of threads to process files (default: 8) + -g,-disableGC prevents imported files from being deleted by the garbage + collector + -v,-verbose displays statistics from the import +</code></pre> +</div> + +<p><strong>info</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: info [-?] [-v] +description: displays information about this program + -?,-help display this help + -v,-verbose displays details session information +</code></pre> +</div> + +<p><strong>insert</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: insert <row> <colfamily> <colqualifier> <value> [-?] [-l <expression>] [-t + <timestamp>] +description: inserts a record + -?,-help display this help + -l,-authorization-label <expression> formatted authorization label expression + -t,-timestamp <timestamp> timestamp to use for insert +</code></pre> +</div> + +<p><strong>listscans</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: listscans [-?] [-np] [-ts <tablet server>] +description: list what scans are currently running in accumulo. See the + org.apache.accumulo.core.client.admin.ActiveScan javadoc for more information + about columns. + -?,-help display this help + -np,-no-pagination disables pagination of output + -ts,-tabletServer <tablet server> list scans for a specific tablet server +</code></pre> +</div> + +<p><strong>masterstate</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: masterstate <NORMAL|SAFE_MODE|CLEAN_STOP> [-?] +description: set the master state: NORMAL, SAFE_MODE or CLEAN_STOP + -?,-help display this help +</code></pre> +</div> + +<p><strong>offline</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: offline [-?] -p <pattern> | -t <tableName> +description: starts the process of taking table offline + -?,-help display this help + -p,-pattern <pattern> regex pattern of table names to flush + -t,-table <tableName> name of a table to flush +</code></pre> +</div> + +<p><strong>online</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: online [-?] -p <pattern> | -t <tableName> +description: starts the process of putting a table online + -?,-help display this help + -p,-pattern <pattern> regex pattern of table names to flush + -t,-table <tableName> name of a table to flush +</code></pre> +</div> + +<p><strong>passwd</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: passwd [-?] [-u <user>] +description: changes a user's password + -?,-help display this help + -u,-user <user> user to operate on +</code></pre> +</div> + +<p><strong>quit</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: quit [-?] +description: exits the shell + -?,-help display this help +</code></pre> +</div> + +<p><strong>renametable</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: renametable <current table name> <new table name> [-?] +description: rename a table + -?,-help display this help +</code></pre> +</div> + +<p><strong>revoke</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: revoke <permission> [-?] -s | -t <table> -u <username> +description: revokes system or table permissions from a user + -?,-help display this help + -s,-system revoke a system permission + -t,-table <table> revoke a table permission on this table + -u,-user <username> user to operate on +</code></pre> +</div> + +<p><strong>scan</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: scan [-?] [-b <start-row>] [-c <<columnfamily>[:<columnqualifier>]>] [-e + <end-row>] [-np] [-s <comma-separated-authorizations>] [-st] +description: scans the table, and displays the resulting records + -?,-help display this help + -b,-begin-row <start-row> begin row (inclusive) + -c,-columns <<columnfamily>[:<columnqualifier>]> comma-separated columns + -e,-end-row <end-row> end row (inclusive) + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + (all user auths are used if this argument is not specified) + -st,-show-timestamps enables displaying timestamps +</code></pre> +</div> + +<p><strong>select</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: select <row> <columnfamily> <columnqualifier> [-?] [-np] [-s + <comma-separated-authorizations>] [-st] +description: scans for and displays a single record + -?,-help display this help + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + -st,-show-timestamps enables displaying timestamps +</code></pre> +</div> + +<p><strong>selectrow</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: selectrow <row> [-?] [-np] [-s <comma-separated-authorizations>] [-st] +description: scans a single row and displays all resulting records + -?,-help display this help + -np,-no-pagination disables pagination of output + -s,-scan-authorizations <comma-separated-authorizations> scan authorizations + -st,-show-timestamps enables displaying timestamps +</code></pre> +</div> + +<p><strong>setauths</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setauths [-?] -c | -s <comma-separated-authorizations> [-u <user>] +description: sets the maximum scan authorizations for a user + -?,-help display this help + -c,-clear-authorizations clears the scan authorizations + -s,-scan-authorizations <comma-separated-authorizations> set the scan + authorizations + -u,-user <user> user to operate on +</code></pre> +</div> + +<p><strong>setgroups</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setgroups <group>=<col fam>,<col fam> <group>=<col fam>,<col fam> + [-?] -t <table> +description: sets the locality groups for a given table (for binary or commas, + use Java API) + -?,-help display this help + -t,-table <table> get locality groups for specified table +</code></pre> +</div> + +<p><strong>setiter</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setiter [-?] -agg | -class <name> | -filter | -nolabel | -regex | -vers + [-majc] [-minc] [-n <itername>] -p <pri> [-scan] [-t <table>] +description: sets a table-specific iterator + -?,-help display this help + -agg,-aggregator an aggregating type + -class,-class-name <name> a java class type + -filter,-filter a filtering type + -majc,-major-compaction applied at major compaction + -minc,-minor-compaction applied at minor compaction + -n,-name <itername> iterator to set + -nolabel,-no-label a no-labeling type + -p,-priority <pri> the order in which the iterator is applied + -regex,-regular-expression a regex matching type + -scan,-scan-time applied at scan time + -t,-table <table> tableName + -vers,-version a versioning type +</code></pre> +</div> + +<p><strong>setscaniter</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setscaniter [-?] -agg | -class <name> | -filter | -nolabel | -regex | + -vers [-n <itername>] -p <pri> [-t <table>] +description: sets a table-specific scan iterator for this shell session + -?,-help display this help + -agg,-aggregator an aggregating type + -class,-class-name <name> a java class type + -filter,-filter a filtering type + -n,-name <itername> iterator to set + -nolabel,-no-label a no-labeling type + -p,-priority <pri> the order in which the iterator is applied + -regex,-regular-expression a regex matching type + -t,-table <table> tableName + -vers,-version a versioning type +</code></pre> +</div> + +<p><strong>systempermissions</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: systempermissions [-?] +description: displays a list of valid system permissions + -?,-help display this help +</code></pre> +</div> + +<p><strong>table</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: table <tableName> [-?] +description: switches to the specified table + -?,-help display this help +</code></pre> +</div> + +<p><strong>tablepermissions</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: tablepermissions [-?] +description: displays a list of valid table permissions + -?,-help display this help +</code></pre> +</div> + +<p><strong>tables</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: tables [-?] [-l] +description: displays a list of all existing tables + -?,-help display this help + -l,-list-ids display internal table ids along with the table name +</code></pre> +</div> + +<p><strong>trace</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: trace [ on | off ] [-?] +description: turns trace logging on or off + -?,-help display this help +</code></pre> +</div> + +<p><strong>user</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: user <username> [-?] +description: switches to the specified user + -?,-help display this help +</code></pre> +</div> + +<p><strong>userpermissions</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: userpermissions [-?] [-u <user>] +description: displays a user's system and table permissions + -?,-help display this help + -u,-user <user> user to operate on +</code></pre> +</div> + +<p><strong>users</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: users [-?] +description: displays a list of existing users + -?,-help display this help +</code></pre> +</div> + +<p><strong>whoami</strong></p> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: whoami [-?] +description: reports the current user name + -?,-help display this help +</code></pre> +</div> + +<hr /> + +<p>** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Administration.html">Administration</a> ** <a href="Contents.html">Contents</a>**</p> + + + </div> + + +<footer> + + <p><a href="https://www.apache.org";><img src="/images/feather-small.gif" alt="Apache Software Foundation" id="asf-logo" height="100" /></a></p> + + <p>Copyright © 2011-2016 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>.</p> + +</footer> + + + </div> + </div> + </div> +</body> +</html>
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/1.3/user_manual/Table_Configuration.html ---------------------------------------------------------------------- diff --git a/1.3/user_manual/Table_Configuration.html b/1.3/user_manual/Table_Configuration.html new file mode 100644 index 0000000..9626f80 --- /dev/null +++ b/1.3/user_manual/Table_Configuration.html @@ -0,0 +1,507 @@ +<!DOCTYPE html> +<html lang="en"> +<head> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<meta charset="utf-8"> +<meta http-equiv="X-UA-Compatible" content="IE=edge"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css"; rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous"> +<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"> +<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css";> +<link href="/css/accumulo.css" rel="stylesheet" type="text/css"> + +<title>User Manual: Table Configuration</title> + +<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js";></script> +<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"; integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> +<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js";></script> +<script> + // show location of canonical site if not currently on the canonical site + $(function() { + var host = window.location.host; + if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') { + $('#non-canonical').show(); + } + }); + + $(function() { + // decorate section headers with anchors + return $("h2, h3, h4, h5, h6").each(function(i, el) { + var $el, icon, id; + $el = $(el); + id = $el.attr('id'); + icon = '<i class="fa fa-link"></i>'; + if (id) { + return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon)); + } + }); + }); + + // configure Google Analytics + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + if (ga.hasOwnProperty('loaded') && ga.loaded === true) { + ga('create', 'UA-50934829-1', 'apache.org'); + ga('send', 'pageview'); + } +</script> + +</head> +<body style="padding-top: 100px"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"/></a> + </div> + <div class="collapse navbar-collapse" id="navbar-items"> + <ul class="nav navbar-nav"> + <li class="nav-link"><a href="/downloads">Download</a></li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/release/accumulo-1.8.0/">1.8.0 (Latest)</a></li> + <li><a href="/release/accumulo-1.7.2/">1.7.2</a></li> + <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li> + <li><a href="/release/">Archive</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/user-manual/">User Manuals</a></li> + <li><a href="/javadocs/">Javadocs</a></li> + <li><a href="/examples/">Examples</a></li> + <li><a href="/notable_features">Features</a></li> + <li><a href="/screenshots">Screenshots</a></li> + <li><a href="/papers">Papers & Presentations</a></li> + <li><a href="/glossary">Glossary</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/get_involved">Get Involved</a></li> + <li><a href="/mailing_list">Mailing Lists</a></li> + <li><a href="/people">People</a></li> + <li><a href="/news">News Archive</a></li> + <li><a href="/projects">Community Projects</a></li> + <li><a href="/thanks">Thanks</a></li> + <li class="divider"></li> + <li class="dropdown-header">Governance</li> + <li><a href="/bylaws">Bylaws</a></li> + <li><a href="/governance/consensusBuilding">Consensus Building</a></li> + <li><a href="/governance/lazyConsensus">Lazy Consensus</a></li> + <li><a href="/governance/releasing">Releasing</a></li> + <li><a href="/governance/voting">Voting</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Development<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://issues.apache.org/jira/browse/ACCUMULO";>Issue Tracker <i class="fa fa-external-link"></i></a></li> + <li><a href="https://github.com/apache/accumulo/pulls";>Pull Requests <i class="fa fa-external-link"></i></a></li> + <li><a href="https://builds.apache.org/view/A/view/Accumulo";>Jenkins Builds <i class="fa fa-external-link"></i></a></li> + <li><a href="https://travis-ci.org/apache/accumulo";>TravisCI Builds <i class="fa fa-external-link"></i></a></li> + <li class="divider"></li> + <li class="dropdown-header">Guides</li> + <li><a href="/source">Source & Guide</a></li> + <li><a href="/git">Git Workflow</a></li> + <li><a href="/versioning">Versioning</a></li> + <li><a href="/contrib">Contrib Projects</a></li> + <li><a href="/rb">Review Board</a></li> + <li><a href="/releasing">Making Releases</a></li> + <li><a href="/verifying_releases">Verifying Releases</a></li> + </ul> + </li> + </ul> + <ul class="nav navbar-nav navbar-right"> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org";>Apache Homepage <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/licenses/LICENSE-2.0";>License <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/sponsorship";>Sponsorship <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/security";>Security <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/thanks";>Thanks <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct <i class="fa fa-external-link"></i></a></li> + </ul> + </li> + </ul> + </div> + </div> +</nav> + + + <div class="container"> + <div class="row"> + <div class="col-md-12"> + + <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;"> + Visit the official site at: <a href="https://accumulo.apache.org";>https://accumulo.apache.org</a> + </div> + <div id="content"> + + <h1 class="title">User Manual: Table Configuration</h1> + + <p>** Next:** <a href="Table_Design.html">Table Design</a> ** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Writing_Accumulo_Clients.html">Writing Accumulo Clients</a> ** <a href="Contents.html">Contents</a>**</p> + +<p><a id="CHILD_LINKS"></a><strong>Subsections</strong></p> + +<ul> + <li><a href="Table_Configuration.html#Locality_Groups">Locality Groups</a></li> + <li><a href="Table_Configuration.html#Constraints">Constraints</a></li> + <li><a href="Table_Configuration.html#Bloom_Filters">Bloom Filters</a></li> + <li><a href="Table_Configuration.html#Iterators">Iterators</a></li> + <li><a href="Table_Configuration.html#Aggregating_Iterators">Aggregating Iterators</a></li> + <li><a href="Table_Configuration.html#Block_Cache">Block Cache</a></li> +</ul> + +<hr /> + +<h2 id="a-idtableconfigurationa-table-configuration"><a id="Table_Configuration"></a> Table Configuration</h2> + +<p>Accumulo tables have a few options that can be configured to alter the default behavior of Accumulo as well as improve performance based on the data stored. These include locality groups, constraints, and iterators.</p> + +<h2 id="a-idlocalitygroupsa-locality-groups"><a id="Locality_Groups"></a> Locality Groups</h2> + +<p>Accumulo supports storing of sets of column families separately on disk to allow clients to scan over columns that are frequently used together efficient and to avoid scanning over column families that are not requested. After a locality group is set Scanner and BatchScanner operations will automatically take advantage of them whenever the fetchColumnFamilies() method is used.</p> + +<p>By default tables place all column families into the same ``defaultâ locality group. Additional locality groups can be configured anytime via the shell or programmatically as follows:</p> + +<h3 id="a-idmanaginglocalitygroupsviatheshella-managing-locality-groups-via-the-shell"><a id="Managing_Locality_Groups_via_the_Shell"></a> Managing Locality Groups via the Shell</h3> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setgroups <group>=<col fam>{,<col fam>}{ <group>=<col fam>{,<col +fam>}} [-?] -t <table> + +user@myinstance mytable> setgroups -t mytable group_one=colf1,colf2 + +user@myinstance mytable> getgroups -t mytable +group_one=colf1,colf2 +</code></pre> +</div> + +<h3 id="a-idmanaginglocalitygroupsviatheclientapia-managing-locality-groups-via-the-client-api"><a id="Managing_Locality_Groups_via_the_Client_API"></a> Managing Locality Groups via the Client API</h3> + +<div class="highlighter-rouge"><pre class="highlight"><code>Connector conn; + +HashMap<String,Set<Text>> localityGroups = + new HashMap<String, Set<Text>>(); + +HashSet<Text> metadataColumns = new HashSet<Text>(); +metadataColumns.add(new Text("domain")); +metadataColumns.add(new Text("link")); + +HashSet<Text> contentColumns = new HashSet<Text>(); +contentColumns.add(new Text("body")); +contentColumns.add(new Text("images")); + +localityGroups.put("metadata", metadataColumns); +localityGroups.put("content", contentColumns); + +conn.tableOperations().setLocalityGroups("mytable", localityGroups); + +// existing locality groups can be obtained as follows +Map<String, Set<Text>> groups = + conn.tableOperations().getLocalityGroups("mytable"); +</code></pre> +</div> + +<p>The assignment of Column Families to Locality Groups can be changed anytime. The physical movement of column families into their new locality groups takes place via the periodic Major Compaction process that takes place continuously in the background. Major Compaction can also be scheduled to take place immediately through the shell:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance mytable> compact -t mytable +</code></pre> +</div> + +<h2 id="a-idconstraintsa-constraints"><a id="Constraints"></a> Constraints</h2> + +<p>Accumulo supports constraints applied on mutations at insert time. This can be used to disallow certain inserts according to a user defined policy. Any mutation that fails to meet the requirements of the constraint is rejected and sent back to the client.</p> + +<p>Constraints can be enabled by setting a table property as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance mytable> config -t mytable -s table.constraint.1=com.test.ExampleConstraint +user@myinstance mytable> config -t mytable -s table.constraint.2=com.test.AnotherConstraint +user@myinstance mytable> config -t mytable -f constraint +---------+--------------------------------+---------------------------- +SCOPE | NAME | VALUE +---------+--------------------------------+---------------------------- +table | table.constraint.1............ | com.test.ExampleConstraint +table | table.constraint.2............ | com.test.AnotherConstraint +---------+--------------------------------+---------------------------- +</code></pre> +</div> + +<p>Currently there are no general-purpose constraints provided with the Accumulo distribution. New constraints can be created by writing a Java class that implements the org.apache.accumulo.core.constraints.Constraint interface.</p> + +<p>To deploy a new constraint, create a jar file containing the class implementing the new constraint and place it in the lib directory of the Accumulo installation. New constraint jars can be added to Accumulo and enabled without restarting but any change to an existing constraint class requires Accumulo to be restarted.</p> + +<p>An example of constraints can be found in <br /> +accumulo/docs/examples/README.constraints with corresponding code under <br /> +accumulo/src/examples/main/java/accumulo/examples/constraints .</p> + +<h2 id="a-idbloomfiltersa-bloom-filters"><a id="Bloom_Filters"></a> Bloom Filters</h2> + +<p>As mutations are applied to an Accumulo table, several files are created per tablet. If bloom filters are enabled, Accumulo will create and load a small data structure into memory to determine whether a file contains a given key before opening the file. This can speed up lookups considerably.</p> + +<p>To enable bloom filters, enter the following command in the Shell:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance> config -t mytable -s table.bloom.enabled=true +</code></pre> +</div> + +<p>An extensive example of using Bloom Filters can be found at <br /> +accumulo/docs/examples/README.bloom .</p> + +<h2 id="a-iditeratorsa-iterators"><a id="Iterators"></a> Iterators</h2> + +<p>Iterators provide a modular mechanism for adding functionality to be executed by TabletServers when scanning or compacting data. This allows users to efficiently summarize, filter, and aggregate data. In fact, the built-in features of cell-level security and age-off are implemented using Iterators.</p> + +<h3 id="a-idsettingiteratorsviatheshella-setting-iterators-via-the-shell"><a id="Setting_Iterators_via_the_Shell"></a> Setting Iterators via the Shell</h3> + +<div class="highlighter-rouge"><pre class="highlight"><code>usage: setiter [-?] -agg | -class <name> | -filter | -nolabel | +-regex | -vers [-majc] [-minc] [-n <itername>] -p <pri> [-scan] +[-t <table>] + +user@myinstance mytable> setiter -t mytable -scan -p 10 -n myiter +</code></pre> +</div> + +<h3 id="a-idsettingiteratorsprogrammaticallya-setting-iterators-programmatically"><a id="Setting_Iterators_Programmatically"></a> Setting Iterators Programmatically</h3> + +<div class="highlighter-rouge"><pre class="highlight"><code>scanner.setScanIterators( + 15, // priority + "com.company.MyIterator", // class name + "myiter"); // name this iterator +</code></pre> +</div> + +<p>Some iterators take additional parameters from client code, as in the following example:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>bscan.setIteratorOption( + "myiter", // iterator reference + "myoptionname", + "myoptionvalue"); +</code></pre> +</div> + +<p>Tables support separate Iterator settings to be applied at scan time, upon minor compaction and upon major compaction. For most uses, tables will have identical iterator settings for all three to avoid inconsistent results.</p> + +<h3 id="a-idversioningiteratorsandtimestampsa-versioning-iterators-and-timestamps"><a id="Versioning_Iterators_and_Timestamps"></a> Versioning Iterators and Timestamps</h3> + +<p>Accumulo provides the capability to manage versioned data through the use of timestamps within the Key. If a timestamp is not specified in the key created by the client then the system will set the timestamp to the current time. Two keys with identical rowIDs and columns but different timestamps are considered two versions of the same key. If two inserts are made into accumulo with the same rowID, column, and timestamp, then the behavior is non-deterministic.</p> + +<p>Timestamps are sorted in descending order, so the most recent data comes first. Accumulo can be configured to return the top k versions, or versions later than a given date. The default is to return the one most recent version.</p> + +<p>The version policy can be changed by changing the VersioningIterator options for a table as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance mytable> config -t mytable -s +table.iterator.scan.vers.opt.maxVersions=3 + +user@myinstance mytable> config -t mytable -s +table.iterator.minc.vers.opt.maxVersions=3 + +user@myinstance mytable> config -t mytable -s +table.iterator.majc.vers.opt.maxVersions=3 +</code></pre> +</div> + +<h4 id="a-idlogicaltimea-logical-time"><a id="Logical_Time"></a> Logical Time</h4> + +<p>Accumulo 1.2 introduces the concept of logical time. This ensures that timestamps set by accumulo always move forward. This helps avoid problems caused by TabletServers that have different time settings. The per tablet counter gives unique one up time stamps on a per mutation basis. When using time in milliseconds, if two things arrive within the same millisecond then both receive the same timestamp.</p> + +<p>A table can be configured to use logical timestamps at creation time as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance> createtable -tl logical +</code></pre> +</div> + +<h4 id="a-iddeletesa-deletes"><a id="Deletes"></a> Deletes</h4> + +<p>Deletes are special keys in accumulo that get sorted along will all the other data. When a delete key is inserted, accumulo will not show anything that has a timestamp less than or equal to the delete key. During major compaction, any keys older than a delete key are omitted from the new file created, and the omitted keys are removed from disk as part of the regular garbage collection process.</p> + +<h3 id="a-idfilteringiteratorsa-filtering-iterators"><a id="Filtering_Iterators"></a> Filtering Iterators</h3> + +<p>When scanning over a set of key-value pairs it is possible to apply an arbitrary filtering policy through the use of a FilteringIterator. These types of iterators return only key-value pairs that satisfy the filter logic. Accumulo has two built-in filtering iterators that can be configured on any table: AgeOff and RegEx. More can be added by writing a Java class that implements the <br /> +org.apache.accumulo.core.iterators.filter.Filter interface.</p> + +<p>To configure the AgeOff filter to remove data older than a certain date or a fixed amount of time from the present. The following example sets a table to delete everything inserted over 30 seconds ago:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance> createtable filtertest +user@myinstance filtertest> setiter -t filtertest -scan -minc -majc -p +10 -n myfilter -filter + +FilteringIterator uses Filters to accept or reject key/value pairs +----------> entering options: <filterPriorityNumber> +<ageoff|regex|filterClass> + +----------> set org.apache.accumulo.core.iterators.FilteringIterator option +(<name> <value>, hit enter to skip): 0 ageoff + +----------> set org.apache.accumulo.core.iterators.FilteringIterator option +(<name> <value>, hit enter to skip): +AgeOffFilter removes entries with timestamps more than <ttl> +milliseconds old + +----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter +currentTime, if set, use the given value as the absolute time in +milliseconds as the current time of day: + +----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter +ttl, time to live (milliseconds): 30000 + +user@myinstance filtertest> +user@myinstance filtertest> scan +user@myinstance filtertest> insert foo a b c +insert successful +user@myinstance filtertest> scan +foo a:b [] c + +... wait 30 seconds ... + +user@myinstance filtertest> scan +user@myinstance filtertest> +</code></pre> +</div> + +<p>To see the iterator settings for a table, use:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@example filtertest> config -t filtertest -f iterator +---------+------------------------------------------+------------------ +SCOPE | NAME | VALUE +---------+------------------------------------------+------------------ +table | table.iterator.majc.myfilter ........... | +10,org.apache.accumulo.core.iterators.FilteringIterator +table | table.iterator.majc.myfilter.opt.0 ..... | +org.apache.accumulo.core.iterators.filter.AgeOffFilter +table | table.iterator.majc.myfilter.opt.0.ttl . | 30000 +table | table.iterator.minc.myfilter ........... | +10,org.apache.accumulo.core.iterators.FilteringIterator +table | table.iterator.minc.myfilter.opt.0 ..... | +org.apache.accumulo.core.iterators.filter.AgeOffFilter +table | table.iterator.minc.myfilter.opt.0.ttl . | 30000 +table | table.iterator.scan.myfilter ........... | +10,org.apache.accumulo.core.iterators.FilteringIterator +table | table.iterator.scan.myfilter.opt.0 ..... | +org.apache.accumulo.core.iterators.filter.AgeOffFilter +table | table.iterator.scan.myfilter.opt.0.ttl . | 30000 +---------+------------------------------------------+------------------ +</code></pre> +</div> + +<h2 id="a-idaggregatingiteratorsa-aggregating-iterators"><a id="Aggregating_Iterators"></a> Aggregating Iterators</h2> + +<p>Accumulo allows aggregating iterators to be configured on tables and column families. When an aggregating iterator is set, the iterator is applied across the values associated with any keys that share rowID, column family, and column qualifier. This is similar to the reduce step in MapReduce, which applied some function to all the values associated with a particular key.</p> + +<p>For example, if an aggregating iterator were configured on a table and the following mutations were inserted:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>Row Family Qualifier Timestamp Value +rowID1 colfA colqA 20100101 1 +rowID1 colfA colqA 20100102 1 +</code></pre> +</div> + +<p>The table would reflect only one aggregate value:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>rowID1 colfA colqA - 2 +</code></pre> +</div> + +<p>Aggregating iterators can be enabled for a table as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>user@myinstance> createtable perDayCounts -a +day=org.apache.accumulo.core.iterators.aggregation.StringSummation + +user@myinstance perDayCounts> insert row1 day 20080101 1 +user@myinstance perDayCounts> insert row1 day 20080101 1 +user@myinstance perDayCounts> insert row1 day 20080103 1 +user@myinstance perDayCounts> insert row2 day 20080101 1 +user@myinstance perDayCounts> insert row3 day 20080101 1 + +user@myinstance perDayCounts> scan +row1 day:20080101 [] 2 +row1 day:20080103 [] 1 +row2 day:20080101 [] 2 +</code></pre> +</div> + +<p>Accumulo includes the following aggregators:</p> + +<ul> + <li><strong>LongSummation</strong>: expects values of type long and adds them.</li> + <li><strong>StringSummation</strong>: expects numbers represented as strings and adds them.</li> + <li><strong>StringMax</strong>: expects numbers as strings and retains the maximum number inserted.</li> + <li><strong>StringMin</strong>: expects numbers as strings and retains the minimum number inserted.</li> +</ul> + +<p>Additional Aggregators can be added by creating a Java class that implements <br /> +<strong>org.apache.accumulo.core.iterators.aggregation.Aggregator</strong> and adding a jar containing that class to Accumuloâs lib directory.</p> + +<p>An example of an aggregator can be found under <br /> +accumulo/src/examples/main/java/org/apache/accumulo/examples/aggregation/SortedSetAggregator.java</p> + +<h2 id="a-idblockcachea-block-cache"><a id="Block_Cache"></a> Block Cache</h2> + +<p>In order to increase throughput of commonly accessed entries, Accumulo employs a block cache. This block cache buffers data in memory so that it doesnât have to be read off of disk. The RFile format that Accumulo prefers is a mix of index blocks and data blocks, where the index blocks are used to find the appropriate data blocks. Typical queries to Accumulo result in a binary search over several index blocks followed by a linear scan of one or more data blocks.</p> + +<p>The block cache can be configured on a per-table basis, and all tablets hosted on a tablet server share a single resource pool. To configure the size of the tablet serverâs block cache, set the following properties:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>tserver.cache.data.size: Specifies the size of the cache for file data blocks. +tserver.cache.index.size: Specifies the size of the cache for file indices. +</code></pre> +</div> + +<p>To enable the block cache for your table, set the following properties:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>table.cache.block.enable: Determines whether file (data) block cache is enabled. +table.cache.index.enable: Determines whether index cache is enabled. +</code></pre> +</div> + +<p>The block cache can have a significant effect on alleviating hot spots, as well as reducing query latency. It is enabled by default for the !METADATA table.</p> + +<hr /> + +<p>** Next:** <a href="Table_Design.html">Table Design</a> ** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Writing_Accumulo_Clients.html">Writing Accumulo Clients</a> ** <a href="Contents.html">Contents</a>**</p> + + + </div> + + +<footer> + + <p><a href="https://www.apache.org";><img src="/images/feather-small.gif" alt="Apache Software Foundation" id="asf-logo" height="100" /></a></p> + + <p>Copyright © 2011-2016 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>.</p> + +</footer> + + + </div> + </div> + </div> +</body> +</html> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/c0655661/1.3/user_manual/Table_Design.html ---------------------------------------------------------------------- diff --git a/1.3/user_manual/Table_Design.html b/1.3/user_manual/Table_Design.html new file mode 100644 index 0000000..6da7cba --- /dev/null +++ b/1.3/user_manual/Table_Design.html @@ -0,0 +1,367 @@ +<!DOCTYPE html> +<html lang="en"> +<head> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<meta charset="utf-8"> +<meta http-equiv="X-UA-Compatible" content="IE=edge"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css"; rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous"> +<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"> +<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css";> +<link href="/css/accumulo.css" rel="stylesheet" type="text/css"> + +<title>User Manual: Table Design</title> + +<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js";></script> +<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"; integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> +<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js";></script> +<script> + // show location of canonical site if not currently on the canonical site + $(function() { + var host = window.location.host; + if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') { + $('#non-canonical').show(); + } + }); + + $(function() { + // decorate section headers with anchors + return $("h2, h3, h4, h5, h6").each(function(i, el) { + var $el, icon, id; + $el = $(el); + id = $el.attr('id'); + icon = '<i class="fa fa-link"></i>'; + if (id) { + return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon)); + } + }); + }); + + // configure Google Analytics + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + if (ga.hasOwnProperty('loaded') && ga.loaded === true) { + ga('create', 'UA-50934829-1', 'apache.org'); + ga('send', 'pageview'); + } +</script> + +</head> +<body style="padding-top: 100px"> + + <nav class="navbar navbar-default navbar-fixed-top"> + <div class="container"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200"/></a> + </div> + <div class="collapse navbar-collapse" id="navbar-items"> + <ul class="nav navbar-nav"> + <li class="nav-link"><a href="/downloads">Download</a></li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/release/accumulo-1.8.0/">1.8.0 (Latest)</a></li> + <li><a href="/release/accumulo-1.7.2/">1.7.2</a></li> + <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li> + <li><a href="/release/">Archive</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/user-manual/">User Manuals</a></li> + <li><a href="/javadocs/">Javadocs</a></li> + <li><a href="/examples/">Examples</a></li> + <li><a href="/notable_features">Features</a></li> + <li><a href="/screenshots">Screenshots</a></li> + <li><a href="/papers">Papers & Presentations</a></li> + <li><a href="/glossary">Glossary</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="/get_involved">Get Involved</a></li> + <li><a href="/mailing_list">Mailing Lists</a></li> + <li><a href="/people">People</a></li> + <li><a href="/news">News Archive</a></li> + <li><a href="/projects">Community Projects</a></li> + <li><a href="/thanks">Thanks</a></li> + <li class="divider"></li> + <li class="dropdown-header">Governance</li> + <li><a href="/bylaws">Bylaws</a></li> + <li><a href="/governance/consensusBuilding">Consensus Building</a></li> + <li><a href="/governance/lazyConsensus">Lazy Consensus</a></li> + <li><a href="/governance/releasing">Releasing</a></li> + <li><a href="/governance/voting">Voting</a></li> + </ul> + </li> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Development<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://issues.apache.org/jira/browse/ACCUMULO";>Issue Tracker <i class="fa fa-external-link"></i></a></li> + <li><a href="https://github.com/apache/accumulo/pulls";>Pull Requests <i class="fa fa-external-link"></i></a></li> + <li><a href="https://builds.apache.org/view/A/view/Accumulo";>Jenkins Builds <i class="fa fa-external-link"></i></a></li> + <li><a href="https://travis-ci.org/apache/accumulo";>TravisCI Builds <i class="fa fa-external-link"></i></a></li> + <li class="divider"></li> + <li class="dropdown-header">Guides</li> + <li><a href="/source">Source & Guide</a></li> + <li><a href="/git">Git Workflow</a></li> + <li><a href="/versioning">Versioning</a></li> + <li><a href="/contrib">Contrib Projects</a></li> + <li><a href="/rb">Review Board</a></li> + <li><a href="/releasing">Making Releases</a></li> + <li><a href="/verifying_releases">Verifying Releases</a></li> + </ul> + </li> + </ul> + <ul class="nav navbar-nav navbar-right"> + <li class="dropdown"> + <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a> + <ul class="dropdown-menu"> + <li><a href="https://www.apache.org";>Apache Homepage <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/licenses/LICENSE-2.0";>License <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/sponsorship";>Sponsorship <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/security";>Security <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/thanks";>Thanks <i class="fa fa-external-link"></i></a></li> + <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct <i class="fa fa-external-link"></i></a></li> + </ul> + </li> + </ul> + </div> + </div> +</nav> + + + <div class="container"> + <div class="row"> + <div class="col-md-12"> + + <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;"> + Visit the official site at: <a href="https://accumulo.apache.org";>https://accumulo.apache.org</a> + </div> + <div id="content"> + + <h1 class="title">User Manual: Table Design</h1> + + <p>** Next:** <a href="High_Speed_Ingest.html">High-Speed Ingest</a> ** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Table_Configuration.html">Table Configuration</a> ** <a href="Contents.html">Contents</a>**</p> + +<p><a id="CHILD_LINKS"></a><strong>Subsections</strong></p> + +<ul> + <li><a href="Table_Design.html#Basic_Table">Basic Table</a></li> + <li><a href="Table_Design.html#RowID_Design">RowID Design</a></li> + <li><a href="Table_Design.html#Indexing">Indexing</a></li> + <li><a href="Table_Design.html#Entity-Attribute_and_Graph_Tables">Entity-Attribute and Graph Tables</a></li> + <li><a href="Table_Design.html#Document-Partitioned_Indexing">Document-Partitioned Indexing</a></li> +</ul> + +<hr /> + +<h2 id="a-idtabledesigna-table-design"><a id="Table_Design"></a> Table Design</h2> + +<h2 id="a-idbasictablea-basic-table"><a id="Basic_Table"></a> Basic Table</h2> + +<p>Since Accumulo tables are sorted by row ID, each table can be thought of as being indexed by the row ID. Lookups performed row ID can be executed quickly, by doing a binary search, first across the tablets, and then within a tablet. Clients should choose a row ID carefully in order to support their desired application. A simple rule is to select a unique identifier as the row ID for each entity to be stored and assign all the other attributes to be tracked to be columns under this row ID. For example, if we have the following data in a comma-separated file:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code> userid,age,address,account-balance +</code></pre> +</div> + +<p>We might choose to store this data using the userid as the rowID and the rest of the data in column families:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>Mutation m = new Mutation(new Text(userid)); +m.put(new Text("age"), age); +m.put(new Text("address"), address); +m.put(new Text("balance"), account_balance); + +writer.add(m); +</code></pre> +</div> + +<p>We could then retrieve any of the columns for a specific userid by specifying the userid as the range of a scanner and fetching specific columns:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>Range r = new Range(userid, userid); // single row +Scanner s = conn.createScanner("userdata", auths); +s.setRange(r); +s.fetchColumnFamily(new Text("age")); + +for(Entry<Key,Value> entry : s) + System.out.println(entry.getValue().toString()); +</code></pre> +</div> + +<h2 id="a-idrowiddesigna-rowid-design"><a id="RowID_Design"></a> RowID Design</h2> + +<p>Often it is necessary to transform the rowID in order to have rows ordered in a way that is optimal for anticipated access patterns. A good example of this is reversing the order of components of internet domain names in order to group rows of the same parent domain together:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>com.google.code +com.google.labs +com.google.mail +com.yahoo.mail +com.yahoo.research +</code></pre> +</div> + +<p>Some data may result in the creation of very large rows - rows with many columns. In this case the table designer may wish to split up these rows for better load balancing while keeping them sorted together for scanning purposes. This can be done by appending a random substring at the end of the row:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>com.google.code_00 +com.google.code_01 +com.google.code_02 +com.google.labs_00 +com.google.mail_00 +com.google.mail_01 +</code></pre> +</div> + +<p>It could also be done by adding a string representation of some period of time such as date to the week or month:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>com.google.code_201003 +com.google.code_201004 +com.google.code_201005 +com.google.labs_201003 +com.google.mail_201003 +com.google.mail_201004 +</code></pre> +</div> + +<p>Appending dates provides the additional capability of restricting a scan to a given date range.</p> + +<h2 id="a-idindexinga-indexing"><a id="Indexing"></a> Indexing</h2> + +<p>In order to support lookups via more than one attribute of an entity, additional indexes can be built. However, because Accumulo tables can support any number of columns without specifying them beforehand, a single additional index will often suffice for supporting lookups of records in the main table. Here, the index has, as the rowID, the Value or Term from the main table, the column families are the same, and the column qualifier of the index table contains the rowID from the main table.</p> + +<p><img src="img2.png" alt="converted table" /></p> + +<p>Note: We store rowIDs in the column qualifier rather than the Value so that we can have more than one rowID associated with a particular term within the index. If we stored this in the Value we would only see one of the rows in which the value appears since Accumulo is configured by default to return the one most recent value associated with a key.</p> + +<p>Lookups can then be done by scanning the Index Table first for occurrences of the desired values in the columns specified, which returns a list of row ID from the main table. These can then be used to retrieve each matching record, in their entirety, or a subset of their columns, from the Main Table.</p> + +<p>To support efficient lookups of multiple rowIDs from the same table, the Accumulo client library provides a BatchScanner. Users specify a set of Ranges to the BatchScanner, which performs the lookups in multiple threads to multiple servers and returns an Iterator over all the rows retrieved. The rows returned are NOT in sorted order, as is the case with the basic Scanner interface.</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>// first we scan the index for IDs of rows matching our query + +Text term = new Text("mySearchTerm"); + +HashSet<Text> matchingRows = new HashSet<Text>(); + +Scanner indexScanner = createScanner("index", auths); +indexScanner.setRange(new Range(term, term)); + +// we retrieve the matching rowIDs and create a set of ranges +for(Entry<Key,Value> entry : indexScanner) + matchingRows.add(new Text(entry.getValue())); + +// now we pass the set of rowIDs to the batch scanner to retrieve them +BatchScanner bscan = conn.createBatchScanner("table", auths, 10); + +bscan.setRanges(matchingRows); +bscan.fetchFamily("attributes"); + +for(Entry<Key,Value> entry : scan) + System.out.println(e.getValue()); +</code></pre> +</div> + +<p>One advantage of the dynamic schema capabilities of Accumulo is that different fields may be indexed into the same physical table. However, it may be necessary to create different index tables if the terms must be formatted differently in order to maintain proper sort order. For example, real numbers must be formatted differently than their usual notation in order to be sorted correctly. In these cases, usually one index per unique data type will suffice.</p> + +<h2 id="a-identity-attributeandgraphtablesa-entity-attribute-and-graph-tables"><a id="Entity-Attribute_and_Graph_Tables"></a> Entity-Attribute and Graph Tables</h2> + +<p>Accumulo is ideal for storing entities and their attributes, especially of the attributes are sparse. It is often useful to join several datasets together on common entities within the same table. This can allow for the representation of graphs, including nodes, their attributes, and connections to other nodes.</p> + +<p>Rather than storing individual events, Entity-Attribute or Graph tables store aggregate information about the entities involved in the events and the relationships between entities. This is often preferrable when single events arenât very useful and when a continuously updated summarization is desired.</p> + +<p>The physical shema for an entity-attribute or graph table is as follows:</p> + +<p><img src="img3.png" alt="converted table" /></p> + +<p>For example, to keep track of employees, managers and products the following entity-attribute table could be used. Note that the weights are not always necessary and are set to 0 when not used.</p> + +<p><img src="img4.png" alt="converted table" /></p> + +<p>To allow efficient updating of edge weights, an aggregating iterator can be configured to add the value of all mutations applied with the same key. These types of tables can easily be created from raw events by simply extracting the entities, attributes, and relationships from individual events and inserting the keys into Accumulo each with a count of 1. The aggregating iterator will take care of maintaining the edge weights.</p> + +<h2 id="a-iddocument-partitionedindexinga-document-partitioned-indexing"><a id="Document-Partitioned_Indexing"></a> Document-Partitioned Indexing</h2> + +<p>Using a simple index as described above works well when looking for records that match one of a set of given criteria. When looking for records that match more than one criterion simultaneously, such as when looking for documents that contain all of the words <code class="highlighter-rouge">the' and </code>whiteâ and `houseâ, there are several issues.</p> + +<p>First is that the set of all records matching any one of the search terms must be sent to the client, which incurs a lot of network traffic. The second problem is that the client is responsible for performing set intersection on the sets of records returned to eliminate all but the records matching all search terms. The memory of the client may easily be overwhelmed during this operation.</p> + +<p>For these reasons Accumulo includes support for a scheme known as sharded indexing, in which these set operations can be performed at the TabletServers and decisions about which records to include in the result set can be made without incurring network traffic.</p> + +<p>This is accomplished via partitioning records into bins that each reside on at most one TabletServer, and then creating an index of terms per record within each bin as follows:</p> + +<p><img src="img5.png" alt="converted table" /></p> + +<p>Documents or records are mapped into bins by a user-defined ingest application. By storing the BinID as the RowID we ensure that all the information for a particular bin is contained in a single tablet and hosted on a single TabletServer since Accumulo never splits rows across tablets. Storing the Terms as column families serves to enable fast lookups of all the documents within this bin that contain the given term.</p> + +<p>Finally, we perform set intersection operations on the TabletServer via a special iterator called the Intersecting Iterator. Since documents are partitioned into many bins, a search of all documents must search every bin. We can use the BatchScanner to scan all bins in parallel. The Intersecting Iterator should be enabled on a BatchScanner within user query code as follows:</p> + +<div class="highlighter-rouge"><pre class="highlight"><code>Text[] terms = {new Text("the"), new Text("white"), new Text("house")}; + +BatchScanner bs = conn.createBatchScanner(table, auths, 20); +bs.setScanIterators(20, IntersectingIterator.class.getName(), "ii"); + +// tells scanner to look for terms in the column family and sends terms +bs.setScanIteratorOption("ii", + IntersectingIterator.columnFamiliesOptionName, + IntersectingIterator.encodeColumns(terms)); + +bs.setRanges(Collections.singleton(new Range())); + +for(Entry<Key,Value> entry : bs) { + System.out.println(" " + entry.getKey().getColumnQualifier()); +} +</code></pre> +</div> + +<p>This code effectively has the BatchScanner scan all tablets of a table, looking for documents that match all the given terms. Because all tablets are being scanned for every query, each query is more expensive than other Accumulo scans, which typically involve a small number of TabletServers. This reduces the number of concurrent queries supported and is subject to what is known as the `stragglerâ problem in which every query runs as slow as the slowest server participating.</p> + +<p>Of course, fast servers will return their results to the client which can display them to the user immediately while they wait for the rest of the results to arrive. If the results are unordered this is quite effective as the first results to arrive are as good as any others to the user.</p> + +<hr /> + +<p>** Next:** <a href="High_Speed_Ingest.html">High-Speed Ingest</a> ** Up:** <a href="accumulo_user_manual.html">Apache Accumulo User Manual Version 1.3</a> ** Previous:** <a href="Table_Configuration.html">Table Configuration</a> ** <a href="Contents.html">Contents</a>**</p> + + + </div> + + +<footer> + + <p><a href="https://www.apache.org";><img src="/images/feather-small.gif" alt="Apache Software Foundation" id="asf-logo" height="100" /></a></p> + + <p>Copyright © 2011-2016 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>.</p> + +</footer> + + + </div> + </div> + </div> +</body> +</html>
