Jekyll build from master:817a0ef Organized documentation
* Moved iterator_testing.md content to development_tools.md * Moved proxy docs from client.md to new proxy.md * Renamed analytics.md to mapreduce.md and moved combiner docs to iterators.md * Reordered development docs Project: http://git-wip-us.apache.org/repos/asf/accumulo-website/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo-website/commit/9ebc5f9a Tree: http://git-wip-us.apache.org/repos/asf/accumulo-website/tree/9ebc5f9a Diff: http://git-wip-us.apache.org/repos/asf/accumulo-website/diff/9ebc5f9a Branch: refs/heads/asf-site Commit: 9ebc5f9a1026b88da4673fcf4ef2b2f623620c4c Parents: 2c7f1e8 Author: Mike Walch <[email protected]> Authored: Fri May 26 10:16:20 2017 -0400 Committer: Mike Walch <[email protected]> Committed: Fri May 26 10:16:20 2017 -0400 ---------------------------------------------------------------------- .../configuration-management.html | 8 +- .../configuration-properties.html | 8 +- docs/unreleased/administration/fate.html | 8 +- .../unreleased/administration/installation.html | 8 +- docs/unreleased/administration/kerberos.html | 8 +- .../administration/monitoring-metrics.html | 8 +- docs/unreleased/administration/multivolume.html | 8 +- docs/unreleased/administration/replication.html | 8 +- docs/unreleased/administration/ssl.html | 8 +- docs/unreleased/administration/tracing.html | 8 +- docs/unreleased/development/analytics.html | 562 -------------- .../development/development_tools.html | 106 ++- .../development/high_speed_ingest.html | 8 +- .../unreleased/development/iterator_design.html | 722 ------------------ .../development/iterator_testing.html | 435 ----------- docs/unreleased/development/iterators.html | 754 +++++++++++++++++++ docs/unreleased/development/mapreduce.html | 516 +++++++++++++ docs/unreleased/development/proxy.html | 459 +++++++++++ docs/unreleased/development/sampling.html | 8 +- docs/unreleased/development/security.html | 8 +- docs/unreleased/development/summaries.html | 8 +- docs/unreleased/getting-started/clients.html | 131 +--- docs/unreleased/getting-started/design.html | 8 +- docs/unreleased/getting-started/shell.html | 8 +- .../getting-started/table_configuration.html | 8 +- .../getting-started/table_design.html | 8 +- docs/unreleased/index.html | 8 +- docs/unreleased/troubleshooting/advanced.html | 8 +- docs/unreleased/troubleshooting/basic.html | 8 +- .../troubleshooting/system-metadata-tables.html | 8 +- docs/unreleased/troubleshooting/tools.html | 8 +- feed.xml | 4 +- 32 files changed, 1936 insertions(+), 1937 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/configuration-management.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/configuration-management.html b/docs/unreleased/administration/configuration-management.html index a4e3b9a..100dddf 100644 --- a/docs/unreleased/administration/configuration-management.html +++ b/docs/unreleased/administration/configuration-management.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/configuration-properties.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/configuration-properties.html b/docs/unreleased/administration/configuration-properties.html index d39e643..d5217b5 100644 --- a/docs/unreleased/administration/configuration-properties.html +++ b/docs/unreleased/administration/configuration-properties.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/fate.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/fate.html b/docs/unreleased/administration/fate.html index 44fb258..176fe0d 100644 --- a/docs/unreleased/administration/fate.html +++ b/docs/unreleased/administration/fate.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/installation.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/installation.html b/docs/unreleased/administration/installation.html index 5d3d55d..028d892 100644 --- a/docs/unreleased/administration/installation.html +++ b/docs/unreleased/administration/installation.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/kerberos.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/kerberos.html b/docs/unreleased/administration/kerberos.html index 2c31e5f..b767a29 100644 --- a/docs/unreleased/administration/kerberos.html +++ b/docs/unreleased/administration/kerberos.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/monitoring-metrics.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/monitoring-metrics.html b/docs/unreleased/administration/monitoring-metrics.html index c407dc7..320ca9c 100644 --- a/docs/unreleased/administration/monitoring-metrics.html +++ b/docs/unreleased/administration/monitoring-metrics.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/multivolume.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/multivolume.html b/docs/unreleased/administration/multivolume.html index 8ed1b03..c85ce38 100644 --- a/docs/unreleased/administration/multivolume.html +++ b/docs/unreleased/administration/multivolume.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/replication.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/replication.html b/docs/unreleased/administration/replication.html index 82b2610..79379bc 100644 --- a/docs/unreleased/administration/replication.html +++ b/docs/unreleased/administration/replication.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/ssl.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/ssl.html b/docs/unreleased/administration/ssl.html index 2a9e708..89c1943 100644 --- a/docs/unreleased/administration/ssl.html +++ b/docs/unreleased/administration/ssl.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/administration/tracing.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/administration/tracing.html b/docs/unreleased/administration/tracing.html index 1a9d0e7..6ded5e5 100644 --- a/docs/unreleased/administration/tracing.html +++ b/docs/unreleased/administration/tracing.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/development/analytics.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/development/analytics.html b/docs/unreleased/development/analytics.html deleted file mode 100644 index 97f2e27..0000000 --- a/docs/unreleased/development/analytics.html +++ /dev/null @@ -1,562 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> -<meta charset="utf-8"> -<meta http-equiv="X-UA-Compatible" content="IE=edge"> -<meta name="viewport" content="width=device-width, initial-scale=1"> -<link href="https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/paper/bootstrap.min.css"; rel="stylesheet" integrity="sha384-awusxf8AUojygHf2+joICySzB780jVvQaVCAt1clU3QsyAitLGul28Qxb2r1e5g+" crossorigin="anonymous"> -<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"> -<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css";> -<link href="/css/accumulo.css" rel="stylesheet" type="text/css"> - -<title>Accumulo Documentation - Analytics</title> - -<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js";></script> -<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"; integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> -<script type="text/javascript" src="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.js";></script> -<script> - // show location of canonical site if not currently on the canonical site - $(function() { - var host = window.location.host; - if (typeof host !== 'undefined' && host !== 'accumulo.apache.org') { - $('#non-canonical').show(); - } - }); - - $(function() { - // decorate section headers with anchors - return $("h2, h3, h4, h5, h6").each(function(i, el) { - var $el, icon, id; - $el = $(el); - id = $el.attr('id'); - icon = '<i class="fa fa-link"></i>'; - if (id) { - return $el.append($("<a />").addClass("header-link").attr("href", "#" + id).html(icon)); - } - }); - }); - - // configure Google Analytics - (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ - (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), - m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) - })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); - - if (ga.hasOwnProperty('loaded') && ga.loaded === true) { - ga('create', 'UA-50934829-1', 'apache.org'); - ga('send', 'pageview'); - } -</script> - -</head> -<body style="padding-top: 100px"> - - <nav class="navbar navbar-default navbar-fixed-top"> - <div class="container"> - <div class="navbar-header"> - <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-items"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a href="/"><img id="nav-logo" alt="Apache Accumulo" class="img-responsive" src="/images/accumulo-logo.png" width="200" - /></a> - </div> - <div class="collapse navbar-collapse" id="navbar-items"> - <ul class="nav navbar-nav"> - <li class="nav-link"><a href="/downloads">Download</a></li> - <li class="dropdown"> - <a class="dropdown-toggle" data-toggle="dropdown" href="#">Releases<span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/release/accumulo-1.8.1/">1.8.1 (Latest)</a></li> - <li><a href="/release/accumulo-1.7.3/">1.7.3</a></li> - <li><a href="/release/accumulo-1.6.6/">1.6.6</a></li> - <li><a href="/release/">Archive</a></li> - </ul> - </li> - <li class="dropdown"> - <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/1.8/accumulo_user_manual.html">User Manual (1.8)</a></li> - <li><a href="/1.8/apidocs">Javadocs (1.8)</a></li> - <li><a href="/1.8/examples">Examples (1.8)</a></li> - <li><a href="/features">Features</a></li> - <li><a href="/glossary">Glossary</a></li> - <li><a href="/external-docs">External Docs</a></li> - <li><a href="/docs-archive/">Archive</a></li> - </ul> - </li> - <li class="dropdown"> - <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="/get_involved">Get Involved</a></li> - <li><a href="/mailing_list">Mailing Lists</a></li> - <li><a href="/people">People</a></li> - <li><a href="/related-projects">Related Projects</a></li> - <li><a href="/contributor/">Contributor Guide</a></li> - </ul> - </li> - </ul> - <ul class="nav navbar-nav navbar-right"> - <li class="dropdown"> - <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation<span class="caret"></span></a> - <ul class="dropdown-menu"> - <li><a href="https://www.apache.org";>Apache Homepage <i class="fa fa-external-link"></i></a></li> - <li><a href="https://www.apache.org/licenses/LICENSE-2.0";>License <i class="fa fa-external-link"></i></a></li> - <li><a href="https://www.apache.org/foundation/sponsorship";>Sponsorship <i class="fa fa-external-link"></i></a></li> - <li><a href="https://www.apache.org/security";>Security <i class="fa fa-external-link"></i></a></li> - <li><a href="https://www.apache.org/foundation/thanks";>Thanks <i class="fa fa-external-link"></i></a></li> - <li><a href="https://www.apache.org/foundation/policies/conduct";>Code of Conduct <i class="fa fa-external-link"></i></a></li> - </ul> - </li> - </ul> - </div> - </div> -</nav> - - <div class="container"> - <div class="row"> - <div class="col-md-12"> - - <div id="non-canonical" style="display: none; background-color: #F0E68C; padding-left: 1em;"> - Visit the official site at: <a href="https://accumulo.apache.org";>https://accumulo.apache.org</a> - </div> - <div id="content"> - - <div class="alert alert-danger" role="alert">This documentation is for an unreleased version of Apache Accumulo that is currently under development! Check out the <a href="/1.8/accumulo_user_manual.html">documentation for the latest release</a>.</div> - -<div class="row"> - <div class="col-md-3"> - <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"> - <div class="panel panel-default"> - - - - - - - - - - <div class="panel-heading" role="tab" id="headingOne"> - <h4 class="panel-title"> - <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsegetting-started" aria-expanded="false" aria-controls="collapsegetting-started"> - Getting started - </a> - </h4> - </div> - <div id="collapsegetting-started" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne"> - <div class="panel-body"> - - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/design">Accumulo Design</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/clients">Accumulo Clients</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/shell">Accumulo Shell</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_design">Table Design</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/getting-started/table_configuration">Table Configuration</a></div> - - </div> - </div> - - - - - - - - - - - - <div class="panel-heading" role="tab" id="headingOne"> - <h4 class="panel-title"> - <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsedevelopment" aria-expanded="true" aria-controls="collapsedevelopment"> - Development - </a> - </h4> - </div> - <div id="collapsedevelopment" class="panel-collapse collapse in" role="tabpanel" aria-labelledby="headingOne"> - <div class="panel-body"> - - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/sampling">Sampling</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/summaries">Summary Statistics</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/security">Security</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - - </div> - </div> - - - - - - - - - - - - <div class="panel-heading" role="tab" id="headingOne"> - <h4 class="panel-title"> - <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapseadministration" aria-expanded="false" aria-controls="collapseadministration"> - Administration - </a> - </h4> - </div> - <div id="collapseadministration" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne"> - <div class="panel-body"> - - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/installation">Installation</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-management">Configuration Management</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/configuration-properties">Configuration Properties</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/monitoring-metrics">Monitoring & Metrics</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/tracing">Tracing</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/fate">FATE</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/multivolume">Multi-Volume Installations</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/ssl">SSL</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/kerberos">Kerberos</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/administration/replication">Replication</a></div> - - </div> - </div> - - - - - - - - - - - - - - - - - - - - - - <div class="panel-heading" role="tab" id="headingOne"> - <h4 class="panel-title"> - <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapsetroubleshooting" aria-expanded="false" aria-controls="collapsetroubleshooting"> - Troubleshooting - </a> - </h4> - </div> - <div id="collapsetroubleshooting" class="panel-collapse collapse" role="tabpanel" aria-labelledby="headingOne"> - <div class="panel-body"> - - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/basic">Basic Troubleshooting</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/advanced">Advanced Troubleshooting</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/tools">Troubleshooting Tools</a></div> - - <div class="row doc-sidebar-link"><a href="/docs/unreleased/troubleshooting/system-metadata-tables">System Metadata Tables</a></div> - - </div> - </div> - - - - </div> - </div> - </div> - <div class="col-md-9"> - - <p><a href="/docs/unreleased/">Accumulo unreleased docs</a> >> Development >> Analytics</p> - - - <h1>Analytics</h1> - - <p>Accumulo supports more advanced data processing than simply keeping keys -sorted and performing efficient lookups. Analytics can be developed by using -MapReduce and Iterators in conjunction with Accumulo tables.</p> - -<h2 id="mapreduce">MapReduce</h2> - -<p>Accumulo tables can be used as the source and destination of MapReduce jobs. To -use an Accumulo table with a MapReduce job (specifically with the new Hadoop API -as of version 0.20), configure the job parameters to use the AccumuloInputFormat -and AccumuloOutputFormat. Accumulo specific parameters can be set via these -two format classes to do the following:</p> - -<ul> - <li>Authenticate and provide user credentials for the input</li> - <li>Restrict the scan to a range of rows</li> - <li>Restrict the input to a subset of available columns</li> -</ul> - -<h3 id="mapper-and-reducer-classes">Mapper and Reducer classes</h3> - -<p>To read from an Accumulo table create a Mapper with the following class -parameterization and be sure to configure the AccumuloInputFormat.</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyMapper</span> <span class="kd">extends</span> <span class="n">Mapper</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">,</span><span class="n">WritableComparable</span><span class="o">,</span><span class="n">Writable</span><span class="o">></span> <span class="o">{</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">map</span><span class="o">(</span><span class="n">Key</span> <span class="n">k</span><span class="o">,</span> <span class="n">Value</span> <span class="n">v</span><span class="o">,</span> <span class="n">Context</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> - <span class="c1">// transform key and value data here</span> - <span class="o">}</span> -<span class="o">}</span> -</code></pre> -</div> - -<p>To write to an Accumulo table, create a Reducer with the following class -parameterization and be sure to configure the AccumuloOutputFormat. The key -emitted from the Reducer identifies the table to which the mutation is sent. This -allows a single Reducer to write to more than one table if desired. A default table -can be configured using the AccumuloOutputFormat, in which case the output table -name does not have to be passed to the Context object within the Reducer.</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyReducer</span> <span class="kd">extends</span> <span class="n">Reducer</span><span class="o"><</span><span class="n">WritableComparable</span><span class="o">,</span> <span class="n">Writable</span><span class="o">,</span> <span class="n">Text</span><span class="o">,</span> <span class="n">Mutation</span><span class="o">></span> <span class="o">{</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">reduce</span><span class="o">(</span><span class="n">WritableComparable</span> <span class="n">key</span><span class="o">,</span> <span class="n">Iterable</span><span class="o"><</span><span class="n">Text</span><span class="o">></span> <span class="n">values</span><span class="o">,</span> <span class="n">Context</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> - <span class="n">Mutation</span> <span class="n">m</span><span class="o">;</span> - <span class="c1">// create the mutation based on input key and value</span> - <span class="n">c</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="k">new</span> <span class="n">Text</span><span class="o">(</span><span class="s">"output-table"</span><span class="o">),</span> <span class="n">m</span><span class="o">);</span> - <span class="o">}</span> -<span class="o">}</span> -</code></pre> -</div> - -<p>The Text object passed as the output should contain the name of the table to which -this mutation should be applied. The Text can be null in which case the mutation -will be applied to the default table name specified in the AccumuloOutputFormat -options.</p> - -<h3 id="accumuloinputformat-options">AccumuloInputFormat options</h3> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Job</span> <span class="n">job</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Job</span><span class="o">(</span><span class="n">getConf</span><span class="o">());</span> -<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="na">setInputInfo</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> - <span class="s">"user"</span><span class="o">,</span> - <span class="s">"passwd"</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> - <span class="s">"table"</span><span class="o">,</span> - <span class="k">new</span> <span class="nf">Authorizations</span><span class="o">());</span> - -<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="na">setZooKeeperInstance</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="s">"myinstance"</span><span class="o">,</span> - <span class="s">"zooserver-one,zooserver-two"</span><span class="o">);</span> -</code></pre> -</div> - -<p><strong>Optional Settings:</strong></p> - -<p>To restrict Accumulo to a set of row ranges:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">></span> <span class="n">ranges</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">>();</span> -<span class="c1">// populate array list of row ranges ...</span> -<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="na">setRanges</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="n">ranges</span><span class="o">);</span> -</code></pre> -</div> - -<p>To restrict Accumulo to a list of columns:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>></span> <span class="n">columns</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>>();</span> -<span class="c1">// populate list of columns</span> -<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="na">fetchColumns</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="n">columns</span><span class="o">);</span> -</code></pre> -</div> - -<p>To use a regular expression to match row IDs:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">IteratorSetting</span> <span class="n">is</span> <span class="o">=</span> <span class="k">new</span> <span class="n">IteratorSetting</span><span class="o">(</span><span class="mi">30</span><span class="o">,</span> <span class="n">RexExFilter</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> -<span class="n">RegExFilter</span><span class="o">.</span><span class="na">setRegexs</span><span class="o">(</span><span class="n">is</span><span class="o">,</span> <span class="s">".*suffix"</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span> -<span class="n">AccumuloInputFormat</span><span class="o">.</span><span class="na">addIterator</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="n">is</span><span class="o">);</span> -</code></pre> -</div> - -<h3 id="accumulomultitableinputformat-options">AccumuloMultiTableInputFormat options</h3> - -<p>The AccumuloMultiTableInputFormat allows the scanning over multiple tables -in a single MapReduce job. Separate ranges, columns, and iterators can be -used for each table.</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">InputTableConfig</span> <span class="n">tableOneConfig</span> <span class="o">=</span> <span class="k">new</span> <span class="n">InputTableConfig</span><span class="o">();</span> -<span class="n">InputTableConfig</span> <span class="n">tableTwoConfig</span> <span class="o">=</span> <span class="k">new</span> <span class="n">InputTableConfig</span><span class="o">();</span> -</code></pre> -</div> - -<p>To set the configuration objects on the job:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">InputTableConfig</span><span class="o">></span> <span class="n">configs</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="o">,</span><span class="n">InputTableConfig</span><span class="o">>();</span> -<span class="n">configs</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"table1"</span><span class="o">,</span> <span class="n">tableOneConfig</span><span class="o">);</span> -<span class="n">configs</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"table2"</span><span class="o">,</span> <span class="n">tableTwoConfig</span><span class="o">);</span> -<span class="n">AccumuloMultiTableInputFormat</span><span class="o">.</span><span class="na">setInputTableConfigs</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="n">configs</span><span class="o">);</span> -</code></pre> -</div> - -<p><strong>Optional settings:</strong></p> - -<p>To restrict to a set of ranges:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">></span> <span class="n">tableOneRanges</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">>();</span> -<span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">></span> <span class="n">tableTwoRanges</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Range</span><span class="o">>();</span> -<span class="c1">// populate array lists of row ranges for tables...</span> -<span class="n">tableOneConfig</span><span class="o">.</span><span class="na">setRanges</span><span class="o">(</span><span class="n">tableOneRanges</span><span class="o">);</span> -<span class="n">tableTwoConfig</span><span class="o">.</span><span class="na">setRanges</span><span class="o">(</span><span class="n">tableTwoRanges</span><span class="o">);</span> -</code></pre> -</div> - -<p>To restrict Accumulo to a list of columns:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>></span> <span class="n">tableOneColumns</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>>();</span> -<span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>></span> <span class="n">tableTwoColumns</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Pair</span><span class="o"><</span><span class="n">Text</span><span class="o">,</span><span class="n">Text</span><span class="o">>>();</span> -<span class="c1">// populate lists of columns for each of the tables ...</span> -<span class="n">tableOneConfig</span><span class="o">.</span><span class="na">fetchColumns</span><span class="o">(</span><span class="n">tableOneColumns</span><span class="o">);</span> -<span class="n">tableTwoConfig</span><span class="o">.</span><span class="na">fetchColumns</span><span class="o">(</span><span class="n">tableTwoColumns</span><span class="o">);</span> -</code></pre> -</div> - -<p>To set scan iterators:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">List</span><span class="o"><</span><span class="n">IteratorSetting</span><span class="o">></span> <span class="n">tableOneIterators</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">IteratorSetting</span><span class="o">>();</span> -<span class="n">List</span><span class="o"><</span><span class="n">IteratorSetting</span><span class="o">></span> <span class="n">tableTwoIterators</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">IteratorSetting</span><span class="o">>();</span> -<span class="c1">// populate the lists of iterator settings for each of the tables ...</span> -<span class="n">tableOneConfig</span><span class="o">.</span><span class="na">setIterators</span><span class="o">(</span><span class="n">tableOneIterators</span><span class="o">);</span> -<span class="n">tableTwoConfig</span><span class="o">.</span><span class="na">setIterators</span><span class="o">(</span><span class="n">tableTwoIterators</span><span class="o">);</span> -</code></pre> -</div> - -<p>The name of the table can be retrieved from the input split:</p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">MyMapper</span> <span class="kd">extends</span> <span class="n">Mapper</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">,</span><span class="n">WritableComparable</span><span class="o">,</span><span class="n">Writable</span><span class="o">></span> <span class="o">{</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="nf">map</span><span class="o">(</span><span class="n">Key</span> <span class="n">k</span><span class="o">,</span> <span class="n">Value</span> <span class="n">v</span><span class="o">,</span> <span class="n">Context</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> - <span class="n">RangeInputSplit</span> <span class="n">split</span> <span class="o">=</span> <span class="o">(</span><span class="n">RangeInputSplit</span><span class="o">)</span><span class="n">c</span><span class="o">.</span><span class="na">getInputSplit</span><span class="o">();</span> - <span class="n">String</span> <span class="n">tableName</span> <span class="o">=</span> <span class="n">split</span><span class="o">.</span><span class="na">getTableName</span><span class="o">();</span> - <span class="c1">// do something with table name</span> - <span class="o">}</span> -<span class="o">}</span> -</code></pre> -</div> - -<h3 id="accumulooutputformat-options">AccumuloOutputFormat options</h3> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kt">boolean</span> <span class="n">createTables</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span> -<span class="n">String</span> <span class="n">defaultTable</span> <span class="o">=</span> <span class="s">"mytable"</span><span class="o">;</span> - -<span class="n">AccumuloOutputFormat</span><span class="o">.</span><span class="na">setOutputInfo</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> - <span class="s">"user"</span><span class="o">,</span> - <span class="s">"passwd"</span><span class="o">.</span><span class="na">getBytes</span><span class="o">(),</span> - <span class="n">createTables</span><span class="o">,</span> - <span class="n">defaultTable</span><span class="o">);</span> - -<span class="n">AccumuloOutputFormat</span><span class="o">.</span><span class="na">setZooKeeperInstance</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="s">"myinstance"</span><span class="o">,</span> - <span class="s">"zooserver-one,zooserver-two"</span><span class="o">);</span> -</code></pre> -</div> - -<p><strong>Optional Settings:</strong></p> - -<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="n">AccumuloOutputFormat</span><span class="o">.</span><span class="na">setMaxLatency</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="mi">300000</span><span class="o">);</span> <span class="c1">// milliseconds</span> -<span class="n">AccumuloOutputFormat</span><span class="o">.</span><span class="na">setMaxMutationBufferSize</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="mi">50000000</span><span class="o">);</span> <span class="c1">// bytes</span> -</code></pre> -</div> - -<p>The <a href="https://github.com/apache/accumulo-examples/blob/master/docs/mapred.md";>MapReduce example</a> -contains a complete example of using MapReduce with Accumulo.</p> - -<h2 id="combiners">Combiners</h2> - -<p>Many applications can benefit from the ability to aggregate values across common -keys. This can be done via Combiner iterators and is similar to the Reduce step in -MapReduce. This provides the ability to define online, incrementally updated -analytics without the overhead or latency associated with batch-oriented -MapReduce jobs.</p> - -<p>All that is needed to aggregate values of a table is to identify the fields over which -values will be grouped, insert mutations with those fields as the key, and configure -the table with a combining iterator that supports the summarizing operation -desired.</p> - -<p>The only restriction on an combining iterator is that the combiner developer -should not assume that all values for a given key have been seen, since new -mutations can be inserted at anytime. This precludes using the total number of -values in the aggregation such as when calculating an average, for example.</p> - -<h3 id="feature-vectors">Feature Vectors</h3> - -<p>An interesting use of combining iterators within an Accumulo table is to store -feature vectors for use in machine learning algorithms. For example, many -algorithms such as k-means clustering, support vector machines, anomaly detection, -etc. use the concept of a feature vector and the calculation of distance metrics to -learn a particular model. The columns in an Accumulo table can be used to efficiently -store sparse features and their weights to be incrementally updated via the use of an -combining iterator.</p> - -<h2 id="statistical-modeling">Statistical Modeling</h2> - -<p>Statistical models that need to be updated by many machines in parallel could be -similarly stored within an Accumulo table. For example, a MapReduce job that is -iteratively updating a global statistical model could have each map or reduce worker -reference the parts of the model to be read and updated through an embedded -Accumulo client.</p> - -<p>Using Accumulo this way enables efficient and fast lookups and updates of small -pieces of information in a random access pattern, which is complementary to -MapReduceâs sequential access model.</p> - - </div> -</div> - - </div> - - -<footer> - - <p><a href="https://www.apache.org/foundation/contributing";><img src="https://www.apache.org/images/SupportApache-small.png"; alt="Support the ASF" id="asf-logo" height="100" /></a></p> - - <p>Copyright © 2011-2017 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version 2.0</a>.</p> - -</footer> - - - </div> - </div> - </div> -</body> -</html> http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/development/development_tools.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/development/development_tools.html b/docs/unreleased/development/development_tools.html index c8afef8..4756323 100644 --- a/docs/unreleased/development/development_tools.html +++ b/docs/unreleased/development/development_tools.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div> @@ -322,6 +322,104 @@ Accumulo requires Hadoop, Zookeeper, the Accumulo master, a tablet server, etc. you want to write a unit test that uses Accumulo, you need a lot of infrastructure in place before your test can run.</p> +<h2 id="iterator-test-harness">Iterator Test Harness</h2> + +<p>Iterators, while extremely powerful, are notoriously difficult to test. While the API defines +the methods an Iterator must implement and each methodâs functionality, the actual invocation +of these methods by Accumulo TabletServers can be surprisingly difficult to mimic in unit tests.</p> + +<p>The Apache Accumulo âIterator Test Harnessâ is designed to provide a generalized testing framework +for all Accumulo Iterators to leverage to identify common pitfalls in user-created Iterators.</p> + +<h3 id="framework-use">Framework Use</h3> + +<p>The harness provides an abstract class for use with JUnit4. Users must define the following for this +abstract class:</p> + +<ul> + <li>A <code class="highlighter-rouge">SortedMap</code> of input data (<code class="highlighter-rouge">Key</code>-<code class="highlighter-rouge">Value</code> pairs)</li> + <li>A <code class="highlighter-rouge">Range</code> to use in tests</li> + <li>A <code class="highlighter-rouge">Map</code> of options (<code class="highlighter-rouge">String</code> to <code class="highlighter-rouge">String</code> pairs)</li> + <li>A <code class="highlighter-rouge">SortedMap</code> of output data (<code class="highlighter-rouge">Key</code>-<code class="highlighter-rouge">Value</code> pairs)</li> + <li>A list of <code class="highlighter-rouge">IteratorTestCase</code>s (these can be automatically discovered)</li> +</ul> + +<p>The majority of effort a user must make is in creating the input dataset and the expected +output dataset for the iterator being tested.</p> + +<h3 id="normal-test-outline">Normal Test Outline</h3> + +<p>Most iterator tests will follow the given outline:</p> + +<div class="language-java highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">java.util.SortedMap</span><span class="o">;</span> + +<span class="kn">import</span> <span class="nn">org.apache.accumulo.core.data.Key</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.core.data.Range</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.core.data.Value</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.iteratortest.IteratorTestCaseFinder</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.iteratortest.IteratorTestInput</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.iteratortest.IteratorTestOutput</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.iteratortest.junit4.BaseJUnit4IteratorTest</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.apache.accumulo.iteratortest.testcases.IteratorTestCase</span><span class="o">;</span> +<span class="kn">import</span> <span class="nn">org.junit.runners.Parameterized.Parameters</span><span class="o">;</span> + +<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyIteratorTest</span> <span class="kd">extends</span> <span class="n">BaseJUnit4IteratorTest</span> <span class="o">{</span> + + <span class="nd">@Parameters</span> + <span class="kd">public</span> <span class="kd">static</span> <span class="n">Object</span><span class="o">[][]</span> <span class="nf">parameters</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">final</span> <span class="n">IteratorTestInput</span> <span class="n">input</span> <span class="o">=</span> <span class="n">createIteratorInput</span><span class="o">();</span> + <span class="kd">final</span> <span class="n">IteratorTestOutput</span> <span class="n">output</span> <span class="o">=</span> <span class="n">createIteratorOutput</span><span class="o">();</span> + <span class="kd">final</span> <span class="n">List</span><span class="o"><</span><span class="n">IteratorTestCase</span><span class="o">></span> <span class="n">testCases</span> <span class="o">=</span> <span class="n">IteratorTestCaseFinder</span><span class="o">.</span><span class="na">findAllTestCases</span><span class="o">();</span> + <span class="k">return</span> <span class="n">BaseJUnit4IteratorTest</span><span class="o">.</span><span class="na">createParameters</span><span class="o">(</span><span class="n">input</span><span class="o">,</span> <span class="n">output</span><span class="o">,</span> <span class="n">tests</span><span class="o">);</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">SortedMap</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">></span> <span class="n">INPUT_DATA</span> <span class="o">=</span> <span class="n">createInputData</span><span class="o">();</span> + <span class="kd">private</span> <span class="kd">static</span> <span class="n">SortedMap</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">></span> <span class="n">OUTPUT_DATA</span> <span class="o">=</span> <span class="n">createOutputData</span><span class="o">();</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">SortedMap</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">></span> <span class="nf">createInputData</span><span class="o">()</span> <span class="o">{</span> + <span class="c1">// TODO -- implement this method</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">SortedMap</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span><span class="n">Value</span><span class="o">></span> <span class="nf">createOutputData</span><span class="o">()</span> <span class="o">{</span> + <span class="c1">// TODO -- implement this method</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">IteratorTestInput</span> <span class="nf">createIteratorInput</span><span class="o">()</span> <span class="o">{</span> + <span class="kd">final</span> <span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span><span class="n">String</span><span class="o">></span> <span class="n">options</span> <span class="o">=</span> <span class="n">createIteratorOptions</span><span class="o">();</span> + <span class="kd">final</span> <span class="n">Range</span> <span class="n">range</span> <span class="o">=</span> <span class="n">createRange</span><span class="o">();</span> + <span class="k">return</span> <span class="k">new</span> <span class="nf">IteratorTestInput</span><span class="o">(</span><span class="n">MyIterator</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">options</span><span class="o">,</span> <span class="n">range</span><span class="o">,</span> <span class="n">INPUT_DATA</span><span class="o">);</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span><span class="n">String</span><span class="o">></span> <span class="nf">createIteratorOptions</span><span class="o">()</span> <span class="o">{</span> + <span class="c1">// TODO -- implement this method</span> + <span class="c1">// Tip: Use INPUT_DATA if helpful in generating output</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">Range</span> <span class="nf">createRange</span><span class="o">()</span> <span class="o">{</span> + <span class="c1">// TODO -- implement this method</span> + <span class="o">}</span> + + <span class="kd">private</span> <span class="kd">static</span> <span class="n">IteratorTestOutput</span> <span class="nf">createIteratorOutput</span><span class="o">()</span> <span class="o">{</span> + <span class="k">return</span> <span class="k">new</span> <span class="nf">IteratorTestOutput</span><span class="o">(</span><span class="n">OUTPUT_DATA</span><span class="o">);</span> + <span class="o">}</span> +<span class="o">}</span> +</code></pre> +</div> + +<h3 id="limitations">Limitations</h3> + +<p>While the provided <code class="highlighter-rouge">IteratorTestCase</code>s should exercise common edge-cases in user iterators, +there are still many limitations to the existing test harness. Some of them are:</p> + +<ul> + <li>Can only specify a single iterator, not many (a âstackâ)</li> + <li>No control over provided IteratorEnvironment for tests</li> + <li>Exercising delete keys (especially with major compactions that do not include all files)</li> +</ul> + +<p>These are left as future improvements to the harness.</p> + <h2 id="mock-accumulo">Mock Accumulo</h2> <p>Mock Accumulo supplies mock implementations for much of the client API. It presently http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/9ebc5f9a/docs/unreleased/development/high_speed_ingest.html ---------------------------------------------------------------------- diff --git a/docs/unreleased/development/high_speed_ingest.html b/docs/unreleased/development/high_speed_ingest.html index c81e311..7cc143b 100644 --- a/docs/unreleased/development/high_speed_ingest.html +++ b/docs/unreleased/development/high_speed_ingest.html @@ -200,9 +200,11 @@ <div class="panel-body"> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_design">Iterator Design</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterators">Iterators</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/iterator_testing">Iterator Testing</a></div> + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/mapreduce">MapReduce</a></div> + + <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/proxy">Proxy</a></div> <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/development_tools">Development Tools</a></div> @@ -214,8 +216,6 @@ <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/high_speed_ingest">High-Speed Ingest</a></div> - <div class="row doc-sidebar-link"><a href="/docs/unreleased/development/analytics">Analytics</a></div> - </div> </div>
![https://www.apache.org/images/SupportApache-small.png"](https://www.apache.org/images/SupportApache-small.png"){kind=link}