Repository: accumulo-website Updated Branches: refs/heads/asf-site 1e7c974f1 -> 41adefb69 refs/heads/master b52d466fd -> a33b3ed9b
Improvemnts to documentation and verifying release page Project: http://git-wip-us.apache.org/repos/asf/accumulo-website/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo-website/commit/a33b3ed9 Tree: http://git-wip-us.apache.org/repos/asf/accumulo-website/tree/a33b3ed9 Diff: http://git-wip-us.apache.org/repos/asf/accumulo-website/diff/a33b3ed9 Branch: refs/heads/master Commit: a33b3ed9b86ec41b862de5f9368e2e75421bc4a8 Parents: b52d466 Author: Mike Walch <[email protected]> Authored: Wed May 31 14:31:35 2017 -0400 Committer: Mike Walch <[email protected]> Committed: Wed May 31 14:32:08 2017 -0400 ---------------------------------------------------------------------- _config.yml | 4 + .../development/development_tools.md | 164 ++++++++----------- .../development/high_speed_ingest.md | 21 ++- _docs-unreleased/development/security.md | 67 +++++--- _docs-unreleased/getting-started/shell.md | 70 ++------ contributor/verifying-release.md | 12 +- 6 files changed, 146 insertions(+), 192 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/_config.yml ---------------------------------------------------------------------- diff --git a/_config.yml b/_config.yml index bae061c..d338ab0 100644 --- a/_config.yml +++ b/_config.yml @@ -59,7 +59,9 @@ defaults: layout: "docs-unreleased" title_prefix: "Accumulo Documentation - " version: "unreleased" + latest_release: "1.8.1" docs_baseurl: "/docs/unreleased" + javadoc_base: "https://static.javadoc.io/org.apache.accumulo"; javadoc_core: "https://static.javadoc.io/org.apache.accumulo/accumulo-core/1.8.1"; skiph1fortitle: "true" # - @@ -70,7 +72,9 @@ defaults: # layout: "docs-2.0" # title_prefix: "Accumulo Documentation - " # version: "2.0" +# latest_release: "2.0.0" # docs_baseurl: "/docs/2.0" +# javadoc_base: "https://static.javadoc.io/org.apache.accumulo"; # javadoc_core: "https://static.javadoc.io/org.apache.accumulo/accumulo-core/2.0.0"; # skiph1fortitle: "true" http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/_docs-unreleased/development/development_tools.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/development/development_tools.md b/_docs-unreleased/development/development_tools.md index f9768f6..54cceed 100644 --- a/_docs-unreleased/development/development_tools.md +++ b/_docs-unreleased/development/development_tools.md @@ -4,10 +4,49 @@ category: development order: 4 --- -Normally, Accumulo consists of lots of moving parts. Even a stand-alone version of -Accumulo requires Hadoop, Zookeeper, the Accumulo master, a tablet server, etc. If -you want to write a unit test that uses Accumulo, you need a lot of infrastructure -in place before your test can run. +Accumulo has several tools that can help developers test their code. + +## MiniAccumuloCluster + +[MiniAccumuloCluster] is a standalone instance of Apache Accumulo for testing. It will +create Zookeeper and Accumulo processes that write all of their data to a single local +directory. [MiniAccumuloCluster] makes it easy to code agaist a real Accumulo instance. +Developers can write realistic-to-end integration tests that mimic the use of a normal +Accumulo instance. + +[MiniAccumuloCluster] is published in a separate jar that should be added to your pom.xml +as a test dependency: + +```xml +<dependency> + <groupId>org.apache.accumulo</groupId> + <artifactId>accumulo-minicluster</artifactId> + <version>${accumulo.version}</version> + <scope>test</scope> +</dependency> +``` + +To start it up, you will need to supply an empty directory and a root password as arguments: + +```java +File tempDirectory = // JUnit and Guava supply mechanisms for creating temp directories +MiniAccumuloCluster accumulo = new MiniAccumuloCluster(tempDirectory, "password"); +accumulo.start(); +``` + +Once we have our mini cluster running, we will want to interact with the Accumulo client API: + +```java +Instance instance = new ZooKeeperInstance(accumulo.getInstanceName(), accumulo.getZooKeepers()); +Connector conn = instance.getConnector("root", new PasswordToken("password")); +``` + +Upon completion of our development code, we will want to shutdown our MiniAccumuloCluster: + +```java +accumulo.stop(); +// delete your temporary folder +``` ## Iterator Test Harness @@ -20,14 +59,26 @@ for all Accumulo Iterators to leverage to identify common pitfalls in user-creat ### Framework Use -The harness provides an abstract class for use with JUnit4. Users must define the following for this -abstract class: +The Iterator Test Harness is published in a separate jar that should be added to your pom.xml as +a test dependency: + +```xml +<dependency> + <groupId>org.apache.accumulo</groupId> + <artifactId>accumulo-iterator-test-harness</artifactId> + <version>${accumulo.version}</version> + <scope>test</scope> +</dependency> +``` + +To use the Iterator test harness, create a class that extends the [BaseJUnit4IteratorTest] class +and defines the following: * A `SortedMap` of input data (`Key`-`Value` pairs) - * A `Range` to use in tests + * A [Range] to use in tests * A `Map` of options (`String` to `String` pairs) * A `SortedMap` of output data (`Key`-`Value` pairs) - * A list of `IteratorTestCase`s (these can be automatically discovered) + * A list of [IteratorTestCase]s (these can be automatically discovered) The majority of effort a user must make is in creating the input dataset and the expected output dataset for the iterator being tested. @@ -94,7 +145,7 @@ public class MyIteratorTest extends BaseJUnit4IteratorTest { ### Limitations -While the provided `IteratorTestCase`s should exercise common edge-cases in user iterators, +While the provided [IteratorTestCase]s should exercise common edge-cases in user iterators, there are still many limitations to the existing test harness. Some of them are: * Can only specify a single iterator, not many (a "stack") @@ -103,94 +154,7 @@ there are still many limitations to the existing test harness. Some of them are: These are left as future improvements to the harness. -## Mock Accumulo - -Mock Accumulo supplies mock implementations for much of the client API. It presently -does not enforce users, logins, permissions, etc. It does support Iterators and Combiners. -Note that MockAccumulo holds all data in memory, and will not retain any data or -settings between runs. - -While normal interaction with the Accumulo client looks like this: - -```java -Instance instance = new ZooKeeperInstance(...); -Connector conn = instance.getConnector(user, passwordToken); -``` - -To interact with the MockAccumulo, just replace the ZooKeeperInstance with MockInstance: - -```java -Instance instance = new MockInstance(); -``` - -In fact, you can use the `--fake` option to the Accumulo shell and interact with -MockAccumulo: - -``` -$ accumulo shell --fake -u root -p '' - -Shell - Apache Accumulo Interactive Shell -- -- version: 2.x.x -- instance name: fake -- instance id: mock-instance-id -- -- type 'help' for a list of available commands -- - -root@fake> createtable test - -root@fake test> insert row1 cf cq value -root@fake test> insert row2 cf cq value2 -root@fake test> insert row3 cf cq value3 - -root@fake test> scan -row1 cf:cq [] value -row2 cf:cq [] value2 -row3 cf:cq [] value3 - -root@fake test> scan -b row2 -e row2 -row2 cf:cq [] value2 - -root@fake test> -``` - -When testing Map Reduce jobs, you can also set the Mock Accumulo on the AccumuloInputFormat -and AccumuloOutputFormat classes: - -```java -// ... set up job configuration -AccumuloInputFormat.setMockInstance(job, "mockInstance"); -AccumuloOutputFormat.setMockInstance(job, "mockInstance"); -``` - -## Mini Accumulo Cluster - -While the Mock Accumulo provides a lightweight implementation of the client API for unit -testing, it is often necessary to write more realistic end-to-end integration tests that -take advantage of the entire ecosystem. The Mini Accumulo Cluster makes this possible by -configuring and starting Zookeeper, initializing Accumulo, and starting the Master as well -as some Tablet Servers. It runs against the local filesystem instead of having to start -up HDFS. - -To start it up, you will need to supply an empty directory and a root password as arguments: - -```java -File tempDirectory = // JUnit and Guava supply mechanisms for creating temp directories -MiniAccumuloCluster accumulo = new MiniAccumuloCluster(tempDirectory, "password"); -accumulo.start(); -``` - -Once we have our mini cluster running, we will want to interact with the Accumulo client API: - -```java -Instance instance = new ZooKeeperInstance(accumulo.getInstanceName(), accumulo.getZooKeepers()); -Connector conn = instance.getConnector("root", new PasswordToken("password")); -``` - -Upon completion of our development code, we will want to shutdown our MiniAccumuloCluster: - -```java -accumulo.stop(); -// delete your temporary folder -``` +[Range]: {{ page.javadoc_core }}/org/apache/accumulo/core/data/Range.html +[IteratorTestCase]: {{ page.javadoc_base }}/accumulo-iterator-test-harness/{{ page.latest_release }}/org/apache/accumulo/iteratortest/testcases/IteratorTestCase.html +[BaseJUnit4IteratorTest]: {{ page.javadoc_base }}/accumulo-iterator-test-harness/{{ page.latest_release }}/org/apache/accumulo/iteratortest/junit4/BaseJUnit4IteratorTest.html +[MiniAccumuloCluster]: {{ page.javadoc_base }}/accumulo-minicluster/{{ page.latest_release }}/org/apache/accumulo/minicluster/MiniAccumuloCluster.html http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/_docs-unreleased/development/high_speed_ingest.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/development/high_speed_ingest.md b/_docs-unreleased/development/high_speed_ingest.md index f52f501..dccafc4 100644 --- a/_docs-unreleased/development/high_speed_ingest.md +++ b/_docs-unreleased/development/high_speed_ingest.md @@ -32,7 +32,7 @@ rate is still subject to the number of machines running ingest clients, and the distribution of rowIDs across the table. The aggregation ingest rate will be suboptimal if there are many inserts into a small number of rowIDs. -## Multiple Ingester Clients +## Multiple Ingest Clients Accumulo is capable of scaling to very high rates of ingest, which is dependent upon not just the number of TabletServers in operation but also the number of ingest @@ -71,7 +71,7 @@ into multiple tablets. zy zz -Run the MapReduce job, using the AccumuloFileOutputFormat to create the files to +Run the MapReduce job, using the [AccumuloFileOutputFormat] to create the files to be introduced to Accumulo. Once this is complete, the files can be added to Accumulo via the shell: @@ -81,8 +81,7 @@ Note that the paths referenced are directories within the same HDFS instance ove which Accumulo is running. Accumulo places any files that failed to be added to the second directory specified. -See the [bulk ingest example](https://github.com/apache/accumulo-examples/blob/master/docs/bulkIngest.md) -for a complete example. +See the [bulk ingest example][bulk-example] for a complete example. ## Logical Time for Bulk Ingest @@ -107,7 +106,13 @@ import file. ## MapReduce Ingest It is possible to efficiently write many mutations to Accumulo in parallel via a -MapReduce job. In this scenario the MapReduce is written to process data that lives -in HDFS and write mutations to Accumulo using the AccumuloOutputFormat. See -the MapReduce section under Analytics for details. The [MapReduce example](https://github.com/apache/accumulo-examples/blob/master/docs/mapred.md) -is also a good reference for example code. +MapReduce job. Typically, a MapReduce job will process data that lives in HDFS +and write mutations to Accumulo using [AccumuloOutputFormat]. For more information +on how use to use MapReduce with Accumulo, see the [MapReduce documentation][mapred-docs] +and the [MapReduce example code][mapred-code]. + +[bulk-example]: https://github.com/apache/accumulo-examples/blob/master/docs/bulkIngest.md +[AccumuloOutputFormat]: {{ page.javadoc_core }}/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.html +[AccumuloFileOutputFormat]: {{ page.javadoc_core }}/org/apache/accumulo/core/client/mapred/AccumuloFileOutputFormat.html +[mapred-docs]: {{ page.docs_baseurl }}/development/mapreduce +[mapred-code]: https://github.com/apache/accumulo-examples/blob/master/docs/mapred.md http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/_docs-unreleased/development/security.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/development/security.md b/_docs-unreleased/development/security.md index 0671d50..2e65d22 100644 --- a/_docs-unreleased/development/security.md +++ b/_docs-unreleased/development/security.md @@ -5,7 +5,7 @@ order: 7 --- Accumulo extends the BigTable data model to implement a security mechanism -known as cell-level security. Every key-value pair has its own security label, stored +known as cell-level security. Every [Key]-[Value] pair has its own security label, stored under the column visibility element of the key, which is used to determine whether a given user meets the security requirements to read the value. This enables data of various security levels to be stored within the same row, and users of varying @@ -14,7 +14,7 @@ degrees of access to query the same table, while preserving data confidentiality ## Security Label Expressions When mutations are applied, users can specify a security label for each value. This is -done as the Mutation is created by passing a ColumnVisibility object to the put() +done as the [Mutation] is created by passing a [ColumnVisibility] object to the put() method: ```java @@ -69,36 +69,37 @@ admin|audit When both `|` and `&` operators are used, parentheses must be used to specify precedence of the operators. -## Authorization +## Authorizations When clients attempt to read data from Accumulo, any security labels present are -examined against the set of authorizations passed by the client code when the -Scanner or BatchScanner are created. If the authorizations are determined to be +examined against an [Authorizations] object passed by the client code when the +[Scanner] or [BatchScanner] are created. If the Authorizations are determined to be insufficient to satisfy the security label, the value is suppressed from the set of results sent back to the client. -Authorizations are specified as a comma-separated list of tokens the user possesses: +[Authorizations] are specified as a comma-separated list of tokens the user possesses: ```java // user possesses both admin and system level access -Authorization auths = new Authorization("admin","system"); +Authorizations auths = new Authorizations("admin","system"); Scanner s = connector.createScanner("table", auths); ``` ## User Authorizations -Each Accumulo user has a set of associated security labels. To manipulate -these in the shell while using the default authorizor, use the setuaths and getauths commands. -These may also be modified for the default authorizor using the java security operations API. +Each Accumulo user has a set of associated security labels. To manipulate these in +the [Accumulo shell][shell], use the `setuaths` and `getauths` commands. They can be +retrieved and modified in Java using `getUserAuthorizations` and `changeUserAuthorizations` +methods of [SecurityOperations]. -When a user creates a scanner a set of Authorizations is passed. If the -authorizations passed to the scanner are not a subset of the users -authorizations, then an exception will be thrown. +When a user creates a [Scanner] or [BatchScanner] a set of [Authorizations] is passed. +If the Authorizations passed to the scanner are not a subset of the user's Authorizations, +then an exception will be thrown. To prevent users from writing data they can not read, add the visibility -constraint to a table. Use the -evc option in the createtable shell command to -enable this constraint. For existing tables use the following shell command to +constraint to a table. Use the -evc option in the `createtable` shell command to +enable this constraint. For existing tables, use the `config` command to enable the visibility constraint. Ensure the constraint number does not conflict with any existing constraints. @@ -110,18 +111,19 @@ disable the bulk import permission. ## Pluggable Security -New in 1.5 of Accumulo is a pluggable security mechanism. It can be broken into three actions -- -authentication, authorization, and permission handling. By default all of these are handled in -Zookeeper, which is how things were handled in Accumulo 1.4 and before. It is worth noting at this -point, that it is a new feature in 1.5 and may be adjusted in future releases without the standard -deprecation cycle. +Accumulo has a pluggable security mechanism. It can be broken into three actions: authentication, +authorization, and permission handling. -Authentication simply handles the ability for a user to verify their integrity. A combination of -principal and authentication token are used to verify a user is who they say they are. An -authentication token should be constructed, either directly through its constructor, but it is -advised to use the `init(Property)` method to populate an authentication token. It is expected that a -user knows what the appropriate token to use for their system is. The default token is -`PasswordToken`. +Authentication verifies the identity of a user. In Accumulo, authentication occurs when +the `getConnector` method of [Instance] is called with a principal (i.e username) +and an [AuthenticationToken] which is an interface with multiple implementations. The most +common implementation is [PasswordToken] which is the default authenticaton method for Accumulo +out of the box. + +```java +Instance instance = new ZooKeeperInstance("myinstance", "zookeeper1,zookeeper2"); +Connector conn = instance.getConnector("user", new PasswordToken("passwd")); +``` Once a user is authenticated by the Authenticator, the user has access to the other actions within Accumulo. All actions in Accumulo are ACLed, and this ACL check is handled by the Permission @@ -167,3 +169,16 @@ cached within the query layer and presented to Accumulo through the Authorizations mechanism. Typically, the query services layer sits between Accumulo and user workstations. + +[shell]: {{ page.docs_baseurl }}/getting-started/shell +[Key]: {{ page.javadoc_core }}/org/apache/accumulo/core/data/Key.html +[Value]: {{ page.javadoc_core }}/org/apache/accumulo/core/data/Value.html +[Mutation]: {{ page.javadoc_core }}/org/apache/accumulo/core/data/Mutation.html +[ColumnVisibility]: {{ page.javadoc_core }}/org/apache/accumulo/core/security/ColumnVisibility.html +[Scanner]: {{ page.javadoc_core }}/org/apache/accumulo/core/client/Scanner.html +[BatchScanner]: {{ page.javadoc_core}}/org/apache/accumulo/core/client/BatchScanner.html +[Authorizations]: {{ page.javadoc_core}}/org/apache/accumulo/core/security/Authorizations.html +[SecurityOperations]: {{ page.javadoc_core}}/org/apache/accumulo/core/client/admin/SecurityOperations.html +[Instance]: {{ page.javadoc_core}}/org/apache/accumulo/core/client/Instance.html +[AuthenticationToken]: {{ page.javadoc_core}}/org/apache/accumulo/core/client/security/tokens/AuthenticationToken.html +[PasswordToken]: {{ page.javadoc_core}}/org/apache/accumulo/core/client/security/tokens/PasswordToken.html http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/_docs-unreleased/getting-started/shell.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/getting-started/shell.md b/_docs-unreleased/getting-started/shell.md index 7fa56d9..8007a28 100644 --- a/_docs-unreleased/getting-started/shell.md +++ b/_docs-unreleased/getting-started/shell.md @@ -27,36 +27,27 @@ and then display the following prompt: ## Basic Administration -The Accumulo shell can be used to create and delete tables, as well as to configure -table and instance specific options. +The `tables` command will list all existings tables. -``` -root@myinstance> tables -accumulo.metadata -accumulo.root - -root@myinstance> createtable mytable - -root@myinstance mytable> + root@myinstance> tables + accumulo.metadata + accumulo.root -root@myinstance mytable> tables -accumulo.metadata -accumulo.root -mytable +The `createtable` command creates a new table. -root@myinstance mytable> createtable testtable + root@myinstance> createtable mytable + root@myinstance mytable> tables + accumulo.metadata + accumulo.root + mytable -root@myinstance testtable> +The `deletetable` command deletes a table. -root@myinstance testtable> deletetable testtable -deletetable { testtable } (yes|no)? yes -Table: [testtable] has been deleted. + root@myinstance testtable> deletetable testtable + deletetable { testtable } (yes|no)? yes + Table: [testtable] has been deleted. -root@myinstance> -``` - -The Shell can also be used to insert updates and scan tables. This is useful for -inspecting tables. +The shell can be used to insert updates and scan tables. This is useful for inspecting tables. root@myinstance mytable> scan @@ -71,13 +62,13 @@ You can use the `-st` option to scan to see the timestamp for the cell, too. ## Table Maintenance -The *compact* command instructs Accumulo to schedule a compaction of the table during which +The `compact` command instructs Accumulo to schedule a compaction of the table during which files are consolidated and deleted entries are removed. root@myinstance mytable> compact -t mytable 07 16:13:53,201 [shell.Shell] INFO : Compaction of table mytable started for given range -The *flush* command instructs Accumulo to write all entries currently in memory for a given table +The `flush` command instructs Accumulo to write all entries currently in memory for a given table to disk. root@myinstance mytable> flush -t mytable @@ -116,30 +107,3 @@ Enter current password for 'root': ********* root@myinstance bobstable> revoke System.CREATE_TABLE -s -u bob ``` - -## JSR-223 Support in the Shell - -The script command can be used to invoke programs written in languages supported by installed JSR-223 -engines. You can get a list of installed engines with the -l argument. Below is an example of the output -of the command when running the Shell with Java 7. - - root@fake> script -l - Engine Alias: ECMAScript - Engine Alias: JavaScript - Engine Alias: ecmascript - Engine Alias: javascript - Engine Alias: js - Engine Alias: rhino - Language: ECMAScript (1.8) - Script Engine: Mozilla Rhino (1.7 release 3 PRERELEASE) - ScriptEngineFactory Info - -A list of compatible languages can be found on [this page](https://en.wikipedia.org/wiki/List_of_JVM_languages). The -rhino javascript engine is provided with the JVM. Typically putting a jar on the classpath is all that is -needed to install a new engine. - - When writing scripts to run in the shell, you will have a variable called connection already available -to you. This variable is a reference to an Accumulo Connector object, the same connection that the Shell -is using to communicate with the Accumulo servers. At this point you can use any of the public API methods -within your script. Reference the script command help to see all of the execution options. Script and script -invocation examples can be found in ACCUMULO-1399. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/a33b3ed9/contributor/verifying-release.md ---------------------------------------------------------------------- diff --git a/contributor/verifying-release.md b/contributor/verifying-release.md index f9a2217..789e334 100644 --- a/contributor/verifying-release.md +++ b/contributor/verifying-release.md @@ -26,11 +26,11 @@ Below are some suggested tests that can be run (feel free to run your own custom * Build the [Accumulo Examples][examples] repo using the release candidate by updating the `accumulo.version` property in the `pom.xml` and using the staging repo. Also, run the unit/integration tests using `mvn verify`. -* Run Accumulo's distributed tests (i.e RandomWalk, ContinuousIngest, etc). Information on these tests can - be found in their respective directories, `test/system/randomwalk` and `test/system/continuous`, which - include instructions on how to run the tests. These tests are intended to be run for days on end while - injecting faults into the system. These are the tests that truly verify the correctness of Accumulo on - real systems. +* Run Accumulo's distributed tests (i.e. random walk, continous ingest). These tests are intended to be run for days + on end while injecting faults into the system. These are the tests that truly verify the correctness of Accumulo on + real systems. Starting with 2.0, these tests are run using the [Accumulo Testing repo][at]. See the [README.md][at-readme] + for more information. Before 2.0, these tests are found in Accumulo tarball at `test/system/randomwalk` and + `test/system/continuous` which include instructions on how to run the tests. ## Project testing goals @@ -128,3 +128,5 @@ own NOTICE file. The contents of the Apache Thrift NOTICE file should be include [5]: https://www.apache.org/legal [examples]: https://github.com/apache/accumulo-examples [versioning]: {{ site.baseurl }}/contributor/versioning +[at]: https://github.com/apache/accumulo-testing +[at-readme]: https://github.com/apache/accumulo-testing/blob/master/README.md
