Repository: hadoop Updated Branches: refs/heads/trunk fdf5a3fd6 -> 8af845358
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/Settings.md ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/content/Settings.md b/hadoop-ozone/docs/content/Settings.md new file mode 100644 index 0000000..bfa644e --- /dev/null +++ b/hadoop-ozone/docs/content/Settings.md @@ -0,0 +1,139 @@ +--- +title: Configuration +weight: 1 +menu: + main: + parent: Starting + weight: 2 +--- +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + + + + +If you are feeling adventurous, you can setup ozone in a real cluster. +Setting up a real cluster requires us to understand the components of Ozone. +Ozone is designed to work concurrently with HDFS. However, Ozone is also +capable of running independently. The components of ozone are the same in both approaches. + +## Ozone Components + +1. Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations. +2. Storage Container Manager - Acts as the block manager. Ozone Manager +requests blocks from SCM, to which clients can write data. +3. Datanodes - Ozone data node code runs inside the HDFS datanode or in the independent deployment case runs an ozone datanode daemon. + + + + +## Setting up an Ozone only cluster + +* Please untar the ozone-0.2.1-SNAPSHOT to the directory where you are going +to run Ozone from. We need Ozone jars on all machines in the cluster. So you +need to do this on all machines in the cluster. + +* Ozone relies on a configuration file called ```ozone-site.xml```. To +generate a template that you can replace with proper values, please run the +following command. This will generate a template called ```ozone-site.xml``` at +the specified path (directory). + +{{< highlight bash >}} +ozone genconf -output <path> +{{< /highlight >}} + +Let us look at the settings inside the generated file (ozone-site.xml) and +how they control ozone. Once the right values are defined, this file +needs to be copied to ```ozone directory/etc/Hadoop```. + + +* **ozone.enabled** This is the most critical setting for ozone. +Ozone is a work in progress and users have to enable this service explicitly. +By default, Ozone is disabled. Setting this flag to `true` enables ozone in the +HDFS or Ozone cluster. + +Here is an example, + +{{< highlight xml >}} + <property> + <name>ozone.enabled</name> + <value>True</value> + </property> +{{< /highlight >}} + +* **ozone.metadata.dirs** Allows Administrators to specify where the + metadata must reside. Usually you pick your fastest disk (SSD if + you have them on your nodes). OzoneManager, SCM and datanode will write the + metadata to this path. This is a required setting, if this is missing Ozone + will fail to come up. + + Here is an example, + +{{< highlight xml >}} + <property> + <name>ozone.metadata.dirs</name> + <value>/data/disk1/meta</value> + </property> +{{< /highlight >}} + +* **ozone.scm.names** Storage container manager(SCM) is a distributed block + service which is used by ozone. This property allows data nodes to discover + SCM's address. Data nodes send heartbeat to SCM. + Until HA feature is complete, we configure ozone.scm.names to be a + single machine. + + Here is an example, + + {{< highlight xml >}} + <property> + <name>ozone.scm.names</name> + <value>scm.hadoop.apache.org</value> + </property> + {{< /highlight >}} + + * **ozone.scm.datanode.id** Data nodes generate a Unique ID called Datanode + ID. This identity is written to the file specified by this path. *Data nodes + will create this path if it doesn't exist already.* + +Here is an example, +{{< highlight xml >}} + <property> + <name>ozone.scm.datanode.id</name> + <value>/data/disk1/meta/node/datanode.id</value> + </property> +{{< /highlight >}} + +* **ozone.om.address** OM server address. This is used by OzoneClient and +Ozone File System. + +Here is an example, +{{< highlight xml >}} + <property> + <name>ozone.om.address</name> + <value>ozonemanager.hadoop.apache.org</value> + </property> +{{< /highlight >}} + + +### Ozone Settings Summary + +| Setting | Value | Comment | +|--------------------------------|------------------------------|------------------------------------------------------------------| +| ozone.enabled | true | This enables SCM and containers in HDFS cluster. | +| ozone.metadata.dirs | file path | The metadata will be stored here. | +| ozone.scm.names | SCM server name | Hostname:port or IP:port address of SCM. | +| ozone.scm.block.client.address | SCM server name and port | Used by services like OM | +| ozone.scm.client.address | SCM server name and port | Used by client-side | +| ozone.scm.datanode.address | SCM server name and port | Used by datanode to talk to SCM | +| ozone.om.address | OM server name | Used by Ozone handler and Ozone file system. | http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/VolumeCommands.md ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/content/VolumeCommands.md b/hadoop-ozone/docs/content/VolumeCommands.md new file mode 100644 index 0000000..45811c5 --- /dev/null +++ b/hadoop-ozone/docs/content/VolumeCommands.md @@ -0,0 +1,100 @@ +--- +title: Volume Commands +menu: + main: + parent: Client + weight: 2 +--- + +Volume commands generally need administrator privileges. The ozone shell supports the following volume commands. + + * [create](#create) + * [delete](#delete) + * [info](#info) + * [list](#list) + * [update](#update) + +### Create + +The volume create command allows an administrator to create a volume and +assign it to a user. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | +| -u, --user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | +| Uri | The name of the volume. | + +{{< highlight bash >}} +ozone sh volume create --quota=1TB --user=bilbo /hive +{{< /highlight >}} + +The above command will create a volume called _hive_ on the ozone cluster. This +volume has a quota of 1TB, and the owner is _bilbo_. + +### Delete + +The volume delete commands allows an administrator to delete a volume. If the +volume is not empty then this command will fail. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the volume. + +{{< highlight bash >}} +ozone sh volume delete /hive +{{< /highlight >}} + +The above command will delete the volume hive, if the volume has no buckets +inside it. + +### Info + +The volume info commands returns the information about the volume including +quota and owner information. +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| Uri | The name of the volume. + +{{< highlight bash >}} +ozone sh volume info /hive +{{< /highlight >}} + +The above command will print out the information about hive volume. + +### List + +The volume list command will list the volumes owned by a user. + +{{< highlight bash >}} +ozone sh volume list --user hadoop +{{< /highlight >}} + +The above command will print out all the volumes owned by the user hadoop. + +### Update + +The volume update command allows changing of owner and quota on a given volume. + +***Params:*** + +| Arguments | Comment | +|--------------------------------|-----------------------------------------| +| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | +| -u, --user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | +| Uri | The name of the volume. | + +{{< highlight bash >}} +ozone sh volume update --quota=10TB /hive +{{< /highlight >}} + +The above command updates the volume quota to 10TB. + +You can try out these commands from the docker instance of the [Alpha +Cluster](runningviadocker.html). http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/content/_index.md ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/content/_index.md b/hadoop-ozone/docs/content/_index.md index 383b2e0..19340bf 100644 --- a/hadoop-ozone/docs/content/_index.md +++ b/hadoop-ozone/docs/content/_index.md @@ -17,86 +17,20 @@ weight: -10 limitations under the License. See accompanying LICENSE file. --> -â¨Ozone is an Object store for Apache Hadoop. It aims to scale to billions of -keys. â¨The following is a high-level overview of the core components of Ozone.â¨â¨ +# Apache Hadoop Ozone - â¨â¨ +Ozone is a scalable, distributed object store for Hadoop. Applications like +Apache Spark, Hive and YARN, can run against Ozone without any +modifications. Ozone comes with a [Java client library]({{< ref "JavaApi.md" +>}}) and a [command line interface] ({{< ref "CommandShell.md#shell" >}}) which makes it easy to use Ozone. This client library supports both RPC and REST protocols. -The main elements of Ozone areâ¨: +Ozone consists of volumes, buckets, and Keys. -## Clients +* Volumes are similar to user accounts. Only administrators can create or delete volumes. +* Buckets are similar to directories. A bucket can contain any number of keys, but buckets cannot contain other buckets. +* Keys are similar to files. A bucket can contain any number of keys. -Ozone ships with a set of ready-made clients. They are â¨Ozone CLI and Freon.⨠- * [Ozone CLI](./OzoneCommandShell.html) is the command line interface like 'hdfs' command.⨠- * Freon is a load generation tool for Ozone.⨠+<a href="{{< ref "RunningViaDocker.md" >}}"><button class="btn btn-danger btn-lg">Getting started</button></a> -## REST Handler - -Ozone provides both an RPC (Remote Procedure Call) as well as a REST -(Representational State Transfer) style interface. This allows clients to be -written in many languages quickly. Ozone strives to maintain a similar -interface between REST and RPC. The Rest handler offers the REST protocol -services of Ozone. - -For most purposes, a client can make one line change to switch from REST to -RPC or vice versa. ⨠- -## Ozone File System - -Ozone file system (TODO: Add documentation) is a Hadoop compatible file system. -This is the important user-visible component of ozone. -This allows Hadoop services and applications like Hive/Spark to run against -Ozone without any change. - -## Ozone Client - -This is like DFSClient in HDFS. This acts as the standard client to talk to -Ozone. All other components that we have discussed so far rely on Ozone client -(TODO: Add Ozone client documentation).⨠- -## Ozone Manager - -Ozone Manager (OM) takes care of the Ozone's namespace. -All ozone entities like volumes, buckets and keys are managed by OM -(TODO: Add OM documentation). In short, OM is the metadata manager for Ozone. -OM talks to blockManager(SCM) to get blocks and passes it on to the Ozone -client. Ozone client writes data to these blocks. -OM will eventually be replicated via Apache Ratis for High Availability.⨠- -## Storage Container Manager -Storage Container Manager (SCM) is the block and cluster manager for Ozone. -SCM along with data nodes offer a service called 'containers'. -A container is a group unrelated of blocks that are managed together -as a single entity. - -SCM offers the following abstractions.â¨â¨ - - - -### Blocks - -Blocks are like blocks in HDFS. They are replicated store of data. - -### Containers - -A collection of blocks replicated and managed together. - -### Pipelines - -SCM allows each container to choose its method of replication. -For example, a container might decide that it needs only one copy of a block -and might choose a stand-alone pipeline. Another container might want to have -a very high level of reliability and pick a RATIS based pipeline. In other -words, SCM allows different kinds of replication strategies to co-exist. - -### Pools - -A group of data nodes is called a pool. For scaling purposes, -we define a pool as a set of machines. This makes management of datanodes -easier. - -### Nodes - -The data node where data is stored. http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/pom.xml b/hadoop-ozone/docs/pom.xml index 92680ad..d8edd15 100644 --- a/hadoop-ozone/docs/pom.xml +++ b/hadoop-ozone/docs/pom.xml @@ -47,10 +47,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> </executions> <configuration> <executable>dev-support/bin/generate-site.sh</executable> - <arguments> - <argument>-b</argument> - <argument>/docs</argument> - </arguments> </configuration> </plugin> <plugin> http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html index 0fdd1ba..1970f73 100644 --- a/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html +++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/_default/single.html @@ -21,7 +21,10 @@ <div class="row"> {{ partial "sidebar.html" . }} <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main"> + <h1>{{ .Title }}</h1> + <div class="col-md-9"> {{ .Content }} + </div> </div> </div> </div> http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html index c1f47a9..220db57 100644 --- a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html +++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/header.html @@ -23,9 +23,9 @@ <title>Documentation for Apache Hadoop Ozone</title> <!-- Bootstrap core CSS --> - <link href="{{ "css/bootstrap.min.css" | relURL }}" rel="stylesheet"> + <link href="css/bootstrap.min.css" rel="stylesheet"> <!-- Custom styles for this template --> - <link href="{{ "css/ozonedoc.css" | relURL }}" rel="stylesheet"> + <link href="css/ozonedoc.css" rel="stylesheet"> </head> http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html index b043911..36bed1a 100644 --- a/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html +++ b/hadoop-ozone/docs/themes/ozonedoc/layouts/partials/sidebar.html @@ -12,6 +12,7 @@ limitations under the License. See accompanying LICENSE file. --> <div class="col-sm-3 col-md-2 sidebar"> + <img src="ozone-logo.png" style="max-width: 100%;"/> <ul class="nav nav-sidebar"> {{ $currentPage := . }} {{ range .Site.Menus.main }} @@ -31,7 +32,12 @@ </li> {{ else }} <li class="{{ if $currentPage.IsMenuCurrent "main" . }}active{{ end }}"> - <a href="{{ .URL }}"> + {{ if eq .URL "/" }} + <a href="index.html"> + {{ else }} + <a href="{{ .URL }}"> + {{ end }} + {{ .Pre }} <span>{{ .Name }}</span> </a> http://git-wip-us.apache.org/repos/asf/hadoop/blob/8af84535/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css ---------------------------------------------------------------------- diff --git a/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css b/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css index 39fae72..7f68b64 100644 --- a/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css +++ b/hadoop-ozone/docs/themes/ozonedoc/static/css/ozonedoc.css @@ -23,10 +23,20 @@ /* Move down content because we have a fixed navbar that is 50px tall */ body { padding-top: 50px; - font-size: 150%; + font-size: 14pt; } +pre { + font-size: 14pt; +} + +a { + color: #448628; +} +a:hover { + color: #245F0B; +} /* * Global add-ons */ @@ -86,7 +96,7 @@ body { .nav-sidebar .active > a:hover, .nav-sidebar .active > a:focus { color: #fff; - background-color: #428bca; + background-color: #73B148; } --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
