Repository: hadoop Updated Branches: refs/heads/HDFS-7240 66552374e -> b9a6441e1
HDFS-11963. Ozone: Documentation: Add getting started page (addendum). Contributed by Yiqun Lin. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b9a6441e Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b9a6441e Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b9a6441e Branch: refs/heads/HDFS-7240 Commit: b9a6441e1b297dd5f6ee8eaebf7ec3f7ec34c019 Parents: 6655237 Author: Yiqun Lin <[email protected]> Authored: Wed Jun 21 10:32:35 2017 +0800 Committer: Yiqun Lin <[email protected]> Committed: Wed Jun 21 10:32:35 2017 +0800 ---------------------------------------------------------------------- .../src/site/markdown/OzoneMetrics.md | 128 +++++++++++++++++++ .../src/site/markdown/Ozonemetrics.md | 128 ------------------- hadoop-project/src/site/site.xml | 3 +- 3 files changed, 129 insertions(+), 130 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/b9a6441e/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneMetrics.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneMetrics.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneMetrics.md new file mode 100644 index 0000000..21e3474 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneMetrics.md @@ -0,0 +1,128 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + + + +HDFS Ozone Metrics +=============== + +<!-- MACRO{toc|fromDepth=0|toDepth=3} --> + +Overview +-------- + +The container metrics that is used in HDFS Ozone. + +### Storage Container Metrics + +The metrics for various storage container operations in HDFS Ozone. + +Storage container is an optional service that can be enabled by setting +'ozone.enabled' to true. +These metrics are only available when ozone is enabled. + +Storage Container Metrics maintains a set of generic metrics for all +container RPC calls that can be made to a datandoe/container. + +Along with the total number of RPC calls containers maintain a set of metrics +for each RPC call. Following is the set of counters maintained for each RPC +operation. + +*Total number of operation* - We maintain an array which counts how +many times a specific operation has been performed. +Eg.`NumCreateContainer` tells us how many times create container has been +invoked on this datanode. + +*Number of bytes involved in a specific command* - This is an array that is +maintained for all operations, but makes sense only for read and write +operations. + +While it is possible to read the bytes in update container, it really makes +no sense, since no data stream involved. Users are advised to use this +metric only when it makes sense. Eg. `BytesReadChunk` -- Tells us how +many bytes have been read from this data using Read Chunk operation. + +*Average Latency of each operation* - The average latency of the operation. +Eg. `LatencyCreateContainerAvgTime` - This tells us the average latency of +Create Container. + +*Quantiles for each of these operations* - The 50/75/90/95/99th percentile +of these operations. Eg. `CreateContainerNanos60s50thPercentileLatency` -- +gives latency of the create container operations at the 50th percentile latency +(1 minute granularity). We report 50th, 75th, 90th, 95th and 99th percentile +for all RPCs. + +So this leads to the containers reporting these counters for each of these +RPC operations. + +| Name | Description | +|:---- |:---- | +| `NumOps` | Total number of container operations | +| `CreateContainer` | Create container operation | +| `ReadContainer` | Read container operation | +| `UpdateContainer` | Update container operations | +| `DeleteContainer` | Delete container operations | +| `ListContainer` | List container operations | +| `PutKey` | Put key operations | +| `GetKey` | Get key operations | +| `DeleteKey` | Delete key operations | +| `ListKey` | List key operations | +| `ReadChunk` | Read chunk operations | +| `DeleteChunk` | Delete chunk operations | +| `WriteChunk` | Write chunk operations| +| `ListChunk` | List chunk operations | +| `CompactChunk` | Compact chunk operations | +| `PutSmallFile` | Put small file operations | +| `GetSmallFile` | Get small file operations | +| `CloseContainer` | Close container operations | + +### Key Space Metrics + +The metrics for various key space manager operations in HDFS Ozone. + +key space manager (KSM) is a service that similar to the Namenode in HDFS. +In the current design of KSM, it maintains metadata of all volumes, buckets and keys. +These metrics are only available when ozone is enabled. + +Following is the set of counters maintained for each key space operation. + +*Total number of operation* - We maintain an array which counts how +many times a specific operation has been performed. +Eg.`NumVolumeCreate` tells us how many times create volume has been +invoked in KSM. + +*Total number of failed operation* - This type operation is opposite to the above +operation. +Eg.`NumVolumeCreateFails` tells us how many times create volume has been invoked +failed in KSM. + +Following are the counters for each of key space operations. + +| Name | Description | +|:---- |:---- | +| `VolumeCreate` | Create volume operation | +| `VolumeUpdates` | Update volume property operation | +| `VolumeInfos` | Get volume information operation | +| `VolumeCheckAccesses` | Check volume access operation | +| `VolumeDeletes` | Delete volume operation | +| `VolumeLists` | List volume operation | +| `BucketCreates` | Create bucket operation | +| `BucketInfos` | Get bucket information operation | +| `BucketUpdates` | Update bucket property operation | +| `BucketDeletes` | Delete bucket operation | +| `BucketLists` | List bucket operation | +| `KeyAllocate` | Allocate key operation | +| `KeyLookup` | Look up key operation | +| `KeyDeletes` | Delete key operation | +| `KeyLists` | List key operation | \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/b9a6441e/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md deleted file mode 100644 index 21e3474..0000000 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md +++ /dev/null @@ -1,128 +0,0 @@ -<!--- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - - - -HDFS Ozone Metrics -=============== - -<!-- MACRO{toc|fromDepth=0|toDepth=3} --> - -Overview --------- - -The container metrics that is used in HDFS Ozone. - -### Storage Container Metrics - -The metrics for various storage container operations in HDFS Ozone. - -Storage container is an optional service that can be enabled by setting -'ozone.enabled' to true. -These metrics are only available when ozone is enabled. - -Storage Container Metrics maintains a set of generic metrics for all -container RPC calls that can be made to a datandoe/container. - -Along with the total number of RPC calls containers maintain a set of metrics -for each RPC call. Following is the set of counters maintained for each RPC -operation. - -*Total number of operation* - We maintain an array which counts how -many times a specific operation has been performed. -Eg.`NumCreateContainer` tells us how many times create container has been -invoked on this datanode. - -*Number of bytes involved in a specific command* - This is an array that is -maintained for all operations, but makes sense only for read and write -operations. - -While it is possible to read the bytes in update container, it really makes -no sense, since no data stream involved. Users are advised to use this -metric only when it makes sense. Eg. `BytesReadChunk` -- Tells us how -many bytes have been read from this data using Read Chunk operation. - -*Average Latency of each operation* - The average latency of the operation. -Eg. `LatencyCreateContainerAvgTime` - This tells us the average latency of -Create Container. - -*Quantiles for each of these operations* - The 50/75/90/95/99th percentile -of these operations. Eg. `CreateContainerNanos60s50thPercentileLatency` -- -gives latency of the create container operations at the 50th percentile latency -(1 minute granularity). We report 50th, 75th, 90th, 95th and 99th percentile -for all RPCs. - -So this leads to the containers reporting these counters for each of these -RPC operations. - -| Name | Description | -|:---- |:---- | -| `NumOps` | Total number of container operations | -| `CreateContainer` | Create container operation | -| `ReadContainer` | Read container operation | -| `UpdateContainer` | Update container operations | -| `DeleteContainer` | Delete container operations | -| `ListContainer` | List container operations | -| `PutKey` | Put key operations | -| `GetKey` | Get key operations | -| `DeleteKey` | Delete key operations | -| `ListKey` | List key operations | -| `ReadChunk` | Read chunk operations | -| `DeleteChunk` | Delete chunk operations | -| `WriteChunk` | Write chunk operations| -| `ListChunk` | List chunk operations | -| `CompactChunk` | Compact chunk operations | -| `PutSmallFile` | Put small file operations | -| `GetSmallFile` | Get small file operations | -| `CloseContainer` | Close container operations | - -### Key Space Metrics - -The metrics for various key space manager operations in HDFS Ozone. - -key space manager (KSM) is a service that similar to the Namenode in HDFS. -In the current design of KSM, it maintains metadata of all volumes, buckets and keys. -These metrics are only available when ozone is enabled. - -Following is the set of counters maintained for each key space operation. - -*Total number of operation* - We maintain an array which counts how -many times a specific operation has been performed. -Eg.`NumVolumeCreate` tells us how many times create volume has been -invoked in KSM. - -*Total number of failed operation* - This type operation is opposite to the above -operation. -Eg.`NumVolumeCreateFails` tells us how many times create volume has been invoked -failed in KSM. - -Following are the counters for each of key space operations. - -| Name | Description | -|:---- |:---- | -| `VolumeCreate` | Create volume operation | -| `VolumeUpdates` | Update volume property operation | -| `VolumeInfos` | Get volume information operation | -| `VolumeCheckAccesses` | Check volume access operation | -| `VolumeDeletes` | Delete volume operation | -| `VolumeLists` | List volume operation | -| `BucketCreates` | Create bucket operation | -| `BucketInfos` | Get bucket information operation | -| `BucketUpdates` | Update bucket property operation | -| `BucketDeletes` | Delete bucket operation | -| `BucketLists` | List bucket operation | -| `KeyAllocate` | Allocate key operation | -| `KeyLookup` | Look up key operation | -| `KeyDeletes` | Delete key operation | -| `KeyLists` | List key operation | \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/b9a6441e/hadoop-project/src/site/site.xml ---------------------------------------------------------------------- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 9fb8f30..889f62e 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -106,7 +106,7 @@ <menu name="Ozone" inherit="top"> <item name="Getting Started" href="hadoop-project-dist/hadoop-hdfs/OzoneGettingStarted.html"/> <item name="Commands Reference" href="hadoop-project-dist/hadoop-hdfs/OzoneCommandShell.html"/> - <item name="Ozone Metrics" href="hadoop-project-dist/hadoop-hdfs/Ozonemetrics.html"/> + <item name="Ozone Metrics" href="hadoop-project-dist/hadoop-hdfs/OzoneMetrics.html"/> </menu> <menu name="MapReduce" inherit="top"> @@ -186,7 +186,6 @@ <item name="Java API docs" href="api/index.html"/> <item name="Unix Shell API" href="hadoop-project-dist/hadoop-common/UnixShellAPI.html"/> <item name="Metrics" href="hadoop-project-dist/hadoop-common/Metrics.html"/> - <item name="Ozone Metrics" href="hadoop-project-dist/hadoop-hdfs/Ozonemetrics.html"/> </menu> <menu name="Configuration" inherit="top"> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
