Repository: hadoop Updated Branches: refs/heads/HDFS-7240 68eab679e -> ec4d25fc5
HDFS-11566. Ozone: Document missing metrics for container operations. Contributed by Yiqun Lin. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ec4d25fc Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ec4d25fc Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ec4d25fc Branch: refs/heads/HDFS-7240 Commit: ec4d25fc537bceae7c9021c313681d5db2f1b753 Parents: 68eab67 Author: Anu Engineer <[email protected]> Authored: Mon Apr 3 15:01:23 2017 -0700 Committer: Anu Engineer <[email protected]> Committed: Mon Apr 3 15:01:23 2017 -0700 ---------------------------------------------------------------------- .../src/site/markdown/Ozonemetrics.md | 89 ++++++++++++++++++++ hadoop-project/src/site/site.xml | 1 + 2 files changed, 90 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/ec4d25fc/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md new file mode 100644 index 0000000..4822607 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Ozonemetrics.md @@ -0,0 +1,89 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + + + +HDFS Ozone Metrics +=============== + +<!-- MACRO{toc|fromDepth=0|toDepth=3} --> + +Overview +-------- + +The container metrics that is used in HDFS Ozone. + +### Storage Container Metrics + +The metrics for various storage container operations in HDFS Ozone. + +Storage container is an optional service that can be enabled by setting +'ozone.enabled' to true. +These metrics are only available when ozone is enabled. + +Storage Container Metrics maintains a set of generic metrics for all +container RPC calls that can be made to a datandoe/container. + +Along with the total number of RPC calls containers maintain a set of metrics +for each RPC call. Following is the set of counters maintained for each RPC +operation. + +*Total number of operation* - We maintain an array which counts how +many times a specific operation has been performed. +Eg.`NumCreateContainer` tells us how many times create container has been +invoked on this datanode. + +*Number of bytes involved in a specific command* - This is an array that is +maintained for all operations, but makes sense only for read and write +operations. + +While it is possible to read the bytes in update container, it really makes +no sense, since no data stream involved. Users are advised to use this +metric only when it makes sense. Eg. `BytesReadChunk` -- Tells us how +many bytes have been read from this data using Read Chunk operation. + +*Average Latency of each operation* - The average latency of the operation. +Eg. `LatencyCreateContainerAvgTime` - This tells us the average latency of +Create Container. + +*Quantiles for each of these operations* - The 50/75/90/95/99th percentile +of these operations. Eg. `CreateContainerNanos60s50thPercentileLatency` -- +gives latency of the create container operations at the 50th percentile latency +(1 minute granularity). We report 50th, 75th, 90th, 95th and 99th percentile +for all RPCs. + +So this leads to the containers reporting these counters for each of these +RPC operations. + +| Name | Description | +|:---- |:---- | +| `NumOps` | Total number of container operations | +| `CreateContainer` | Create container operation | +| `ReadContainer` | Read container operation | +| `UpdateContainer` | Update container operations | +| `DeleteContainer` | Delete container operations | +| `ListContainer` | List container operations | +| `PutKey` | Put key operations | +| `GetKey` | Get key operations | +| `DeleteKey` | Delete key operations | +| `ListKey` | List key operations | +| `ReadChunk` | Read chunk operations | +| `DeleteChunk` | Delete chunk operations | +| `WriteChunk` | Write chunk operations| +| `ListChunk` | List chunk operations | +| `CompactChunk` | Compact chunk operations | +| `PutSmallFile` | Put small file operations | +| `GetSmallFile` | Get small file operations | +| `CloseContainer` | Close container operations | + http://git-wip-us.apache.org/repos/asf/hadoop/blob/ec4d25fc/hadoop-project/src/site/site.xml ---------------------------------------------------------------------- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index ae3aef5..bf1fd40 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -180,6 +180,7 @@ <item name="Java API docs" href="api/index.html"/> <item name="Unix Shell API" href="hadoop-project-dist/hadoop-common/UnixShellAPI.html"/> <item name="Metrics" href="hadoop-project-dist/hadoop-common/Metrics.html"/> + <item name="Ozone Metrics" href="hadoop-project-dist/hadoop-hdfs/Ozonemetrics.html"/> </menu> <menu name="Configuration" inherit="top"> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
