Updated Branches: refs/heads/master 583453721 -> acba59b16
http://git-wip-us.apache.org/repos/asf/accumulo/blob/acba59b1/server/src/main/resources/docs/metrics.html ---------------------------------------------------------------------- diff --git a/server/src/main/resources/docs/metrics.html b/server/src/main/resources/docs/metrics.html new file mode 100644 index 0000000..00f0a5b --- /dev/null +++ b/server/src/main/resources/docs/metrics.html @@ -0,0 +1,182 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> +<head> +<title>Accumulo Metrics</title> +<link rel='stylesheet' type='text/css' href='documentation.css' media='screen'/> +</head> +<body> + +<h1>Apache Accumulo Documentation : Metrics</h1> + +As of version 1.2, metrics for the Master, Tablet Servers, and Loggers are available. A new configuration file, accumulo-metrics.xml, is located in the conf directory and can +be modified to turn metrics collection on or off, and to enable file logging if desired. This file can be modified at runtime and the changes will be seen after a few seconds. +Except where specified all time values are in milliseconds. +<h1>Master Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=MasterMetricsMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getPingCount();</td><td>Number of pings to tablet servers</td></tr> + <tr><td>public long getPingAvgTime();</td><td>Average time for each ping</td></tr> + <tr class="highlight"><td>public long getPingMinTime();</td><td>Minimum time for each ping</td></tr> + <tr><td>public long getPingMaxTime();</td><td>Maximum time for each ping</td></tr> + <tr class="highlight"><td>public String getTServerWithHighestPingTime();</td><td>tablet server with highest ping</td></tr> + <tr><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +<h1>Logging Server Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=LogWriterMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getCloseCount();</td><td>Number of closed log files</td></tr> + <tr><td>public long getCloseAvgTime();</td><td>Average time to close a log file</td></tr> + <tr class="highlight"><td>public long getCloseMinTime();</td><td>Minimum time to close a log file</td></tr> + <tr><td>public long getCloseMaxTime();</td><td>Maximum time to close a log file</td></tr> + <tr class="highlight"><td>public long getCopyCount();</td><td>Number of log files copied</td></tr> + <tr><td>public long getCopyAvgTime();</td><td>Average time to copy a log file</td></tr> + <tr class="highlight"><td>public long getCopyMinTime();</td><td>Minimum time to copy a log file</td></tr> + <tr><td>public long getCopyMaxTime();</td><td>Maximum time to copy a log file</td></tr> + <tr class="highlight"><td>public long getCreateCount();</td><td>Number of log files created</td></tr> + <tr><td>public long getCreateMinTime();</td><td>Minimum time to create a log file</td></tr> + <tr class="highlight"><td>public long getCreateMaxTime();</td><td>Maximum time to create a log file</td></tr> + <tr><td>public long getCreateAvgTime();</td><td>Average time to create a log file</td></tr> + <tr class="highlight"><td>public long getLogAppendCount();</td><td>Number of times logs have been appended</td></tr> + <tr><td>public long getLogAppendMinTime();</td><td>Minimum time to append to a log file</td></tr> + <tr class="highlight"><td>public long getLogAppendMaxTime();</td><td>Maximum time to append to a log file</td></tr> + <tr><td>public long getLogAppendAvgTime();</td><td>Average time to append to a log file</td></tr> + <tr class="highlight"><td>public long getLogFlushCount();</td><td>Number of log file flushes</td></tr> + <tr><td>public long getLogFlushMinTime();</td><td>Minimum time to flush a log file</td></tr> + <tr class="highlight"><td>public long getLogFlushMaxTime();</td><td>Maximum time to flush a log file</td></tr> + <tr><td>public long getLogFlushAvgTime();</td><td>Average time to flush a log file</td></tr> + <tr class="highlight"><td>public long getLogExceptionCount();</td><td>Number of log exceptions</td></tr> + <tr><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +<h1>Tablet Server Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public int getOnlineCount();</td><td>Number of tablets online</td></tr> + <tr><td>public int getOpeningCount();</td><td>Number of tablets that are being opened</td></tr> + <tr class="highlight"><td>public int getUnopenedCount();</td><td>Number or unopened tablets</td></tr> + <tr><td>public int getMajorCompactions();</td><td>Number of Major Compactions currently running</td></tr> + <tr class="highlight"><td>public int getMajorCompactionsQueued();</td><td>Number of Major Compactions yet to run</td></tr> + <tr><td>public int getMinorCompactions();</td><td>Number of Minor Compactions currently running</td></tr> + <tr class="highlight"><td>public int getMinorCompactionsQueued();</td><td>Number of Minor Compactions yet to run</td></tr> + <tr><td>public int getShutdownStage();</td><td>Current stage in the shutdown process</td></tr> + <tr class="highlight"><td>public long getEntries();</td><td>Number of entries in all the tablets</td></tr> + <tr><td>public long getEntriesInMemory();</td><td>Number of entries in memory on all tablet servers</td></tr> + <tr class="highlight"><td>public long getQueries();</td><td>Number of queries currently running on all the tablet servers</td></tr> + <tr><td>public long getIngest();</td><td>Number of entries currently being ingested on all the tablet servers</td></tr> + <tr class="highlight"><td>public long getTotalMinorCompactions();</td><td>Number of Minor Compactions completed</td></tr> + <tr><td>public double getHoldTime();</td><td>Number of seconds that ingest is waiting for memory to be freed on tablet servers</td></tr> + <tr class="highlight"><td>public String getName();</td><td>Address of the master</td></tr> + </tbody> +</table> +<h1>Tablet Server Minor Compaction Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerMinCMetricsMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getMinorCompactionCount();</td><td>Number of completed Minor Compactions on all tablet servers</td></tr> + <tr><td>public long getMinorCompactionAvgTime();</td><td>Average time to complete Minor Compaction</td></tr> + <tr class="highlight"><td>public long getMinorCompactionMinTime();</td><td>Minimum time to complete Minor Compaction</td></tr> + <tr><td>public long getMinorCompactionMaxTime();</td><td>Maximum time to complete Minor Compaction</td></tr> + <tr class="highlight"><td>public long getMinorCompactionQueueCount();</td><td>Number of Minor Compactions yet to be run</td></tr> + <tr><td>public long getMinorCompactionQueueAvgTime();</td><td>Average time Minor Compaction is in the queue</td></tr> + <tr class="highlight"><td>public long getMinorCompactionQueueMinTime();</td><td>Minimum time Minor Compaction is in the queue</td></tr> + <tr><td>public long getMinorCompactionQueueMaxTime();</td><td>Maximum time Minor Compaction is in the queue</td></tr> + <tr class="highlight"><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +<h1>Tablet Server Scan Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerScanMetricsMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getScanCount();</td><td>Number of scans completed</td></tr> + <tr><td>public long getScanAvgTime();</td><td>Average time for scan operation</td></tr> + <tr class="highlight"><td>public long getScanMinTime();</td><td>Minimum time for scan operation</td></tr> + <tr><td>public long getScanMaxTime();</td><td>Maximum time for scan operation</td></tr> + <tr class="highlight"><td>public long getResultCount();</td><td>Number of scans that returned a result</td></tr> + <tr><td>public long getResultAvgSize();</td><td>Average size of scan result</td></tr> + <tr class="highlight"><td>public long getResultMinSize();</td><td>Minimum size of scan result</td></tr> + <tr><td>public long getResultMaxSize();</td><td>Maximum size of scan result</td></tr> + <tr class="highlight"><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +<h1>Tablet Server Update Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=TabletServerUpdateMetricsMBean,name= <current thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getPermissionErrorCount();</td><td>Number of permission errors</td></tr> + <tr><td>public long getUnknownTabletErrorCount();</td><td>Number of unknown tablet errors</td></tr> + <tr class="highlight"><td>public long getMutationArrayAvgSize();</td><td>Average size of mutation array</td></tr> + <tr><td>public long getMutationArrayMinSize();</td><td>Minimum size of mutation array</td></tr> + <tr class="highlight"><td>public long getMutationArrayMaxSize();</td><td>Maximum size of mutation array</td></tr> + <tr><td>public long getCommitPrepCount();</td><td>Number of commit preparations</td></tr> + <tr class="highlight"><td>public long getCommitPrepMinTime();</td><td>Minimum time for commit preparation</td></tr> + <tr><td>public long getCommitPrepMaxTime();</td><td>Maximum time for commit preparatation</td></tr> + <tr class="highlight"><td>public long getCommitPrepAvgTime();</td><td>Average time for commit preparation</td></tr> + <tr><td>public long getConstraintViolationCount();</td><td>Number of constraint violations</td></tr> + <tr class="highlight"><td>public long getWALogWriteCount();</td><td>Number of writes to the Write Ahead Log</td></tr> + <tr><td>public long getWALogWriteMinTime();</td><td>Minimum time of a write to the Write Ahead Log</td></tr> + <tr class="highlight"><td>public long getWALogWriteMaxTime();</td><td>Maximum time of a write to the Write Ahead Log</td></tr> + <tr><td>public long getWALogWriteAvgTime();</td><td>Average time of a write to the Write Ahead Log</td></tr> + <tr class="highlight"><td>public long getCommitCount();</td><td>Number of commits</td></tr> + <tr><td>public long getCommitMinTime();</td><td>Minimum time for a commit</td></tr> + <tr class="highlight"><td>public long getCommitMaxTime();</td><td>Maximum time for a commit</td></tr> + <tr><td>public long getCommitAvgTime();</td><td>Average time for a commit</td></tr> + <tr class="highlight"><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +<h1>Thrift Server Metrics</h1> +<p>JMX Object Name: org.apache.accumulo.server.metrics:type=ThriftMetricsMBean,name= <thread name></p> +<table> + <thead> + <tr><td>Method Name</td><td>Description</td></tr> + </thead> + <tbody> + <tr class="highlight"><td>public long getIdleCount();</td><td>Number of times the Thrift server has been idle</td></tr> + <tr><td>public long getIdleMinTime();</td><td>Minimum amount of time the Thrift server has been idle</td></tr> + <tr class="highlight"><td>public long getIdleMaxTime();</td><td>Maximum amount of time the Thrift server has been idle</td></tr> + <tr><td>public long getIdleAvgTime();</td><td>Average time the Thrift server has been idle</td></tr> + <tr class="highlight"><td>public long getExecutionCount();</td><td>Number of calls processed by the Thrift server</td></tr> + <tr><td>public long getExecutionMinTime();</td><td>Minimum amount of time executing method</td></tr> + <tr class="highlight"><td>public long getExecutionMaxTime();</td><td>Maximum amount of time executing method</td></tr> + <tr><td>public long getExecutionAvgTime();</td><td>Average time executing methods</td></tr> + <tr class="highlight"><td>public void reset();</td><td>Resets all counters to zero</td></tr> + </tbody> +</table> +</body> +</html> http://git-wip-us.apache.org/repos/asf/accumulo/blob/acba59b1/server/src/main/resources/docs/timestamps.html ---------------------------------------------------------------------- diff --git a/server/src/main/resources/docs/timestamps.html b/server/src/main/resources/docs/timestamps.html new file mode 100644 index 0000000..52290c7 --- /dev/null +++ b/server/src/main/resources/docs/timestamps.html @@ -0,0 +1,160 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<html> +<head> +<title>Accumulo Timestamps</title> +<link rel='stylesheet' type='text/css' href='documentation.css' media='screen'/> +</head> +<body> + +<h1>Apache Accumulo Documentation : Timestamps</h1> + +<p>Everything inserted into accumulo has a timestamp. If the user does not +set it, then the system will set the timestamp. The timestamp is the last +thing accumulo sorts on. So when two keys have the same row, column family, +column qualifier, and column visibility then the timestamp of the two keys is +compared. + +<p>Timestamps are sorted in descending order, so the most recent data comes +first. When a table is created in accumulo, by default it has a versioning +iterator that only shows the most recent. In the example below two identical +things are inserted. The scan after that only shows the most recent version. +However when the versioning iterator configuration is changed, then both are +seen. When data is inserted with a lower timestamp than existing data, it will +fall behind the existing data and may not be seen depending on the versioning +settings. This is why the insert made with a timestamp of 500 is not seen in +the scan below. + +<p><pre> +root@ac12> createtable foo +root@ac12 foo> +root@ac12 foo> +root@ac12 foo> insert r1 cf1 cq1 value1 +root@ac12 foo> insert r1 cf1 cq1 value2 +root@ac12 foo> scan -st +r1 cf1:cq1 [] 1279906856203 value2 +root@ac12 foo> config -t foo -f iterator +---------+---------------------------------------------+----------------------------------------------------------------------------------------------------- +SCOPE | NAME | VALUE +---------+---------------------------------------------+----------------------------------------------------------------------------------------------------- +table | table.iterator.majc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator +table | table.iterator.majc.vers.opt.maxVersions .. | 1 +table | table.iterator.minc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator +table | table.iterator.minc.vers.opt.maxVersions .. | 1 +table | table.iterator.scan.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator +table | table.iterator.scan.vers.opt.maxVersions .. | 1 +---------+---------------------------------------------+----------------------------------------------------------------------------------------------------- +root@ac12 foo> config -t foo -s table.iterator.scan.vers.opt.maxVersions=3 +root@ac12 foo> config -t foo -s table.iterator.minc.vers.opt.maxVersions=3 +root@ac12 foo> config -t foo -s table.iterator.majc.vers.opt.maxVersions=3 +root@ac12 foo> scan -st +r1 cf1:cq1 [] 1279906856203 value2 +r1 cf1:cq1 [] 1279906853170 value1 +root@ac12 foo> insert -t 600 r1 cf1 cq1 value3 +root@ac12 foo> insert -t 500 r1 cf1 cq1 value4 +root@ac12 foo> scan -st +r1 cf1:cq1 [] 1279906856203 value2 +r1 cf1:cq1 [] 1279906853170 value1 +r1 cf1:cq1 [] 600 value3 +root@ac12 foo> + +</pre> + +<p>Deletes are special keys in accumulo that get sorted along will all the other +data. When a delete key is inserted, accumulo will not show anything that has +a timestamp less than or equal to the delete key. In the example below an +insert is made with timestamp 5 and then a delete is inserted with timestamp 3. +The scan after that show that the delete marker does not hide the key. However +when a delete is inserted with timestamp 5, then nothing can be seen. Once a +delete marker is inserted, it is there until a full major compaction occurs. +That is why the insert made after the delete can not be seen. The insert after +the flush and compact commands can be seen because the delete marker is gone. +The flush forced a minor compaction and compact forced a full major compaction. + +<p><pre> +root@ac12> createtable bar +root@ac12 bar> insert -t 5 r1 cf1 cq1 val1 +root@ac12 bar> scan -st +r1 cf1:cq1 [] 5 val1 +root@ac12 bar> delete -t 3 r1 cf1 cq1 +root@ac12 bar> scan +r1 cf1:cq1 [] val1 +root@ac12 bar> scan -st +r1 cf1:cq1 [] 5 val1 +root@ac12 bar> delete -t 5 r1 cf1 cq1 +root@ac12 bar> scan -st +root@ac12 bar> insert -t 5 r1 cf1 cq1 val2 +root@ac12 bar> scan -st +root@ac12 bar> flush -t bar +23 14:01:36,587 [shell.Shell] INFO : Flush of table bar initiated... +root@ac12 bar> compact -t bar +23 14:02:00,042 [shell.Shell] INFO : Compaction of table bar scheduled for 20100723140200EDT +root@ac12 bar> insert -t 5 r1 cf1 cq1 val1 +root@ac12 bar> scan +r1 cf1:cq1 [] val1 +</pre> + +<p>If two inserts are made into accumulo with the same row, column, and +timestamp, then the behavior is non-deterministic. + +<p>Accumulo 1.2 introduces the concept of logical time. This ensures that +timestamps set by accumulo always move forward. There have been many problems +caused by tablet servers with different system times. In the case where a +tablet servers time is in the future, tablets hosted on that tablet server and +then migrated will have future timestamps in their data. This can cause newer +keys to fall behind existing keys, which can result in seeing older data or not +seeing data if a new key falls behind on old delete. Logical time prevents +this by ensuring that accumulo set time stamps never go backwards, on a per +tablet basis. So if a tablet servers time is a year in the future, then any +tablet hosted there will generate timestamps a year in the future even when +later hosted on a server with correct time. Logical time can be configured on a +per table basis to either set time in millis or to use a per tablet counter. +The per tablet counter gives unique one up time stamps on a per mutation +basis. When using time in millis, if two things arrive within the same +millisecond then both receive the same timestamp. + +<p>The example below shows a table created using a per tablet counter for +timestamps. Two inserts are made, the first gets timestamp 0 the second 1. +After that the table is split into two tablets and two more inserts are made. +These inserts get the same timestamp because they are made on different +tablets. When the original tablet is split into two, the two child tablets +inherit the next timestamp of their parent and start from there. So do not +expect this configuration to offer unique timestamps across a table. Its only +purpose is to uniquely order events within a tablet. + +<p><pre> +root@ac12 foo> createtable -tl logical +root@ac12 logical> insert 000892 person name "John Doe" +root@ac12 logical> insert 003042 person name "Jane Doe" +root@ac12 logical> scan -st +000892 person:name [] 0 John Doe +003042 person:name [] 1 Jane Doe +root@ac12 logical> +root@ac12 logical> addsplits -t logical 002000 +root@ac12 logical> insert 003042 person address "123 Somewhere" +root@ac12 logical> insert 000892 person address "123 Nowhere" +root@ac12 logical> scan -st +000892 person:address [] 2 123 Nowhere +000892 person:name [] 0 John Doe +003042 person:address [] 2 123 Somewhere +003042 person:name [] 1 Jane Doe +root@ac12 logical> + +</pre> + +</body> +</html>
