Repository: flume Updated Branches: refs/heads/trunk 23473e1d5 -> 84b6263cb
FLUME-1342 adding jmx metrics tables to docs This PR adds a few tables to the User Guide that describe the metrics published by sorurces, sinks and channels. I used simple unix tools to gather the data then I wrote a small utility to convert it to csv. Then I used an online converter https://www.tablesgenerator.com/ to generate the rst tables and then a little manual editing. I discovered some rst formatting problems in the FlumeUserGuide.rst, corrected them, too. It was rather painful process to gather the data and find a decent representation. So far this PR only contains the end result. I would be happy to share the utilities, just don't know what would be the best way. This closes #242 Reviewers: Denes Arvay, Ferenc Szabo (Endre Major via Ferenc Szabo) Project: http://git-wip-us.apache.org/repos/asf/flume/repo Commit: http://git-wip-us.apache.org/repos/asf/flume/commit/84b6263c Tree: http://git-wip-us.apache.org/repos/asf/flume/tree/84b6263c Diff: http://git-wip-us.apache.org/repos/asf/flume/diff/84b6263c Branch: refs/heads/trunk Commit: 84b6263cbd4c771144186d4c561e12ff569e1887 Parents: 23473e1 Author: Endre Major <[email protected]> Authored: Fri Nov 23 11:47:15 2018 +0100 Committer: Ferenc Szabo <[email protected]> Committed: Fri Nov 23 11:47:15 2018 +0100 ---------------------------------------------------------------------- flume-ng-doc/sphinx/FlumeUserGuide.rst | 182 +++++++++++++++++++++++++++- 1 file changed, 178 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flume/blob/84b6263c/flume-ng-doc/sphinx/FlumeUserGuide.rst ---------------------------------------------------------------------- diff --git a/flume-ng-doc/sphinx/FlumeUserGuide.rst b/flume-ng-doc/sphinx/FlumeUserGuide.rst index 492d90e..df4d5c2 100644 --- a/flume-ng-doc/sphinx/FlumeUserGuide.rst +++ b/flume-ng-doc/sphinx/FlumeUserGuide.rst @@ -1993,7 +1993,7 @@ selector.type replicating or multiplexing selector.* replicating Depends on the selector.type value interceptors -- Space-separated list of interceptors interceptors.* -==================== =========== ================================================================= +==================== =========== ================================================================ For example, a syslog UDP source for agent named a1: @@ -2387,9 +2387,7 @@ hdfs.filePrefix FlumeData Name prefixed to files created by Flume in hdfs.fileSuffix -- Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*) hdfs.inUsePrefix -- Prefix that is used for temporal files that flume actively writes into hdfs.inUseSuffix ``.tmp`` Suffix that is used for temporal files that flume actively writes into -hdfs.emptyInUseSuffix false If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output - ``hdfs.inUseSuffix`` is removed from the output file name. - If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead. +hdfs.emptyInUseSuffix false If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output ``hdfs.inUseSuffix`` is removed from the output file name. If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead. hdfs.rollInterval 30 Number of seconds to wait before rolling current file (0 = never roll based on time interval) hdfs.rollSize 1024 File size to trigger roll, in bytes (0: never roll based on file size) @@ -4962,6 +4960,182 @@ Monitoring in Flume is still a work in progress. Changes can happen very often. Several Flume components report metrics to the JMX platform MBean server. These metrics can be queried using Jconsole. +Available Component Metrics +--------------------------- + +The following tables show what metrics are available for components. Each component only maintains a +set of metrics, indicated by an 'x', the unmaintained ones show default values, that is 0. +These tables tell you where you can expect meaningful data. +The name of the metrics should be descriptive enough, for more information you have to dig into the +source code of the components. + +Sources 1 +~~~~~~~~~ + ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| | Avro | Exec | HTTP | JMS | Kafka | MultiportSyslogTCP | Scribe | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| AppendAcceptedCount | x | | | | | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| AppendBatchAcceptedCount | x | | x | x | | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| AppendBatchReceivedCount | x | | x | x | | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| AppendReceivedCount | x | | | | | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| ChannelWriteFail | x | | x | x | x | x | x | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| EventAcceptedCount | x | x | x | x | x | x | x | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| EventReadFail | | | x | x | x | x | x | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| EventReceivedCount | x | x | x | x | x | x | x | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| GenericProcessingFail | | | x | | | x | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| KafkaCommitTimer | | | | | x | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| KafkaEmptyCount | | | | | x | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| KafkaEventGetTimer | | | | | x | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ +| OpenConnectionCount | x | | | | | | | ++--------------------------+------+------+------+-----+-------+--------------------+--------+ + +Sources 2 +~~~~~~~~~ + ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| | SequenceGenerator | SpoolDirectory | SyslogTcp | SyslogUDP | Taildir | Thrift | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| AppendAcceptedCount | | | | | | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| AppendBatchAcceptedCount | x | x | | | x | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| AppendBatchReceivedCount | | x | | | x | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| AppendReceivedCount | | | | | | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| ChannelWriteFail | x | x | x | x | x | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| EventAcceptedCount | x | x | x | x | x | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| EventReadFail | | x | x | x | x | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| EventReceivedCount | | x | x | x | x | x | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| GenericProcessingFail | | x | | | x | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| KafkaCommitTimer | | | | | | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| KafkaEmptyCount | | | | | | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| KafkaEventGetTimer | | | | | | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ +| OpenConnectionCount | | | | | | | ++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+ + +Sinks 1 +~~~~~~~ + ++------------------------+-------------+------------+---------------+-------+--------+ +| | Avro/Thrift | AsyncHBase | ElasticSearch | HBase | HBase2 | ++------------------------+-------------+------------+---------------+-------+--------+ +| BatchCompleteCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| BatchEmptyCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| BatchUnderflowCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| ChannelReadFail | x | | | | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| ConnectionClosedCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| ConnectionCreatedCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| ConnectionFailedCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| EventDrainAttemptCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| EventDrainSuccessCount | x | x | x | x | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| EventWriteFail | x | | | | x | ++------------------------+-------------+------------+---------------+-------+--------+ +| KafkaEventSendTimer | | | | | | ++------------------------+-------------+------------+---------------+-------+--------+ +| RollbackCount | | | | | | ++------------------------+-------------+------------+---------------+-------+--------+ + +Sinks 2 +~~~~~~~ + ++------------------------+-----------+------+------+-------+-----------+-------------+ +| | HDFSEvent | Hive | Http | Kafka | Morphline | RollingFile | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| BatchCompleteCount | x | x | | | x | | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| BatchEmptyCount | x | x | | x | x | | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| BatchUnderflowCount | x | x | | x | x | | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| ChannelReadFail | x | x | x | x | x | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| ConnectionClosedCount | x | x | | | | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| ConnectionCreatedCount | x | x | | | | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| ConnectionFailedCount | x | x | | | | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| EventDrainAttemptCount | x | x | x | | x | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| EventDrainSuccessCount | x | x | x | x | x | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| EventWriteFail | x | x | x | x | x | x | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| KafkaEventSendTimer | | | | x | | | ++------------------------+-----------+------+------+-------+-----------+-------------+ +| RollbackCount | | | | x | | | ++------------------------+-----------+------+------+-------+-----------+-------------+ + +Channels +~~~~~~~~ + ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| | File | Kafka | Memory | PseudoTxnMemory | SpillableMemory | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| ChannelCapacity | x | | x | | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| ChannelSize | x | | x | x | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| CheckpointBackupWriteErrorCount | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| CheckpointWriteErrorCount | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventPutAttemptCount | x | x | x | x | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventPutErrorCount | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventPutSuccessCount | x | x | x | x | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventTakeAttemptCount | x | x | x | x | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventTakeErrorCount | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| EventTakeSuccessCount | x | x | x | x | x | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| KafkaCommitTimer | | x | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| KafkaEventGetTimer | | x | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| KafkaEventSendTimer | | x | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| Open | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| RollbackCounter | | x | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ +| Unhealthy | x | | | | | ++---------------------------------+------+-------+--------+-----------------+-----------------+ + JMX Reporting -------------
