----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5145/ -----------------------------------------------------------
Review request for Flume. Summary ------- Code review for "Create metric collection infrastructure". This patch adds fixes for the following - Added MetricAware interface, MetricContext data-structure, MetricCollector - Implemented MetricAware in AvroSource, HDFSEventSink, and AvroSink - Monitoring plugin processor: Added support in application bootstrap code to parse and configure monitoring plugin(s) - Monitoring plugin implementation: Added Ganglia 3.0 and 3.1 plugin to emit metrics to gmond host:port Configuration for monitoring plugin: ------------------------------- agent.monitoring = ganglia_monitoring jmx_monitoring # plugin type: ganglia31 for version 3.1 compatible, ganglia3 for version 3.0 compatible and wire protocol is different for both versions. agent.monitoring.ganglia_monitoring.type = ganglia31 # hostname where gmond daemon runs to collect and persist metrics in RRD agent.monitoring.ganglia_monitoring.hostName = ganglia_host # gmond port agent.monitoring.ganglia_monitoring.port = 8649 # polls every x seconds to collect metrics from metrics collector and emit them to gmond agent.monitoring.ganglia_monitoring.pollingInterval = 5 agent.monitoring.jmx_monitoring.type = jmx agent.monitoring.jmx_monitoring.port = 8081 TODO: ------ - Convert all the sources, channels, sinks to use MetricAware interface and let them be polled by MetricCollector - Aggregation of metrics in collector side? Most of the counters uses AtomicLong and increments the value upon success/failure, how do we aggregare for minutes, hours? I think monitoring solution does the aggregation (e.g, ganglia) - MetricContext: Different data type (Double, Float) to store decimal values. - Metrics: Other than counts, what else we needed? JVM metrics (we can get from java.lang.management MXBeans) - Channel: We can keep track of capacity, number of puts, number of takes, remaining capacity (or exception count?) - Monitoring plugins: JMX, REST - Embedded Jetty container: Can serve simple web ui to display the metrics (as like DN, NN does) Thanks, Mubarak Diffs ----- trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java 1338454 trunk/flume-ng-node/src/main/java/org/apache/flume/node/NodeConfiguration.java 1338454 trunk/flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 1338454 trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1338454 trunk/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1338454 trunk/flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 1338454 trunk/flume-ng-node/src/main/java/org/apache/flume/conf/file/AbstractFileConfigurationProvider.java 1338454 trunk/flume-ng-node/src/main/java/org/apache/flume/conf/file/SimpleNodeConfiguration.java 1338454 trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1338454 trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 1338454 trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1338530 trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/plugin/Ganglia3Monitoring.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/MetricContext.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/plugin/Ganglia31Monitoring.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/MetricAware.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/MetricCollector.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/Monitoring.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/MonitoringFactory.java PRE-CREATION trunk/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 1338454 trunk/flume-ng-core/src/main/java/org/apache/flume/metrics/DefaultMonitoringFactory.java PRE-CREATION trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/monitoring/MonitoringType.java PRE-CREATION trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/monitoring/MonitoringConfiguration.java PRE-CREATION trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java 1338454 trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java 1338454 trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/BasicConfigurationConstants.java 1338454 trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java 1338454 Diff: https://reviews.apache.org/r/5145/diff Testing ------- Yes. Tested in lab environment. Thanks, Mubarak
