If you are using Hortonworks distro and installed Flume via Ambari .. Then you should be able to monitor much of that from Ambari. If you used command line installation, then you can do what Iain suggested below. -roshan
From: iain wright <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, August 5, 2015 3:42 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: How can i get flume performance metrix? Jmx For CPU/mem/GC/etc. you should be able to add this to JAVA_OPTS in flume-env.sh (im not 100% sure hortonworks has this, or that it hasn't been abstracted/pulled into an init script or something) -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false IE: JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" For tracking events through the pipeline, im not sure if those are in JMX, i don't believe they are, we use the /metrics endpoint flume serves up: [root@ip-172-16-1-119 ec2-user]# curl -s http://localhost:16742/metrics | python -mjson.tool { "CHANNEL.channel1": { "ChannelCapacity": "100000000", "ChannelFillPercentage": "0.0", "ChannelSize": "0", "EventPutAttemptCount": "42355657", "EventPutSuccessCount": "42355657", "EventTakeAttemptCount": "42391346", "EventTakeSuccessCount": "42355657", "StartTime": "1438627774367", "StopTime": "0", "Type": "CHANNEL" }, "SINK.sink1": { "BatchCompleteCount": "423556", "BatchEmptyCount": "35688", "BatchUnderflowCount": "1", "ConnectionClosedCount": "1634", "ConnectionCreatedCount": "1639", "ConnectionFailedCount": "20", "EventDrainAttemptCount": "42355657", "EventDrainSuccessCount": "42355657", "StartTime": "1438627774370", "StopTime": "0", "Type": "SINK" }, "SOURCE.source1": { "AppendAcceptedCount": "0", "AppendBatchAcceptedCount": "423562", "AppendBatchReceivedCount": "423562", "AppendReceivedCount": "0", "EventAcceptedCount": "42355657", "EventReceivedCount": "42355657", "OpenConnectionCount": "0", "StartTime": "1438627774387", "StopTime": "0", "Type": "SOURCE" } } You can poll this, awk/grep/parse/etc. & store values of channelsize or take deltas of event drained success to extrapolate an average throughput Note that flume would need to be started with these properties for /metrics to work: -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=16742 Hope that helps -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Wed, Aug 5, 2015 at 2:39 PM, Sutanu Das <[email protected]<mailto:[email protected]>> wrote: Hi Team, Can you please advise how can I see the Flume jmx performance metrix in our production? Our management is interested to know Flume performance usage as far as cpu, mem, gc, events, etc..... We use Hortonworks Flume 1.5.2.2
