That was it! I added flume.reporter.poller.period in flume-conf.xml with a value of 60000 and the reporting traffic is now back to a sensible level...:-).
Thanks so much for your replies, they really helped me out. -----Original Message----- From: Jeong-shik Jang [mailto:[email protected]] Sent: Thursday, 17 January 2013 9:32 PM To: [email protected] Subject: Re: Constant Traffic on port 35872 You may want to check property named flume.reporter.poller.period in flume-conf.xml or flume-site.xml; default value is 2000 (millis) in case of not being defined. if you cannot find the property in flume-conf.xml, add it to flume-site.xml and set value to 60000 (60 sec). I am not sure which version of flume-og you are using, but in what I am using (v0.9.4), the related class seems to have been deprecated. - JS On 1/17/13 4:00 PM, Alexander Alten-Lorenz wrote: > Depends on the architecture, since the nodes are configured per master's > webui. The master calls regularly the in-memory config and spread them > around. This is needed for HA, as example. > Flume 1.x up has another architecture. > > - Alex > > On Jan 17, 2013, at 5:02 AM, James Stewart <[email protected]> > wrote: > >> Yeah, I've just realised that it's*exactly* the same data that is returned >> when you connect to http://my.flume.node:35862, (for monitoring etc). Even >> the order in which the metrics are sent is the same. >> >> So it seems that the node is generating this configuration data and pumping >> it back to the master every 1-2 seconds. This produces ~40-80Kb/sec of >> largely unnecessary traffic per node, which soon adds up over a WAN. >> >> I can understand why this config data would be sent back to the master >> occasionally but I don't understand why it does so every 1-2 seconds, >> ignoring flume.config.heartbeat.period. >> >> From: Mike Percy [mailto:[email protected]] >> Sent: Thursday, 17 January 2013 2:37 PM >> To: [email protected] >> Subject: Re: Constant Traffic on port 35872 >> >> I doubt it's the Thrift RPC layer. It's most likely the app. >> >> On Wed, Jan 16, 2013 at 5:53 PM, James Stewart >> <[email protected]<mailto:[email protected]>> wrote: >> I thought it was only 'heartbeats' that were supposed to go via port 35872, >> so I reduced flume.config.heartbeat.period on the nodes to 60 sec. According >> to the master it's only seeing heartbeats every 60 sec now, and yet I still >> get constantly spammed with data on port 35872 from every node. >> >> It does look like metric collection or config reporting of some kind, like >> it's reporting the configuration of the sources/sinks and even data about >> the JVM: >> >> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... >> rt.vmname...!Java HotSpot(TM) 64-Bit Server >> VM....name...(pn-opsynxsr0202.aus.optiver.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun >> Microsystems Inc. >> ..........max.....@......mem.other.used................mem.other.committed.....s......mem.heap.max<mailto:..........max.....@......mem.other.used................mem.other.committed.....s......mem.heap.max>........... >> >> But it's just the same data over and over again every second. This traffic >> is travelling across a WAN and with a lot of nodes it's a significant enough >> amount of data to be a problem. >> >> I don't know much about Java, but could this be something to do with Thrift? >> >> >> From: Mike Percy [mailto:[email protected]<mailto:[email protected]>] >> Sent: Thursday, 17 January 2013 12:19 PM >> To: [email protected]<mailto:[email protected]> >> Subject: Re: Constant Traffic on port 35872 >> >> I know next to nothing about Flume OG but if I had to guess I'd say it's >> either a heartbeat or metrics collection. Why do you want it to stop? >> >> On Wed, Jan 16, 2013 at 5:06 PM, James Stewart >> <[email protected]<mailto:[email protected]>> wrote: >> Hello all, >> >> I'm using flume 0.9.4 - before anybody mentions it, we aren't in a position >> to upgrade at the moment due to custom decorators + sinks. >> >> I'm seeing constant traffic from my various flume nodes back to my master on >> port 35872. Even after increasing my timeout period to 60 sec and disabling >> all custom sources/sinks/decorators, I am still constantly receiving packets >> from all of my nodes back to my master. I have included a dump of the tcp >> packets below - I receive this same traffic from every node every 1-2 sec. >> >> Does anybody have any idea what is generating this traffic and how I can >> stop it? >> >> Many thanks, >> >> James >> >> >> V..?..-MLogicalNodeManager.my_logical_node_apps.sourceConfig....null...1LogicalNodeManager.my_logical_node_apps.physicalnode....my_physical_node.my_domain.com...:LogicalNodeManager.my_physical_node.my_domain.com.sinkConfig....null...-LogicalNodeManager.my_logical_node_apps.hostname....my_physical_node.my_domain.com....AgentWALAckManager.name....AgentWALAckManager...8LogicalNodeManager.my_physical_node.my_domain.com.nodename....my_physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.my_domain.com.sourceConfig....null....LogicalNodeManager.name....LogicalNodeManager...)LogicalNodeManager.my_logical_node_apps.name....my_logical_node_apps.../LogicalNodeManager.my_logical_node_apps.sinkConfig....null...*LogicalNodeManager.my_logical_node_apps.state....IDLE...8LogicalNodeManager.my_physical_node.my_domain.com.hostname....my_physical_node.my_domain.com...-LogicalNodeManager.my_logical_node_apps.nodename....my_logical_node_apps...9LogicalNodeManager.my_physical_node.my_domain.com.sink.name....NullSink....name....pn-my_physical_node.my_domain.com....LogicalNodeManager.my_logical_node_apps.sink.name....NullSink....LogicalNodeManager.my_logical_node_apps.sink.type....NullSink...7LogicalNodeManager.my_physical_node.my_domain.com.version....Thu >> Jan 01 10:00:00 EST >> 1970...9LogicalNodeManager.my_physical_node.my_domain.com.sink.type....NullSink...5LogicalNodeManager.my_physical_node.my_domain.com.state....IDLE...,LogicalNodeManager.my_logical_node_apps.version....Thu >> Jan 17 11:45:44 EST 2013... >> 11:58:22.630881 IP my_physical_node.my_domain.com.53367 > >> my_master.35872: . 331777:333225(1448) ack 2613 win 6 <nop,nop,timestamp >> 1456047935 3787992397> E.....@.<<mailto:E.....@.%3c>... >> I.. >> B...w. ."..CC1L........... >> ...?..-M4LogicalNodeManager.my_physical_node.my_domain.com.name....my >> _physical_node.my_domain.com...<LogicalNodeManager.my_physical_node.m >> y_domain.com.physicalnode....my_physical_node.my_domain.com<http://my >> _physical_node.my_domain.com> >> .......<LogicalNodeManager.my_physical_node.my_domain.com.reconfigures...........1LogicalNodeManager.my_logical_node_apps.reconfigures...........9LogicalNodeManager.my_logical_node_apps.sink.number >> of >> bytes...........:LogicalNodeManager.my_logical_node_apps.sink.n............name...sr0202.aus.optiver.com.my_physical_node.my_domain.com.NullSourceber >> of events............ >> NullSource....type... >> ...lSource >> ............name....NullSink....type....NullSinksr0202_apps.NullSink >> ............name...sr0202.aus.optiver.com.my_logical_node_apps.NullSo >> urce >> NullSource....type... >> ...lSource >> ............name....NullSink....type....NullSinksr0202.aus.optiver.co >> m.NullSink ...........(pn-my_logical_node.aus.optiumber of >> bytes........ >> 11:58:22.630891 IP my_physical_node.my_domain.com.53367 > >> my_master.35872: P 333225:333700(475) ack 2613 win 6 >> <nop,nop,timestamp 1456047935 3787992397> >> E.....@.<..c<mailto:E.....@.%3c..c> >> I.. >> ..w. .".:CC1L........... >> ............rt.starttime....Thu Jan 17 11:47:07 EST 2013... >> rt.vmname...!Java HotSpot(TM) 64-Bit Server >> VM....name...(pn-my_physical_node.my_domain.com.jvm-Info....rt.vmversion....16.3-b01....rt.vmvendor....Sun >> Microsystems Inc. >> ..........max.....@......mem.other.used................mem.other.committed.....s......mem.heap.max<mailto:..........max.....@......mem.other.used................mem.other.committed.....s......mem.heap.max>........... >> >> ________________________________ >> Information contained in this communication (including any attachments) is >> confidential and may be privileged or subject to copyright. If you have >> received this communication in error you are not authorised to use the >> information in any way and Optiver requests that you notify the sender by >> return email, destroy all copies and delete the information from your >> system. Optiver does not represent, warrant or guarantee that this >> communication is free from computer viruses or other defects or that the >> integrity of this communication has been maintained. Any views expressed in >> this communication are those of the individual sender. Optiver does not >> accept liability for any loss or damage caused directly or indirectly by >> this communication or its use. >> >> Please consider the environment before printing this email. >> >> > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > > -- Jeong-shik Jang / [email protected] Gruter, Inc., R&D Team Leader www.gruter.com Enjoy Connecting
