However, since the arrive order on Kafka (main sink) is not a particular problem to me, my current solution would be:
* memory channel * sinkgroup with 2 sinks: ** Kafka ** File_roll (write events on '/data/x' directory, in case Kafka is down) * periodically check the presence of files in '/data/x' and, in the case, re-push them to Kafka I still don't know whether it is possible to re-push File-roll files on Kafka using bin/flume-ng Whatever hints would be appreciated. Many thanks On Fri, Oct 16, 2015 at 4:32 PM, Simone Roselli <[email protected]> wrote: > Hi Phil, > > thanks for your reply. > > Yes, setting up a file-channel configuration is consuming CPU up to 80/90% > > My settings: > # Channel configuration > agent1.channels.ch1.type = file > agent1.channels.ch1.checkpointDir = /opt/flume-ng/chekpoint > agent1.channels.ch1.dataDirs = /opt/flume-ng/data > agent1.channels.ch1.capacity = 1000000 > agent1.channels.ch1.transactionCapacity = 10000 > > # flume-env.sh > export JAVA_OPTS="-Xms512m -Xmx2048m" > > # top > 22079 flume-ng 20 0 6924752 785536 17132 S 83.7% 2.4 3:53.19 java > > Do you have any tuning for the GC ? > > Thanks > Simone > > > > On Thu, Oct 15, 2015 at 7:59 PM, Phil Scala <[email protected]> > wrote: > >> Hi Simone >> >> >> >> I wonder why you’re seeing 90% CPU use when you use a file channel. I >> would expect high disk I/O. To counter, I have on a single server 4 spool >> dir sources, each going to a separate file channel. Also on an SSD based >> server. I do not see any CPU or even disk IO utilization. I am pushing >> about 10 million events per day across all 4 sources and has been running >> reliably for 2 years now. >> >> >> >> I would always use a file channel, any memory channel runs the risk of >> data loss if the node were to fail. I would be as worried about the local >> node failing seeing that a 3 node kafka cluster losing 2 nodes before it >> would lose quorum. >> >> >> >> Not sure what your data source is, if you can add more flume nodes of >> course that would help. >> >> >> >> Have you given ample heap space, seeing maybe GC’s causing the high CPU? >> >> >> >> >> >> Phil >> >> >> >> >> >> *From:* Simone Roselli [mailto:[email protected]] >> *Sent:* Friday, October 09, 2015 12:33 AM >> *To:* [email protected] >> *Subject:* Flume-ng 1.6 reliable setup >> >> >> >> Hi, >> >> >> >> I'm currently plan to migrate from Flume 0.9 to Flume-ng 1.6, but I'm >> having troubles trying to find a reliable setup for this one. >> >> >> >> My sink is a 3 nodes Kafka cluster. I must avoid *to lose events in case >> the main sink is down*, broken or unreachable for a while. >> >> >> >> In Flume 0.9, I use a memory channel with the *store on failure *feature, >> which starts writing events on the local disk in case the target sink is >> not available. >> >> >> >> In Flume-ng 1.6 the same behaviour would be accomplished by setting up a >> *Spillable >> memory channel, *but the problem with this solution is written in the >> end of the channel's description: "*This channel is currently >> experimental and not recommended for use in production."* >> >> >> >> In Flume-ng 1.6, it's possible to setup a pool of *Failover sinks*. So, >> I was thinking to hypothetically configure a *File Roll *as Secondary >> sink in case the Primary is down. However, once the Primary sink would be >> back online, the data placed on the Secondary sink (local disk) won't be >> automatically pushed on the Primary one. >> >> >> >> Another option would be setting up a *file channel*: write each event on >> the disk and then sink. Without mentioning that I don't love the idea to >> write/delete each single event continuously on a SSD, this setup is taking >> 90% of CPU. The same exactly configuration but using a memory channel takes >> 3%. >> >> >> >> Other solutions to evaluate ? >> >> >> >> Simone >> >> >> > >
