Hi Alex, Thanks for your reply. My comment inline.
On 12/07/20 19:38, "alo alt" <[email protected]> wrote: >Hi, > >On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote: > >inline > >> Hi, >> >> Currently we are using a modified version of flume-0.9.3 in production. >> >> I tried release 1.1.0. The basic command is still bin/flume-ng node, >>not bin/flume-ng agent (by doc). > >Did you build Flume from git? In 1.1.x the notation was changed into >agent, node is deprecated. In the 1.1.0 package it is still bin/flume-ng node. But it has been changed in the trunk. > > >> No document for HDFS sink, I need to read the source code to config it. >>HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can >>rotate files by count of events, but the last hadoop file cannot be >>closed. > >http://archive.cloudera.com/cdh4/cdh/4/flume-ng/ Very nice user guide in cdh4! But it is not easy to find the user guide from flume apache home page. I finally find this one: https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/doc s/FlumeUserGuide.html > >> >> Rotating the hadoop files by time interval does not work. It seems that >>problem is here: >> A mistake by timing 1000 in BucketWriter: >> >> lastProcessTime = System.currentTimeMillis() * 1000; >> >> So the lines in 'if' will never get executed: >> >> if ((rollInterval > 0) >> >> && (rollInterval < (System.currentTimeMillis() - >>lastProcessTime) / 1000)) >> >> doRotate = true; >> >> But the code has been changed aggressively in later versions, so it is >>no longer a bug in the trunk. >> >> I tested NG trunk (1.3 snapshot). The code is aggressively changed >>compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or >>1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation >>works perfectly, last hadoop file closed correctly). >> > >Flume looks into the exported classpath and determine the installed >hadoop version to use it. Yes. I tested flume with several hadoop versions. It seems that flume may not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not supported in 0.20.2 (writer.syncFs() is implemented by calling sync(), maybe using writer.sync() in flume is ok?) > > >> Will keep an eye on NG and wait for later releases. > > >cheers, > Alex > > >-- >Alexander Alten-Lorenz >http://mapredit.blogspot.com >German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > Cheers, Yongkun Wang (Kun) >
