Hi Alex, I created a ticket in JIRA as suggested. https://issues.apache.org/jira/browse/FLUME-1391
Cheers, Kun On 12/07/23 15:33, "alo alt" <[email protected]> wrote: >Hi Yongkun Wang, > >> >> My comment inline. >> >> On 12/07/20 19:38, "alo alt" <[email protected]> wrote: >> >>> Hi, >>> >>> On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote: >>> >>> inline >>> >>>> Hi, >>>> >>>> Currently we are using a modified version of flume-0.9.3 in >>>>production. >>>> >>>> I tried release 1.1.0. The basic command is still bin/flume-ng node, >>>> not bin/flume-ng agent (by doc). >>> >>> Did you build Flume from git? In 1.1.x the notation was changed into >>> agent, node is deprecated. >> >> In the 1.1.0 package it is still bin/flume-ng node. But it has been >> changed in the trunk. >> > >Yes, thats right. > > > >>> >>> >>>> No document for HDFS sink, I need to read the source code to config >>>>it. >>>> HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can >>>> rotate files by count of events, but the last hadoop file cannot be >>>> closed. >>> >>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng/ >> >> Very nice user guide in cdh4! >> But it is not easy to find the user guide from flume apache home page. I >> finally find this one: >> >>https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/d >>oc >> s/FlumeUserGuide.html >> > >much better! > > >>> >>>> >>>> Rotating the hadoop files by time interval does not work. It seems >>>>that >>>> problem is here: >>>> A mistake by timing 1000 in BucketWriter: >>>> >>>> lastProcessTime = System.currentTimeMillis() * 1000; >>>> >>>> So the lines in 'if' will never get executed: >>>> >>>> if ((rollInterval > 0) >>>> >>>> && (rollInterval < (System.currentTimeMillis() - >>>> lastProcessTime) / 1000)) >>>> >>>> doRotate = true; >>>> >>>> But the code has been changed aggressively in later versions, so it is >>>> no longer a bug in the trunk. >>>> >>>> I tested NG trunk (1.3 snapshot). The code is aggressively changed >>>> compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or >>>> 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based >>>>rotation >>>> works perfectly, last hadoop file closed correctly). >>>> >>> >>> Flume looks into the exported classpath and determine the installed >>> hadoop version to use it. >> >> Yes. I tested flume with several hadoop versions. It seems that flume >>may >> not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not >> supported in 0.20.2 (writer.syncFs() is implemented by calling sync(), >> maybe using writer.sync() in flume is ok?) >> > >Good point, please file a jira about. I don't think that's a blocker for >1.2.x, but could be one for the next release. Thanks. > >cheers, > Alex
