Hi Yongkun Wang, > > My comment inline. > > On 12/07/20 19:38, "alo alt" <[email protected]> wrote: > >> Hi, >> >> On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote: >> >> inline >> >>> Hi, >>> >>> Currently we are using a modified version of flume-0.9.3 in production. >>> >>> I tried release 1.1.0. The basic command is still bin/flume-ng node, >>> not bin/flume-ng agent (by doc). >> >> Did you build Flume from git? In 1.1.x the notation was changed into >> agent, node is deprecated. > > In the 1.1.0 package it is still bin/flume-ng node. But it has been > changed in the trunk. >
Yes, thats right. >> >> >>> No document for HDFS sink, I need to read the source code to config it. >>> HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can >>> rotate files by count of events, but the last hadoop file cannot be >>> closed. >> >> http://archive.cloudera.com/cdh4/cdh/4/flume-ng/ > > Very nice user guide in cdh4! > But it is not easy to find the user guide from flume apache home page. I > finally find this one: > https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/doc > s/FlumeUserGuide.html > much better! >> >>> >>> Rotating the hadoop files by time interval does not work. It seems that >>> problem is here: >>> A mistake by timing 1000 in BucketWriter: >>> >>> lastProcessTime = System.currentTimeMillis() * 1000; >>> >>> So the lines in 'if' will never get executed: >>> >>> if ((rollInterval > 0) >>> >>> && (rollInterval < (System.currentTimeMillis() - >>> lastProcessTime) / 1000)) >>> >>> doRotate = true; >>> >>> But the code has been changed aggressively in later versions, so it is >>> no longer a bug in the trunk. >>> >>> I tested NG trunk (1.3 snapshot). The code is aggressively changed >>> compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or >>> 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation >>> works perfectly, last hadoop file closed correctly). >>> >> >> Flume looks into the exported classpath and determine the installed >> hadoop version to use it. > > Yes. I tested flume with several hadoop versions. It seems that flume may > not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not > supported in 0.20.2 (writer.syncFs() is implemented by calling sync(), > maybe using writer.sync() in flume is ok?) > Good point, please file a jira about. I don't think that's a blocker for 1.2.x, but could be one for the next release. Thanks. cheers, Alex
