Thank you so much for the valuable reply Harsh..I'll give it a try with 205
Regards, Mohammad Tariq On Mon, Jun 11, 2012 at 12:53 PM, Harsh J <[email protected]> wrote: > Hi Mohammad, > > The 0.20.203 version of Hadoop does not support sync() operations, > which is required for sinks that flush periodically. You need to be at > least on Apache Hadoop 0.20.205/Any CDH3 version for sync() support on > 0.20.x/1.x. > > On Sun, Jun 10, 2012 at 5:35 AM, Mohammad Tariq <[email protected]> wrote: >> Hello list, >> >> I am trying to collect apache server logs into the Hdfs using >> Flume-NG, but getting this error - >> >> mohammad@ubuntu:~/flume-1.2.0-incubating-SNAPSHOT$ bin/flume-ng agent >> -n agent1 -f conf/agent1.conf >> Warning: No configuration directory set! Use --conf <dir> to override. >> Info: Including Hadoop libraries found via >> (/home/mohammad/hadoop-0.20.203.0/bin/hadoop) for HDFS access >> Info: Excluding >> /home/mohammad/hadoop-0.20.203.0/bin/../lib/slf4j-api-1.4.3.jar from >> classpath >> Info: Excluding >> /home/mohammad/hadoop-0.20.203.0/bin/../lib/slf4j-log4j12-1.4.3.jar >> from classpath >> Info: Including HBASE libraries found via >> (/home/mohammad/hbase-0.90.4/bin/hbase) for HBASE access >> Info: Excluding /home/mohammad/hbase-0.90.4/lib/slf4j-api-1.5.8.jar >> from classpath >> Info: Excluding >> /home/mohammad/hbase-0.90.4/lib/slf4j-log4j12-1.5.8.jar from classpath >> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp >> '/home/mohammad/flume-1.2.0-incubating-SNAPSHOT/lib/*:/home/mohammad/hadoop-0.20.203.0/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/mohammad/hadoop-0.20.203.0/bin/..:/home/mohammad/hadoop-0.20.203.0/bin/../hadoop-core-0.20.203.0.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/aspectjrt-1.6.5.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/aspectjtools-1.6.5.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-beanutils-1.7.0.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-cli-1.2.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-codec-1.4.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-collections-3.2.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-configuration-1.6.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-daemon-1.0.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-digester-1.8.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-el-1.0.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-httpclient-3.0.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-lang-2.4.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-logging-1.1.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-logging-api-1.0.4.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-math-2.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/commons-net-1.4.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/core-3.1.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/hsqldb-1.8.0.10.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jackson-core-asl-1.0.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jackson-mapper-asl-1.0.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jasper-compiler-5.5.12.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jasper-runtime-5.5.12.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jets3t-0.6.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jetty-6.1.26.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jetty-util-6.1.26.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jsch-0.1.42.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/junit-4.5.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/kfs-0.2.2.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/log4j-1.2.15.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/mockito-all-1.8.5.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/oro-2.0.8.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/servlet-api-2.5-20081211.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/xmlenc-0.52.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jsp-2.1/jsp-2.1.jar:/home/mohammad/hadoop-0.20.203.0/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/home/mohammad/hbase-0.90.4/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/mohammad/hbase-0.90.4:/home/mohammad/hbase-0.90.4/hbase-0.90.4.jar:/home/mohammad/hbase-0.90.4/hbase-0.90.4-tests.jar:/home/mohammad/hbase-0.90.4/lib/activation-1.1.jar:/home/mohammad/hbase-0.90.4/lib/asm-3.1.jar:/home/mohammad/hbase-0.90.4/lib/avro-1.3.3.jar:/home/mohammad/hbase-0.90.4/lib/commons-cli-1.2.jar:/home/mohammad/hbase-0.90.4/lib/commons-codec-1.4.jar:/home/mohammad/hbase-0.90.4/lib/commons-configuration-1.6.jar:/home/mohammad/hbase-0.90.4/lib/commons-el-1.0.jar:/home/mohammad/hbase-0.90.4/lib/commons-httpclient-3.1.jar:/home/mohammad/hbase-0.90.4/lib/commons-lang-2.5.jar:/home/mohammad/hbase-0.90.4/lib/commons-logging-1.1.1.jar:/home/mohammad/hbase-0.90.4/lib/commons-net-1.4.1.jar:/home/mohammad/hbase-0.90.4/lib/core-3.1.1.jar:/home/mohammad/hbase-0.90.4/lib/guava-r06.jar:/home/mohammad/hbase-0.90.4/lib/hadoop-core-0.20.203.0.jar:/home/mohammad/hbase-0.90.4/lib/hadoop-core-0.20-append-r1056497.jar:/home/mohammad/hbase-0.90.4/lib/jackson-core-asl-1.5.5.jar:/home/mohammad/hbase-0.90.4/lib/jackson-jaxrs-1.5.5.jar:/home/mohammad/hbase-0.90.4/lib/jackson-mapper-asl-1.4.2.jar:/home/mohammad/hbase-0.90.4/lib/jackson-xc-1.5.5.jar:/home/mohammad/hbase-0.90.4/lib/jasper-compiler-5.5.23.jar:/home/mohammad/hbase-0.90.4/lib/jasper-runtime-5.5.23.jar:/home/mohammad/hbase-0.90.4/lib/jaxb-api-2.1.jar:/home/mohammad/hbase-0.90.4/lib/jaxb-impl-2.1.12.jar:/home/mohammad/hbase-0.90.4/lib/jersey-core-1.4.jar:/home/mohammad/hbase-0.90.4/lib/jersey-json-1.4.jar:/home/mohammad/hbase-0.90.4/lib/jersey-server-1.4.jar:/home/mohammad/hbase-0.90.4/lib/jettison-1.1.jar:/home/mohammad/hbase-0.90.4/lib/jetty-6.1.26.jar:/home/mohammad/hbase-0.90.4/lib/jetty-util-6.1.26.jar:/home/mohammad/hbase-0.90.4/lib/jruby-complete-1.6.0.jar:/home/mohammad/hbase-0.90.4/lib/jsp-2.1-6.1.14.jar:/home/mohammad/hbase-0.90.4/lib/jsp-api-2.1-6.1.14.jar:/home/mohammad/hbase-0.90.4/lib/jsr311-api-1.1.1.jar:/home/mohammad/hbase-0.90.4/lib/log4j-1.2.16.jar:/home/mohammad/hbase-0.90.4/lib/protobuf-java-2.3.0.jar:/home/mohammad/hbase-0.90.4/lib/servlet-api-2.5-6.1.14.jar:/home/mohammad/hbase-0.90.4/lib/stax-api-1.0.1.jar:/home/mohammad/hbase-0.90.4/lib/thrift-0.2.0.jar:/home/mohammad/hbase-0.90.4/lib/xmlenc-0.52.jar:/home/mohammad/hbase-0.90.4/lib/zookeeper-3.3.2.jar:/home/mohammad/hbase-0.90.4/conf' >> -Djava.library.path=:/home/mohammad/hadoop-0.20.203.0/bin/../lib/native/Linux-amd64-64:/usr/lib/jvm/java-6-sun-1.6.0.30/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.30/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.30/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> org.apache.flume.node.Application -n agent1 -f conf/agent1.conf >> 12/06/10 05:23:15 INFO lifecycle.LifecycleSupervisor: Starting >> lifecycle supervisor 1 >> 12/06/10 05:23:15 INFO node.FlumeNode: Flume node starting - agent1 >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Node >> manager starting >> 12/06/10 05:23:15 INFO properties.PropertiesFileConfigurationProvider: >> Configuration provider starting >> 12/06/10 05:23:15 INFO lifecycle.LifecycleSupervisor: Starting >> lifecycle supervisor 10 >> 12/06/10 05:23:15 INFO properties.PropertiesFileConfigurationProvider: >> Reloading configuration file:conf/agent1.conf >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Processing:HDFS >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Processing:HDFS >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Processing:HDFS >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Processing:HDFS >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: >> agent1 >> 12/06/10 05:23:15 INFO conf.FlumeConfiguration: Post-validation flume >> configuration contains configuration for agents: [agent1] >> 12/06/10 05:23:15 INFO properties.PropertiesFileConfigurationProvider: >> Creating channels >> 12/06/10 05:23:15 INFO properties.PropertiesFileConfigurationProvider: >> created channel MemoryChannel-2 >> 12/06/10 05:23:15 INFO sink.DefaultSinkFactory: Creating instance of >> sink HDFS typehdfs >> 12/06/10 05:23:15 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Starting >> new configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: { >> source:org.apache.flume.source.ExecSource@8f0c85e }} >> sinkRunners:{HDFS=SinkRunner: { >> policy:org.apache.flume.sink.DefaultSinkProcessor@77f297e7 >> counterGroup:{ name:null counters:{} } }} >> channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@57d2fc36} >> } >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Starting >> Channel MemoryChannel-2 >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Waiting >> for channel: MemoryChannel-2 to start. Sleeping for 500 ms >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink >> HDFS >> 12/06/10 05:23:15 INFO nodemanager.DefaultLogicalNodeManager: Starting >> Source tail >> 12/06/10 05:23:15 INFO source.ExecSource: Exec source starting with >> command:tail -f /var/log/apache2/access.log >> 12/06/10 05:23:16 INFO hdfs.BucketWriter: Creating >> hdfs://localhost:9000/flume/FlumeData.1339285995904.tmp >> 12/06/10 05:23:16 ERROR hdfs.HDFSEventSink: process failed >> java.lang.NoSuchMethodError: >> org.apache.hadoop.io.SequenceFile$Writer.syncFs()V >> at >> org.apache.flume.sink.hdfs.HDFSSequenceFile.sync(HDFSSequenceFile.java:77) >> at >> org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:276) >> at >> org.apache.flume.sink.hdfs.BucketWriter.access$500(BucketWriter.java:46) >> at >> org.apache.flume.sink.hdfs.BucketWriter$4.run(BucketWriter.java:265) >> at >> org.apache.flume.sink.hdfs.BucketWriter$4.run(BucketWriter.java:262) >> at >> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:120) >> at >> org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:262) >> at >> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:322) >> at >> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:671) >> at >> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:668) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" >> java.lang.NoSuchMethodError: >> org.apache.hadoop.io.SequenceFile$Writer.syncFs()V >> at >> org.apache.flume.sink.hdfs.HDFSSequenceFile.sync(HDFSSequenceFile.java:77) >> at >> org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:276) >> at >> org.apache.flume.sink.hdfs.BucketWriter.access$500(BucketWriter.java:46) >> at >> org.apache.flume.sink.hdfs.BucketWriter$4.run(BucketWriter.java:265) >> at >> org.apache.flume.sink.hdfs.BucketWriter$4.run(BucketWriter.java:262) >> at >> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:120) >> at >> org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:262) >> at >> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:322) >> at >> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:671) >> at >> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:668) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> 12/06/10 05:23:46 INFO hdfs.BucketWriter: Renaming >> hdfs://localhost:9000/flume/FlumeData.1339285995904.tmp to >> hdfs://localhost:9000/flume/FlumeData.1339285995904 >> >> I have written following configuration for my agent - >> >> agent1.sources = tail >> agent1.channels = MemoryChannel-2 >> agent1.sinks = HDFS >> >> agent1.sources.tail.type = exec >> agent1.sources.tail.command = tail -f /var/log/apache2/access.log >> >> agent1.sources.tail.channels = MemoryChannel-2 >> agent1.sinks.HDFS.channel = MemoryChannel-2 >> >> agent1.sinks.HDFS.type = hdfs >> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume >> agent1.sinks.HDFS.hdfs.file.Type = SequenceFile >> >> agent1.channels.MemoryChannel-2.type = memory >> >> Need help..Many thanks. >> >> Regards, >> Mohammad Tariq > > > > -- > Harsh J
