A word of caution here. If these issues are also relevant to the Apache distribution then some form of the discussion should come back here so that it benefits users of the vanilla distribution.
Ralph On May 11, 2012, at 1:17 PM, Mike Percy wrote: > bcc: flume-user@incubator.apache.org > > This issue involves a version of Flume distributed with CDH, so moving to > cdh-u...@cloudera.org which is the proper support channel. > > Thanks, > Mike > On Friday, May 11, 2012 at 1:02 PM, lexs...@gmail.com wrote: > >> Hello, >> >> I am new to Flume. I am trying to use hdfs as a sink. My first >> problem is that flume hangs during initialization. >> Same problem as: >> >> https://issues.apache.org/jira/browse/FLUME-1077 >> >> and >> >> https://issues.apache.org/jira/browse/FLUME-1093 >> >> My version of Hadoop is Cloudera's "cdh3u4", and Flume version is >> "flume-ng-1.1.0-cdh3u4" >> I noticed that Hadoop and Flume have different versions of Guava. >> >> If I remove Flume's Guava jar and use Hadoop's: >> >> <code> >> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1 >> --classpath >> ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar >> </code> >> >> I get this error: >> >> <code> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> com/google/common/util/concurrent/ThreadFactoryBuilder >> at >> org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54) >> at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42) >> at org.apache.flume.node.Application.run(Application.java:135) >> at org.apache.flume.node.Application.main(Application.java:70) >> Caused by: java.lang.ClassNotFoundException: >> com.google.common.util.concurrent.ThreadFactoryBuilder >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 4 more >> </code> >> >> If I put back Flume's Guava jar: >> >> <code> >> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1 >> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar >> </code> >> >> I get this error: >> >> <code> >> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" >> java.lang.NoClassDefFoundError: >> org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap >> at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217) >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) >> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563) >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67) >> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228) >> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183) >> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57) >> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49) >> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95) >> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117) >> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361) >> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 21 more >> </code> >> >> Has anybody been able to successfully run Flume with a HDFS sink? >> What's the magic combination? >> >> -- >> Alex Soto >> lexs...@gmail.com >> >> - >> Amicus Plato, sed magis amica veritas. >
