Hi, This is my fault. At this point FC is dependent on Hadoop. We want to remove it, but it will take some time. This is in https://issues.apache.org/jira/browse/FLUME-1285
The simplest way to solve this is to put the hadoop command in your path before you start flume. Brock On Wed, Aug 29, 2012 at 10:38 AM, Steve Johnson <[email protected]> wrote: > Good morning. I'm trying to test Flume for use for our companies log > collection. These are application transaction logs (not syslogs) so I need > the agents channel data to be fully recoverable. I have done some initial > setup testing using the memory channel for my initial tests and it's been > going well. However, I'm unable to get the FileChannel operable. Below is > the exception I'm getting. I'm currently running this on OSX 10.7.4 (Darwin > Kernel Version 11.4.0). I've installed using the flume-ng 1.2.0 binary. > > I searched the flume mail lists and didn't find anything, but I was able to > find something on google here, but I didn't see any more updates on the > issues since late June: > http://www.mail-archive.com/[email protected]/msg07419.html > > I've pasted the exception below I'm getting at startup. Also, here is my > confiugration, thanks in advance for any help: > testagent.sources = netcatSource > testagent.channels = fileChannel > testagent.sinks = fileSink > > # For each one of the sources, the type is defined > testagent.sources.netcatSource.type = netcat > > # The channel can be defined as follows. > testagent.sources.netcatSource.channels = fileChannel > > # Each sink's type must be defined > testagent.sinks.fileSink.type = FILE_ROLL > > #Specify the channel the sink should use > testagent.sinks.fileSink.channel = fileChannel > > # Each channel's type is defined. > testagent.channels.fileChannel.type = file > > # Other config values specific to each type of channel(sink or source) > # can be defined as well > # In this case, it specifies the capacity of the memory channel > testagent.sources.netcatSource.bind = 0.0.0.0 > testagent.sources.netcatSource.port = 6166 > testagent.sources.netcatSource.max-line-length = 32768 > testagent.channels.fileChannel.transactionCapacity = 1024000 > testagent.channels.fileChannel.maxFileSize = 10240000000 > testagent.channels.fileChannel.checkpointDir = > /opt/dotomi/flume/file_channel/checkpoint > testagent.channels.fileChannel.dataDirs = > /opt/dotomi/flume/file_channel/data > testagent.channels.fileChannel.capacity = 512000000 > testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir > testagent.sinks.fileSink.sink.rollInterval = 0 > > > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting > new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner: { > source:o > rg.apache.flume.source.NetcatSource@3c50507 }} > sinkRunners:{fileSink=SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107f counterGroup:{ > nam > e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel { > dataDirs: [/opt/dotomi/flume/file_channel/data] }} } > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting > Channel fileChannel > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Waiting > for channel: fileChannel to start. Sleeping for 500 ms > 2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... > 2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to start > FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] } > - Excep > tion follows. > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.flume.channel.file.Log$Builder.build(Log.java:139) > at > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226) > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 24 more > 2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... > > -- > Steve Johnson > [email protected] -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
