Good morning. I'm trying to test Flume for use for our companies log collection. These are application transaction logs (not syslogs) so I need the agents channel data to be fully recoverable. I have done some initial setup testing using the memory channel for my initial tests and it's been going well. However, I'm unable to get the FileChannel operable. Below is the exception I'm getting. I'm currently running this on OSX 10.7.4 (Darwin Kernel Version 11.4.0). I've installed using the flume-ng 1.2.0 binary.
I searched the flume mail lists and didn't find anything, but I was able to find something on google here, but I didn't see any more updates on the issues since late June: http://www.mail-archive.com/[email protected]/msg07419.html I've pasted the exception below I'm getting at startup. Also, here is my confiugration, thanks in advance for any help: testagent.sources = netcatSource testagent.channels = fileChannel testagent.sinks = fileSink # For each one of the sources, the type is defined testagent.sources.netcatSource.type = netcat # The channel can be defined as follows. testagent.sources.netcatSource.channels = fileChannel # Each sink's type must be defined testagent.sinks.fileSink.type = FILE_ROLL #Specify the channel the sink should use testagent.sinks.fileSink.channel = fileChannel # Each channel's type is defined. testagent.channels.fileChannel.type = file # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel testagent.sources.netcatSource.bind = 0.0.0.0 testagent.sources.netcatSource.port = 6166 testagent.sources.netcatSource.max-line-length = 32768 testagent.channels.fileChannel.transactionCapacity = 1024000 testagent.channels.fileChannel.maxFileSize = 10240000000 testagent.channels.fileChannel.checkpointDir = /opt/dotomi/flume/file_channel/checkpoint testagent.channels.fileChannel.dataDirs = /opt/dotomi/flume/file_channel/data testagent.channels.fileChannel.capacity = 512000000 testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir testagent.sinks.fileSink.sink.rollInterval = 0 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner: { source:o rg.apache.flume.source.NetcatSource@3c50507 }} sinkRunners:{fileSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107f counterGroup:{ nam e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }} } 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel fileChannel 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Waiting for channel: fileChannel to start. Sleeping for 500 ms 2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... 2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to start FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] } - Excep tion follows. java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.flume.channel.file.Log$Builder.build(Log.java:139) at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 24 more 2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... -- Steve Johnson [email protected]
