Thanks, easy enough solution, this worked for me. On Wed, Aug 29, 2012 at 10:42 AM, Brock Noland <[email protected]> wrote:
> Hi, > > This is my fault. At this point FC is dependent on Hadoop. We want to > remove it, but it will take some time. This is in > https://issues.apache.org/jira/browse/FLUME-1285 > > The simplest way to solve this is to put the hadoop command in your > path before you start flume. > > Brock > > On Wed, Aug 29, 2012 at 10:38 AM, Steve Johnson <[email protected]> > wrote: > > Good morning. I'm trying to test Flume for use for our companies log > > collection. These are application transaction logs (not syslogs) so I > need > > the agents channel data to be fully recoverable. I have done some > initial > > setup testing using the memory channel for my initial tests and it's been > > going well. However, I'm unable to get the FileChannel operable. Below > is > > the exception I'm getting. I'm currently running this on OSX 10.7.4 > (Darwin > > Kernel Version 11.4.0). I've installed using the flume-ng 1.2.0 binary. > > > > I searched the flume mail lists and didn't find anything, but I was able > to > > find something on google here, but I didn't see any more updates on the > > issues since late June: > > http://www.mail-archive.com/[email protected]/msg07419.html > > > > I've pasted the exception below I'm getting at startup. Also, here is my > > confiugration, thanks in advance for any help: > > testagent.sources = netcatSource > > testagent.channels = fileChannel > > testagent.sinks = fileSink > > > > # For each one of the sources, the type is defined > > testagent.sources.netcatSource.type = netcat > > > > # The channel can be defined as follows. > > testagent.sources.netcatSource.channels = fileChannel > > > > # Each sink's type must be defined > > testagent.sinks.fileSink.type = FILE_ROLL > > > > #Specify the channel the sink should use > > testagent.sinks.fileSink.channel = fileChannel > > > > # Each channel's type is defined. > > testagent.channels.fileChannel.type = file > > > > # Other config values specific to each type of channel(sink or source) > > # can be defined as well > > # In this case, it specifies the capacity of the memory channel > > testagent.sources.netcatSource.bind = 0.0.0.0 > > testagent.sources.netcatSource.port = 6166 > > testagent.sources.netcatSource.max-line-length = 32768 > > testagent.channels.fileChannel.transactionCapacity = 1024000 > > testagent.channels.fileChannel.maxFileSize = 10240000000 > > testagent.channels.fileChannel.checkpointDir = > > /opt/dotomi/flume/file_channel/checkpoint > > testagent.channels.fileChannel.dataDirs = > > /opt/dotomi/flume/file_channel/data > > testagent.channels.fileChannel.capacity = 512000000 > > testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir > > testagent.sinks.fileSink.sink.rollInterval = 0 > > > > > > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: > Starting > > new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner: > { > > source:o > > rg.apache.flume.source.NetcatSource@3c50507 }} > > sinkRunners:{fileSink=SinkRunner: { > > policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107fcounterGroup:{ > > nam > > e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel { > > dataDirs: [/opt/dotomi/flume/file_channel/data] }} } > > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: > Starting > > Channel fileChannel > > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: > Waiting > > for channel: fileChannel to start. Sleeping for 500 ms > > 2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel > > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... > > 2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to > start > > FileChannel fileChannel { dataDirs: > [/opt/dotomi/flume/file_channel/data] } > > - Excep > > tion follows. > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > at > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > at org.apache.flume.channel.file.Log$Builder.build(Log.java:139) > > at > > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226) > > at > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:680) > > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.Writable > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > ... 24 more > > 2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel > > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }... > > > > -- > > Steve Johnson > > [email protected] > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ > -- Steve Johnson [email protected]
