Ah, yes, I remember that patch. It basically copied the Writables interfaces/classes I think. Yeah, copying those would be a nice workaround (I think those dont have any implementation anyway).
Hari -- Hari Shreedharan On Wednesday, April 10, 2013 at 1:44 PM, Brock Noland wrote: > I think we just need to copy 4-5 classes over from hadoop. Someone uploaded a > patch I just haven't had time to work with it. > > > On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <[email protected] > (mailto:[email protected])> wrote: > > Yes. We will need to remove the format upgrade code and the old format code > > from the file channel as well. I think it should be possible to do it, but > > I suspect it is just that no one had the time to look at this as yet. > > > > -- > > Hari Shreedharan > > > > > > On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote: > > > > > Hari, > > > i recall some discussion around the desire to get rid of this > > > dependency. i guess it means removing that piece of code that supports > > > the format upgrade ? are there any plans to do that ? > > > -roshan > > > > > > > > > > > > On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez > > > <[email protected] > > > (mailto:[email protected])> wrote: > > > > I did not realize I could use pre-built binaries on Windows. Thank you, > > > > I used this to grab the hadoop-common.jar from our existing Hadoop > > > > cluster and there are no startup errors now. > > > > > > > > Thanks to you, Brock and Hari. > > > > > > > > -Paul > > > > > > > > From: Israel Ekpo [mailto:[email protected]] > > > > Sent: Wednesday, April 10, 2013 12:07 PM > > > > To: [email protected] (mailto:[email protected]) > > > > Subject: Re: FileChannel on Windows > > > > > > > > Paul, > > > > > > > > If Flume 1.3.1 is what you are looking for, you don't have to build it > > > > from source. > > > > > > > > You can just download it directly from the site. Its already released. > > > > > > > > Paul Chavez <[email protected] > > > > (mailto:[email protected])> wrote: > > > > > > > > Hello, > > > > > > > > I've been using flume on Windows to send logs to an avro source for > > > > awhile now, using the AvroClient class. I'm now trying to get a full > > > > agent running to hopefully leverage the spoolDir and httpSource > > > > functionality. Part of my requirements is a persistent channel so I'm > > > > attempting to get a config running the utilizes the fileChannel. > > > > > > > > I followed the instructions at > > > > http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to > > > > build flume, same as I did for the AvroClient. However it seems I need > > > > some HDFS libraries. Can anyone point me to instructions for building > > > > the JAR(s) I need to get this working? > > > > > > > > Currently I'm running into the following exception: > > > > 10 Apr 2013 11:50:29,317 INFO [lifecycleSupervisor-1-0] > > > > (org.apache.flume.channel.file.FileChannel.start:285) - Starting > > > > FileChannel fc_Default { dataDirs: [c:\flume_data\log] }... > > > > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] > > > > (org.apache.flume.channel.file.FileChannel.start:314) - Failed to > > > > start the file channel [channel=fc_Default] > > > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > > > > at java.lang.ClassLoader.defineClass1(Native Method) > > > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > > > at > > > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > > > at org.apache.flume.channel.file.Log$Builder.build(Log.java:212) > > > > at > > > > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302) > > > > at > > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) > > > > at > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > > > at > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > Caused by: java.lang.ClassNotFoundException: > > > > org.apache.hadoop.io.Writable > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > > > ... 24 more > > > > 10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] > > > > (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238) > > > > - Unable to start FileChannel fc_Default { dataDirs: > > > > [c:\flume_data\log] } - Exception follows. > > > > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable > > > > at java.lang.ClassLoader.defineClass1(Native Method) > > > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > > > at > > > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > > > at org.apache.flume.channel.file.Log$Builder.build(Log.java:212) > > > > at > > > > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302) > > > > at > > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) > > > > at > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > > > at > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > Caused by: java.lang.ClassNotFoundException: > > > > org.apache.hadoop.io.Writable > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > > > ... 24 more > > > > 10 Apr 2013 11:50:29,317 INFO [lifecycleSupervisor-1-0] > > > > (org.apache.flume.channel.file.FileChannel.stop:329) - Stopping > > > > FileChannel fc_Default { dataDirs: [c:\flume_data\log] }... > > > > > > > > > > > > > > > > Thank you, > > > > Paul Chavez > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
