The logs that Kamal sent were from the latest code in trunk. What I sent below includes modifications I made to try to determine why the FileChannel just disappeared when it started. Sine the code doesn't normally catch Errors it just percolated all the way up the call stack.
Ralph On Jun 27, 2012, at 6:03 PM, Hari Shreedharan <[email protected]> wrote: > Ralph/Kamal, > > Yes, we have had this issue before. I don't think what Kamal is seeing is the > same as this one though, unless the logs Kamal sent are older than the > current trunk(since that does not show the exception or the error in the > logs). > > Thanks > Hari > > -- > Hari Shreedharan > > > On Wednesday, June 27, 2012 at 5:46 PM, Ralph Goers wrote: > >> I tried this on two machines and got the same problem as Kamal. In looking >> at the stack trace below I noticed that the FileChannel appears nowhere in >> the thread dump. So I added a catch for Error to the FileChannel's start >> method and got the error below. It appears there is a dependency on >> something in Hadoop in the Log class. >> >> 2012-06-27 17:33:06,125 ERROR file.FileChannel: Error encountered starting >> the file channel >> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:615) >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:58) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:197) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> at org.apache.flume.channel.file.Log$Builder.build(Log.java:131) >> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:206) >> at >> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 24 more >> 2012-06-27 17:33:06,129 DEBUG lifecycle.LifecycleSupervisor: Status check >> complete >> 2012-06-27 17:33:06,173 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,224 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,275 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,325 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,376 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,427 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,478 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,529 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,580 DEBUG lifecycle.LifecycleController: Still want one >> of states:[STOP, ERROR] sleeping:50ms >> 2012-06-27 17:33:06,619 INFO nodemanager.DefaultLogicalNodeManager: Starting >> Sink loggerSink >> 2012-06-27 17:33:06,620 DEBUG lifecycle.LifecycleSupervisor: Supervising >> service:SinkRunner: { >> policy:org.apache.flume.sink.DefaultSinkProcessor@21a80a69 counterGroup:{ >> name:null counters:{} } } >> policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@5328f6ee >> desiredState:START >> 2012-06-27 17:33:06,620 INFO nodemanager.DefaultLogicalNodeManager: Starting >> Source seqGenSrc >> 2012-06-27 17:33:06,620 DEBUG lifecycle.LifecycleSupervisor: Supervising >> service:PollableSourceRunner: { >> source:org.apache.flume.source.SequenceGeneratorSource@51b48197 >> counterGroup:{ name:null counters:{} } } >> policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@38b5dac4 >> desiredState:START >> 2012-06-27 17:33:06,620 DEBUG lifecycle.LifecycleSupervisor: checking >> process:SinkRunner: { >> policy:org.apache.flume.sink.DefaultSinkProcessor@21a80a69 counterGroup:{ >> name:null counters:{} } } supervisoree:{ status:{ lastSeen:null >> lastSeenState:null desiredState:START firstSeen:null failures:0 >> discard:false } >> policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@5328f6ee >> } >> 2012-06-27 17:33:06,621 DEBUG lifecycle.LifecycleSupervisor: first time >> seeing SinkRunner: { >> policy:org.apache.flume.sink.DefaultSinkProcessor@21a80a69 counterGroup:{ >> name:null counters:{} } } >> 2012-06-27 17:33:06,621 DEBUG lifecycle.LifecycleSupervisor: Want to >> transition SinkRunner: { >> policy:org.apache.flume.sink.DefaultSinkProcessor@21a80a69 counterGroup:{ >> name:null counters:{} } } from IDLE to START (failures:0) >> 2012-06-27 17:33:06,622 DEBUG lifecycle.LifecycleSupervisor: checking >> process:PollableSourceRunner: { >> source:org.apache.flume.source.SequenceGeneratorSource@51b48197 >> counterGroup:{ name:null counters:{} } } supervisoree:{ status:{ >> lastSeen:null lastSeenState:null desiredState:START firstSeen:null >> failures:0 discard:false } >> policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@38b5dac4 >> } >> 2012-06-27 17:33:06,622 DEBUG lifecycle.LifecycleSupervisor: first time >> seeing PollableSourceRunner: { >> source:org.apache.flume.source.SequenceGeneratorSource@51b48197 >> counterGroup:{ name:null counters:{} } } >> 2012-06-27 17:33:06,622 DEBUG lifecycle.LifecycleSupervisor: Want to >> transition PollableSourceRunner: { >> source:org.apache.flume.source.SequenceGeneratorSource@51b48197 >> counterGroup:{ name:null counters:{} } } from IDLE to START (failures:0) >> 2012-06-27 17:33:06,622 INFO source.SequenceGeneratorSource: Sequence >> generator source starting >> 2012-06-27 17:33:06,622 DEBUG source.SequenceGeneratorSource: Sequence >> generator source started >> 2012-06-27 17:33:06,623 DEBUG lifecycle.LifecycleSupervisor: Status check >> complete >> 2012-06-27 17:33:06,623 DEBUG flume.SinkRunner: Polling sink runner starting >> 2012-06-27 17:33:06,624 DEBUG lifecycle.LifecycleSupervisor: Status check >> complete >> 2012-06-27 17:33:06,624 DEBUG source.PollableSourceRunner: Polling runner >> starting. Source:org.apache.flume.source.SequenceGeneratorSource@51b48197 >> 2012-06-27 17:33:06,625 ERROR flume.SinkRunner: Unable to deliver event. >> Exception follows. >> java.lang.IllegalStateException: Channel closed [channel=fileChannel] >> at com.google.common.base.Preconditions.checkState(Preconditions.java:145) >> at >> org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:239) >> at >> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) >> at org.apache.flume.sink.LoggerSink.process(LoggerSink.java:61) >> at >> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >> at java.lang.Thread.run(Thread.java:680) >> 2012-06-27 17:33:06,626 ERROR source.PollableSourceRunner: Unhandled >> exception, logging and sleeping for 5000ms >> java.lang.IllegalStateException: Channel closed [channel=fileChannel] >> at com.google.common.base.Preconditions.checkState(Preconditions.java:145) >> at >> org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:239) >> at >> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) >> at >> org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:255) >> at >> org.apache.flume.source.SequenceGeneratorSource.process(SequenceGeneratorSource.java:48) >> at >> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:137) >> at java.lang.Thread.run(Thread.java:680) >> >> >> >> >> On Jun 27, 2012, at 11:33 AM, Kamal Bahadur wrote: >> >>> Here is the jstack output: >>> >>> W025572:flume-1.2.0-incubating-SNAPSHOT kbahadur$ jstack 18628 >>> 2012-06-27 11:25:16 >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424 mixed >>> mode): >>> >>> "Attach Listener" daemon prio=9 tid=7fc9b5063800 nid=0x117b33000 waiting on >>> condition [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "lifecycleSupervisor-1-1" prio=5 tid=7fc9b595e800 nid=0x118a09000 waiting >>> on condition [118a08000] >>> java.lang.Thread.State: TIMED_WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <7f9b20ec0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) >>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) >>> at >>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "conf-file-poller-0" prio=5 tid=7fc9b729f000 nid=0x118906000 waiting on >>> condition [118905000] >>> java.lang.Thread.State: TIMED_WAITING (sleeping) >>> at java.lang.Thread.sleep(Native Method) >>> at >>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:116) >>> at >>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:225) >>> at >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) >>> - locked <7f9b21250> (a >>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider) >>> at >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) >>> at >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "lifecycleSupervisor-1-0" prio=5 tid=7fc9b5958000 nid=0x118803000 waiting >>> on condition [118802000] >>> java.lang.Thread.State: TIMED_WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <7f9b20ec0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) >>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) >>> at >>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "lifecycleSupervisor-1-2" prio=5 tid=7fc9b729e000 nid=0x118700000 waiting >>> on condition [1186ff000] >>> java.lang.Thread.State: TIMED_WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <7f9cbad60> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) >>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) >>> at >>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "lifecycleSupervisor-1-1" prio=5 tid=7fc9b729d800 nid=0x1185fd000 waiting >>> for monitor entry [1185fc000] >>> java.lang.Thread.State: BLOCKED (on object monitor) >>> at >>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:207) >>> - waiting to lock <7f9b21250> (a >>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "lifecycleSupervisor-1-0" prio=5 tid=7fc9b729c800 nid=0x1184fa000 waiting >>> on condition [1184f9000] >>> java.lang.Thread.State: TIMED_WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <7f9cbad60> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) >>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) >>> at >>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>> at java.lang.Thread.run(Thread.java:680) >>> >>> "Low Memory Detector" daemon prio=5 tid=7fc9b400c000 nid=0x118045000 >>> runnable [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "C2 CompilerThread1" daemon prio=9 tid=7fc9b400b800 nid=0x117f42000 waiting >>> on condition [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "C2 CompilerThread0" daemon prio=9 tid=7fc9b400a800 nid=0x117e3f000 waiting >>> on condition [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Signal Dispatcher" daemon prio=9 tid=7fc9b400a000 nid=0x117d3c000 runnable >>> [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fc9b4009000 >>> nid=0x117c39000 waiting on condition [00000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Finalizer" daemon prio=8 tid=7fc9b5929000 nid=0x117975000 in Object.wait() >>> [117974000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <7f9d32020> (a java.lang.ref.ReferenceQueue$Lock) >>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) >>> - locked <7f9d32020> (a java.lang.ref.ReferenceQueue$Lock) >>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) >>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >>> >>> "Reference Handler" daemon prio=10 tid=7fc9b5928000 nid=0x117872000 in >>> Object.wait() [117871000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <7f9d3a030> (a java.lang.ref.Reference$Lock) >>> at java.lang.Object.wait(Object.java:485) >>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >>> - locked <7f9d3a030> (a java.lang.ref.Reference$Lock) >>> >>> "main" prio=5 tid=7fc9b5000000 nid=0x10fc7d000 waiting on condition >>> [10fc7c000] >>> java.lang.Thread.State: TIMED_WAITING (sleeping) >>> at java.lang.Thread.sleep(Native Method) >>> at >>> org.apache.flume.lifecycle.LifecycleController.waitForOneOf(LifecycleController.java:73) >>> at >>> org.apache.flume.lifecycle.LifecycleController.waitForOneOf(LifecycleController.java:49) >>> at org.apache.flume.node.Application.run(Application.java:167) >>> at org.apache.flume.node.Application.main(Application.java:70) >>> >>> "VM Thread" prio=9 tid=7fc9b5923800 nid=0x11776f000 runnable >>> >>> "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fc9b5800000 >>> nid=0x112fce000 runnable >>> >>> "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fc9b5801000 >>> nid=0x1130d1000 runnable >>> >>> "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fc9b5801800 >>> nid=0x1131d4000 runnable >>> >>> "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fc9b5802000 >>> nid=0x1132d7000 runnable >>> >>> "Gang worker#4 (Parallel GC Threads)" prio=9 tid=7fc9b5802800 >>> nid=0x1133da000 runnable >>> >>> "Gang worker#5 (Parallel GC Threads)" prio=9 tid=7fc9b5803800 >>> nid=0x1134dd000 runnable >>> >>> "Gang worker#6 (Parallel GC Threads)" prio=9 tid=7fc9b5804000 >>> nid=0x1135e0000 runnable >>> >>> "Gang worker#7 (Parallel GC Threads)" prio=9 tid=7fc9b5804800 >>> nid=0x1136e3000 runnable >>> >>> "Concurrent Mark-Sweep GC Thread" prio=9 tid=7fc9b58e2000 nid=0x1173e9000 >>> runnable >>> "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=7fc9b58e0800 >>> nid=0x1169e3000 runnable >>> >>> "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=7fc9b58e1800 >>> nid=0x116ae6000 runnable >>> >>> "VM Periodic Task Thread" prio=10 tid=7fc9b401d800 nid=0x118148000 waiting >>> on condition >>> >>> "Exception Catcher Thread" prio=10 tid=7fc9b5001000 nid=0x10fdf7000 >>> runnable >>> JNI global references: 1243 >>> >>> >>> On Wed, Jun 27, 2012 at 8:31 AM, Kamal Bahadur <[email protected] >>> (mailto:[email protected])>wrote: >>> >>>> I got latest changes this morning but still same error. >>>> >>>> Here is the SVN log: http://pastebin.com/npNM8GLN >>>> >>>> Here is the debug log: http://pastebin.com/nkWne3Mc >>>> >>>> I did rm -rf ~/.flume before starting flume. >>>> >>>> Here is my config: >>>> >>>> agent.sources = seqGenSrc >>>> agent.channels = memoryChannel >>>> agent.sinks = loggerSink >>>> >>>> agent.sources.seqGenSrc.type = seq >>>> >>>> agent.sources.seqGenSrc.channels = memoryChannel >>>> >>>> agent.sinks.loggerSink.type = logger >>>> >>>> agent.sinks.loggerSink.channel = memoryChannel >>>> >>>> agent.channels.memoryChannel.type = file >>>> >>>> Thanks, >>>> Kamal >>>> >>>> >>>> On Tue, Jun 26, 2012 at 8:37 PM, Mike Percy <[email protected] >>>> (mailto:[email protected])> wrote: >>>> >>>>> Also, add this to your command line: >>>>> >>>>> -Dflume.root.logger=DEBUG,console >>>>> >>>>> this will print DEBUG level info to the console. >>>>> >>>>> Also, try rm -rf ~/.flume before starting flume. >>>>> >>>>> Regards, >>>>> Mike >>>>> >>>>> >>>>> On Tuesday, June 26, 2012 at 8:24 PM, Hari Shreedharan wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Could you send the git log or svn info? This seems to be a bug which >>>>> was fixed. I have not seen this issue since then. >>>>>> >>>>>> Thanks >>>>>> Hari >>>>>> >>>>>> -- >>>>>> Hari Shreedharan >>>>>> >>>>>> >>>>>> On Tuesday, June 26, 2012 at 8:13 PM, Kamal Bahadur wrote: >>>>>> >>>>>>> Thanks Mike and Hari for the response. I am starting fresh so I don't >>>>> have >>>>>>> to worry about the data. Here is what I did: >>>>>>> >>>>>>> 1. Deleted the source code >>>>>>> 2. Downloaded latest code from trunk >>>>>>> 3. Copied flume-conf.properties.template and created >>>>>>> >>>>>> >>>>> >>>>> flume-conf.properties. >>>>>>> 4. Changed the channel type to file. (also commented >>>>>> >>>>> >>>>> memoryChannel.capacity >>>>>>> line) >>>>>>> 5. Modified log4j.properties to log to console >>>>>>> 6. Started using this command: *bin/flume-ng agent -n agent -f >>>>>>> conf/flume-conf.properties -c conf* >>>>>>> 6. Got exact same error as before: >>>>>>> >>>>>>> [kamal@localhost flume]$ bin/flume-ng agent -n agent -f >>>>>>> conf/flume-conf.properties -c conf >>>>>>> + exec /usr/local/jdk/bin/java -Xmx20m -cp >>>>>>> '/home/kamal/flume/flume/conf:/home/kamal/flume/flume/lib/*' >>>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f >>>>>>> conf/flume-conf.properties >>>>>>> 2012-06-26 20:12:01,556 (main) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] >>>>>>> Starting lifecycle supervisor 1 >>>>>>> 2012-06-26 20:12:01,559 (main) [INFO - >>>>>>> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node >>>>>>> starting - agent >>>>>>> 2012-06-26 20:12:01,562 (lifecycleSupervisor-1-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] >>>>>>> Node manager starting >>>>>>> 2012-06-26 20:12:01,564 (lifecycleSupervisor-1-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] >>>>>>> Starting lifecycle supervisor 10 >>>>>>> 2012-06-26 20:12:01,562 (lifecycleSupervisor-1-2) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] >>>>>>> Configuration provider starting >>>>>>> 2012-06-26 20:12:01,564 (lifecycleSupervisor-1-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] >>>>>>> Node manager started >>>>>>> 2012-06-26 20:12:01,567 (lifecycleSupervisor-1-2) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] >>>>>>> Configuration provider started >>>>>>> 2012-06-26 20:12:01,567 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] >>>>>>> Checking file:conf/flume-conf.properties for changes >>>>>>> 2012-06-26 20:12:01,567 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] >>>>>>> Reloading configuration file:conf/flume-conf.properties >>>>>>> 2012-06-26 20:12:01,578 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] >>>>>>> Processing:loggerSink >>>>>>> 2012-06-26 20:12:01,578 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] >>>>>>> Created context for loggerSink: channel >>>>>>> 2012-06-26 20:12:01,578 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] >>>>>>> Added sinks: loggerSink Agent: agent >>>>>>> 2012-06-26 20:12:01,578 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] >>>>>>> Processing:loggerSink >>>>>>> 2012-06-26 20:12:01,579 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] >>>>>>> Starting validation of configuration for agent: agent, >>>>>>> initial-configuration: AgentConfiguration[agent] >>>>>>> SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }} >>>>>>> CHANNELS: {memoryChannel={ parameters:{type=file} }} >>>>>>> SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} >>>>>>> >>>>>> >>>>> >>>>> }} >>>>>>> >>>>>>> 2012-06-26 20:12:01,582 (conf-file-poller-0) [DEBUG - >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] >>>>>>> Created channel memoryChannel >>>>>>> 2012-06-26 20:12:01,597 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] >>>>>>> Creating sink: loggerSink using LOGGER >>>>>>> 2012-06-26 20:12:01,600 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] >>>>>>> Post validation configuration for agent >>>>>>> AgentConfiguration created without Configuration stubs for which only >>>>>>> >>>>>> >>>>> >>>>> basic >>>>>>> syntactical validation was performed[agent] >>>>>>> SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }} >>>>>>> CHANNELS: {memoryChannel={ parameters:{type=file} }} >>>>>>> AgentConfiguration created with Configuration stubs for which full >>>>>>> validation was performed[agent] >>>>>>> SINKS: {loggerSink=ComponentConfiguration[loggerSink] >>>>>>> CONFIG: >>>>>>> CHANNEL:memoryChannel >>>>>>> } >>>>>>> >>>>>>> 2012-06-26 20:12:01,600 (conf-file-poller-0) [DEBUG - >>>>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] >>>>>>> Channels:memoryChannel >>>>>>> >>>>>>> 2012-06-26 20:12:01,601 (conf-file-poller-0) [DEBUG - >>>>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] >>>>>>> Sinks loggerSink >>>>>>> >>>>>>> 2012-06-26 20:12:01,601 (conf-file-poller-0) [DEBUG - >>>>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] >>>>>>> Sources seqGenSrc >>>>>>> >>>>>>> 2012-06-26 20:12:01,601 (conf-file-poller-0) [INFO - >>>>> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] >>>>>>> Post-validation flume configuration contains configuration for agents: >>>>>>> [agent] >>>>>>> 2012-06-26 20:12:01,601 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] >>>>>>> Creating channels >>>>>>> 2012-06-26 20:12:01,601 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:68)] >>>>>>> Creating instance of channel memoryChannel type file >>>>>>> 2012-06-26 20:12:01,608 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] >>>>>>> created channel memoryChannel >>>>>>> 2012-06-26 20:12:01,608 (conf-file-poller-0) [DEBUG - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:74)] >>>>>>> Creating instance of source seqGenSrc, type seq >>>>>>> 2012-06-26 20:12:01,620 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] >>>>>>> Creating instance of sink loggerSink typelogger >>>>>>> 2012-06-26 20:12:01,624 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:93)] >>>>>>> Starting new configuration:{ >>>>>> >>>>> >>>>> sourceRunners:{seqGenSrc=PollableSourceRunner: >>>>>>> { >>>>>> >>>>> >>>>> source:org.apache.flume.source.SequenceGeneratorSource@7c4c51counterGroup >>>>> :{ >>>>>>> name:null counters:{} } }} >>>>>>> sinkRunners:{loggerSink=SinkRunner: { >>>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@60765a16counterGroup:{ >>>>>>> name:null counters:{} } }} >>>>>>> >>>>>> >>>>> >>>>> channels:{memoryChannel=org.apache.flume.channel.file.FileChannel@7a148bd3 >>>>> } >>>>>>> } >>>>>>> 2012-06-26 20:12:01,625 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:100)] >>>>>>> Starting Channel memoryChannel >>>>>>> 2012-06-26 20:12:01,625 (lifecycleSupervisor-1-1) [INFO - >>>>>>> org.apache.flume.channel.file.FileChannel.start(FileChannel.java:197)] >>>>>>> Starting FileChannel with dataDir >>>>>>> >>>>>> >>>>> >>>>> [/home/kamal/.flume/file-channel/data] >>>>>>> 2012-06-26 20:12:01,628 (conf-file-poller-0) [INFO - >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:02,128 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:02,629 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:03,129 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:03,630 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:04,131 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:04,631 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:05,132 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:05,633 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:06,133 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> 2012-06-26 20:12:06,634 (conf-file-poller-0) [INFO - >>>>>>> >>>>>> >>>>> >>>>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] >>>>>>> Waiting for channel: memoryChannel to start. Sleeping for 500 ms >>>>>>> >>>>>>> On Tue, Jun 26, 2012 at 6:57 PM, Mike Percy <[email protected] >>>>>>> (mailto:[email protected])(mailto: >>>>> [email protected] (mailto:[email protected]))> wrote: >>>>>>> >>>>>>>> Try enabling debug logging. Also yes, use latest trunk. And make >>>>> sure the >>>>>>>> directory contents for the checkpoint and data dirs are empty… the >>>>>>> >>>>>> >>>>> >>>>> format >>>>>>>> changed on trunk last night. So if you need to keep data around >>>>>>> >>>>>> >>>>> >>>>> that is in >>>>>>>> existing file channels make sure to start up without a source and >>>>>>> >>>>>> >>>>> >>>>> drain the >>>>>>>> channel to another agent or file before shutting it down and >>>>>>> >>>>>> >>>>> >>>>> deleting the >>>>>>>> file channel data directories. >>>>>>>> >>>>>>>> Side note: not sure putting the file channel data dirs on /tmp is >>>>> safe for >>>>>>>> production, but it depends on your setup. Some operating systems >>>>>>> >>>>>> >>>>> >>>>> purge /tmp >>>>>>>> on startup. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mike >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, June 26, 2012 at 6:30 PM, Hari Shreedharan wrote: >>>>>>>> >>>>>>>>> Is this on trunk? If you rebase on trunk, you should be ok. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Hari >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Hari Shreedharan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, June 26, 2012 at 6:08 PM, Kamal Bahadur wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am using the latest code from the HEAD. Following is my >>>>>>>> >>>>>>>> configuration: I >>>>>>>>>> am using file channel and it looks like flume-ng gets stuck >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> during >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> startup >>>>>>>>>> of file-channel. No errors found in the log as usual. >>>>>>>>>> >>>>>>>>>> I have already checked for permissions on the directories and >>>>> they >>>>>>>> >>>>>>>> look ok. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Kamal >>>>>>>>>> >>>>>>>>>> agent.sources = avroSrc >>>>>>>>>> agent.channels = fileChannel >>>>>>>>>> agent.sinks = cassandraSink >>>>>>>>>> >>>>>>>>>> agent.sources.avroSrc.type = avro >>>>>>>>>> agent.sources.avroSrc.bind = 0.0.0.0 >>>>>>>>>> agent.sources.avroSrc.port = 8800 >>>>>>>>>> >>>>>>>>>> agent.sources.avroSrc.channels = fileChannel >>>>>>>>>> >>>>>>>>>> agent.sinks.cassandraSink.type = >>>>> com.glsink.cassandra.CassandraSink >>>>>>>>>> agent.sinks.cassandraSink.nodes = >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> host1:9160,host2:9160,host3:9160 >>>>>>>>>> agent.sinks.cassandraSink.cluster = GL CLuster >>>>>>>>>> agent.sinks.cassandraSink.keyspace = Global_Logging >>>>>>>>>> agent.sinks.cassandraSink.columnFamily = Audit_Log >>>>>>>>>> agent.sinks.cassandraSink.loadBalancingPolicy = >>>>>>>>>> me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy >>>>>>>>>> agent.sinks.cassandraSink.username = USERDEV >>>>>>>>>> agent.sinks.cassandraSink.password = pwd152== >>>>>>>>>> agent.sinks.cassandraSink.batch-size = 100 >>>>>>>>>> >>>>>>>>>> agent.sinks.cassandraSink.channel = fileChannel >>>>>>>>>> >>>>>>>>>> agent.channels.fileChannel.type = file >>>>>>>>>> >>>>>>>>>> agent.channels.fileChannel.checkpointDir = /tmp/checkpoint >>>>>>>>>> agent.channels.fileChannel.dataDirs = /tmp/data >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Debug message: >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,588 DEBUG conf.FlumeConfiguration: Starting >>>>>>>> >>>>>>>> validation >>>>>>>>>> of configuration for agent: agent, initial-configuration: >>>>>>>>>> AgentConfiguration[agent] >>>>>>>>>> SOURCES: {avroSrc={ parameters:{port=8800, channels=fileChannel, >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> type=avro, >>>>>>>>>> bind=0.0.0.0} }} >>>>>>>>>> CHANNELS: {fileChannel={ >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> parameters:{checkpointDir=/tmp/checkpoint, >>>>>>>>>> dataDirs=/tmp/data, type=FILE} }} >>>>>>>>>> SINKS: {cassandraSink={ >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> parameters:{loadBalancingPolicy=me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy, >>>>>>>>>> username=USERDEV , batch-size=100, nodes= < >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> http://10.136.239.61:9160 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <removed>, >>>>>>>>>> keyspace=Global_Logging, cluster=GL CLuster, >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> columnFamily=Audit_Log, >>>>>>>>>> type=com.glsink.cassandra.CassandraSink, channel=fileChannel, >>>>>>>>>> password=pwd152==} }} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,594 DEBUG conf.FlumeConfiguration: Created >>>>> channel >>>>>>>>>> fileChannel >>>>>>>>>> 2012-06-25 16:12:18,608 DEBUG conf.FlumeConfiguration: Creating >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> sink: >>>>>>>>>> cassandraSink using OTHER >>>>>>>>>> 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Post >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> validation >>>>>>>>>> configuration for agent >>>>>>>>>> AgentConfiguration created without Configuration stubs for >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> which only >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> basic >>>>>>>>>> syntactical validation was performed[agent] >>>>>>>>>> SOURCES: {avroSrc={ parameters:{port=8800, channels=fileChannel, >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> type=avro, >>>>>>>>>> bind=0.0.0.0} }} >>>>>>>>>> CHANNELS: {fileChannel={ >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> parameters:{checkpointDir=/tmp/checkpoint, >>>>>>>>>> dataDirs=/tmp/data, type=FILE} }} >>>>>>>>>> SINKS: {cassandraSink={ >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> parameters:{loadBalancingPolicy=me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy, >>>>>>>>>> username=USERDEV , batch-size=100, nodes= < >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> http://10.136.239.61:9160 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <removed>, >>>>>>>>>> keyspace=Global_Logging, cluster=GL CLuster, >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> columnFamily=Audit_Log, >>>>>>>>>> type=com.glsink.cassandra.CassandraSink, channel=fileChannel, >>>>>>>>>> password=pwd152==} }} >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: >>>>>>>> >>>>>>>> Channels:fileChannel >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Sinks >>>>>>>> >>>>>>>> cassandraSink >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Sources >>>>> avroSrc >>>>>>>>>> >>>>>>>>>> 2012-06-25 16:12:18,610 INFO conf.FlumeConfiguration: >>>>> Post-validation >>>>>>>> >>>>>>>> flume >>>>>>>>>> configuration contains configuration for agents: [agent] >>>>>>>>>> 2012-06-25 16:12:18,610 INFO >>>>>>>>>> properties.PropertiesFileConfigurationProvider: Creating >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> channels >>>>>>>>>> 2012-06-25 16:12:18,610 DEBUG channel.DefaultChannelFactory: >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> Creating >>>>>>>>>> instance of channel fileChannel type FILE >>>>>>>>>> 2012-06-25 16:12:18,624 INFO >>>>>>>>>> properties.PropertiesFileConfigurationProvider: created channel >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> fileChannel >>>>>>>>>> 2012-06-25 16:12:18,624 DEBUG source.DefaultSourceFactory: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> Creating >>>>>>>>>> instance of source avroSrc, type avro >>>>>>>>>> 2012-06-25 16:12:18,646 INFO sink.DefaultSinkFactory: Creating >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> instance of >>>>>>>>>> sink cassandraSink >>>>>>>>>> typecom.intuit.ifs.globallogging.glsink.cassandra.CassandraSink >>>>>>>>>> 2012-06-25 16:12:18,647 DEBUG sink.DefaultSinkFactory: Sink type >>>>>>>>>> com.intuit.ifs.globallogging.glsink.cassandra.CassandraSink is >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> a custom >>>>>>>>>> type >>>>>>>>>> 2012-06-25 16:12:18,665 DEBUG cassandra.CassandraSink: >>>>>>>>>> 2012-06-25 16:12:18,665 INFO cassandra.CassandraSink: >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> Configiring >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Cassandra >>>>>>>>>> sink >>>>>>>>>> 2012-06-25 16:12:18,665 INFO cassandra.CassandraSink: Configured >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Cassandra >>>>>>>>>> sink >>>>>>>>>> 2012-06-25 16:12:18,667 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>>> Starting new configuration:{ >>>>>>>>>> sourceRunners:{avroSrc=EventDrivenSourceRunner: { >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> source:AvroSource: { >>>>>>>>>> bindAddress:0.0.0.0 port:8800 } }} >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> sinkRunners:{cassandraSink=SinkRunner: { >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@7a148bd3counterGroup:{ >>>>>>>>>> name:null counters:{} } }} >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> channels:{fileChannel=org.apache.flume.channel.file.FileChannel@5ae80842} >>>>>>>> } >>>>>>>>>> 2012-06-25 16:12:18,667 INFO >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>>> Starting Channel fileChannel >>>>>>>>>> 2012-06-25 16:12:18,667 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>>> 2012-06-25 16:12:18,668 INFO file.FileChannel: Starting >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> FileChannel >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> with >>>>>>>>>> dataDir [/tmp/data] >>>>>>>>>> 2012-06-25 16:12:19,170 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>>> 2012-06-25 16:12:19,672 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>>> 2012-06-25 16:12:20,175 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>>> 2012-06-25 16:12:20,678 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>>> 2012-06-25 16:12:21,180 INFO >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> nodemanager.DefaultLogicalNodeManager: >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Waiting >>>>>>>>>> for channel: fileChannel to start. Sleeping for 500 ms >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> > >
