Or add the directory which include the libs into the classpath like the flume-libs.
cheers Alex On Aug 1, 2012, at 4:13 PM, Brock Noland <[email protected]> wrote: > export PATH=$PATH:/path/to/hadoop > > This could be done in the flume-env.sh file. > > Brock > > On Wed, Aug 1, 2012 at 9:12 AM, mardan Khan <[email protected]> wrote: > >> Thanks Brock >> >> How to add the hadoop command to path. Can you please give me an example. >> >> Thanks >> >> On Wed, Aug 1, 2012 at 3:07 PM, Brock Noland <[email protected]> wrote: >> >>> Hi, >>> >>> I think this is because you need more than just the hadoop-core.jar file. >>> If you add the "hadoop" command to your path, the flume-ng script should >>> pickup these dependencies automatically. >>> >>> Brock >>> >>> >>> On Tue, Jul 31, 2012 at 12:25 PM, mardan Khan <[email protected]>wrote: >>> >>>> HI, >>>> >>>> I am posting again as I am still struggling for the solution. I have >>>> simple configuration file which upload the data into hadoop but give me >>>> error message: Agent Failed because dependencies were not found. I am using >>>> the following: >>>> >>>> 1). Flume1.2.0 >>>> 2) Hadoop-1.0.3 >>>> 3). Window 7 >>>> 4). Cygwin. >>>> >>>> >>>> Configuration File: >>>> >>>> agent1.sources = source1 >>>> agent1.sinks = sink1 >>>> agent1.channels = channel1 >>>> >>>> agent1.sources.source1.type = netcat >>>> agent1.sources.source1.bind = localhost >>>> agent1.sources.source1.port = 23 >>>> >>>> agent1.sinks.sink1.type = logger >>>> agent1.sinks.sink1.type = hdfs >>>> agent1.sinks.sink1.hdfs.path = >>>> hdfs://localhost:9000/user/cyg_server/flume >>>> agent1.channels.channel1.type = memory >>>> agent1.channels.channel1.capacity = 1000 >>>> agent1.channels.channel1.transactionCapactiy = 100 >>>> >>>> agent1.sources.source1.channels = channel1 >>>> agent1.sinks.sink1.channel = channel1 >>>> >>>> *ERROR MESSAGE* >>>> >>>> mukhtaj@mukhtaj-PC ~/apache-flume >>>> $ bin/flume-ng agent -n agent1 -c conf -f >>>> conf/flume-conf.properties.template >>>> cygpath: can't convert empty path >>>> + /cygdrive/c/java/jdk1.7.0_01/bin/java -Xmx20m -cp >>>> 'C:\cygwin\home\mukhtaj\apac >>>> he-flume\conf;C:\cygwin\home\mukhtaj\apache-flume\lib\*' >>>> -Djava.library.path= or >>>> g.apache.flume.node.Application -n agent1 -f >>>> conf/flume-conf.properties.template >>>> >>>> 2012-07-31 18:17:56,120 (main) [INFO - >>>> org.apache.flume.lifecycle.LifecycleSuper >>>> visor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1 >>>> 2012-07-31 18:17:56,124 (main) [INFO - >>>> org.apache.flume.node.FlumeNode.start(Flu >>>> meNode.java:54)] Flume node starting - agent1 >>>> 2012-07-31 18:17:56,128 (lifecycleSupervisor-1-0) [INFO - >>>> org.apache.flume.node. >>>> >>>> nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] >>>> Node manager starting >>>> 2012-07-31 18:17:56,128 (lifecycleSupervisor-1-1) [INFO - >>>> org.apache.flume.conf. >>>> >>>> file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.j >>>> ava:67)] Configuration provider starting >>>> 2012-07-31 18:17:56,130 (lifecycleSupervisor-1-0) [INFO - >>>> org.apache.flume.lifec >>>> ycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting >>>> lifecycle >>>> supervisor 9 >>>> 2012-07-31 18:17:56,131 (lifecycleSupervisor-1-0) [DEBUG - >>>> org.apache.flume.node >>>> >>>> .nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191) >>>> ] Node manager started >>>> 2012-07-31 18:17:56,132 (lifecycleSupervisor-1-1) [DEBUG - >>>> org.apache.flume.conf >>>> >>>> .file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider. >>>> java:86)] Configuration provider started >>>> 2012-07-31 18:17:56,132 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.file >>>> >>>> .AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigura >>>> tionProvider.java:188)] Checking >>>> file:conf\flume-conf.properties.template for ch >>>> anges >>>> 2012-07-31 18:17:56,134 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.file. >>>> >>>> AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurat >>>> ionProvider.java:195)] Reloading configuration >>>> file:conf\flume-conf.properties.t >>>> emplate >>>> 2012-07-31 18:17:56,140 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.Flume >>>> Configuration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] >>>> Proce >>>> ssing:sink1 >>>> 2012-07-31 18:17:56,141 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] >>>> Crea >>>> ted context for sink1: hdfs.path >>>> 2012-07-31 18:17:56,142 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.Flume >>>> Configuration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] >>>> Proce >>>> ssing:sink1 >>>> 2012-07-31 18:17:56,143 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.Flume >>>> Configuration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] >>>> Proce >>>> ssing:sink1 >>>> 2012-07-31 18:17:56,145 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.Flume >>>> Configuration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] >>>> Added >>>> sinks: sink1 Agent: agent1 >>>> 2012-07-31 18:17:56,146 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] >>>> Starting >>>> validation of configuration for agent: agent1, initial-configuration: >>>> AgentConf >>>> iguration[agent1] >>>> SOURCES: {source1={ parameters:{port=23, channels=channel1, type=netcat, >>>> bind=lo >>>> calhost} }} >>>> CHANNELS: {channel1={ parameters:{transactionCapactiy=100, >>>> capacity=1000, type=m >>>> emory} }} >>>> SINKS: {sink1={ >>>> parameters:{hdfs.path=hdfs://localhost:9000/user/cyg_server/flum >>>> e, type=hdfs, channel=channel1} }} >>>> >>>> 2012-07-31 18:17:56,156 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> >>>> eConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] >>>> Created channel channel1 >>>> 2012-07-31 18:17:56,174 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] >>>> Cr >>>> eating sink: sink1 using HDFS >>>> 2012-07-31 18:17:56,176 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] >>>> Post val >>>> idation configuration for agent1 >>>> AgentConfiguration created without Configuration stubs for which only >>>> basic synt >>>> actical validation was performed[agent1] >>>> SOURCES: {source1={ parameters:{port=23, channels=channel1, type=netcat, >>>> bind=lo >>>> calhost} }} >>>> CHANNELS: {channel1={ parameters:{transactionCapactiy=100, >>>> capacity=1000, type=m >>>> emory} }} >>>> SINKS: {sink1={ >>>> parameters:{hdfs.path=hdfs://localhost:9000/user/cyg_server/flum >>>> e, type=hdfs, channel=channel1} }} >>>> >>>> 2012-07-31 18:17:56,177 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration.validateConfiguration(FlumeConfiguration.java:117)] >>>> Channels:chan >>>> nel1 >>>> >>>> 2012-07-31 18:17:56,177 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks >>>> sink1 >>>> >>>> 2012-07-31 18:17:56,178 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.Flum >>>> eConfiguration.validateConfiguration(FlumeConfiguration.java:119)] >>>> Sources sourc >>>> e1 >>>> >>>> 2012-07-31 18:17:56,178 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.Flume >>>> Configuration.validateConfiguration(FlumeConfiguration.java:122)] >>>> Post-validatio >>>> n flume configuration contains configuration for agents: [agent1] >>>> 2012-07-31 18:17:56,178 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.prope >>>> >>>> rties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurati >>>> onProvider.java:249)] Creating channels >>>> 2012-07-31 18:17:56,179 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.channel.D >>>> efaultChannelFactory.create(DefaultChannelFactory.java:68)] Creating >>>> instance of >>>> channel channel1 type memory >>>> 2012-07-31 18:17:56,238 (conf-file-poller-0) [INFO - >>>> org.apache.flume.instrument >>>> ation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] >>>> Monitoried co >>>> unter group for type: CHANNEL, name: channel1, registered successfully. >>>> 2012-07-31 18:17:56,239 (conf-file-poller-0) [INFO - >>>> org.apache.flume.conf.prope >>>> >>>> rties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurati >>>> onProvider.java:273)] created channel channel1 >>>> 2012-07-31 18:17:56,239 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.source.De >>>> faultSourceFactory.create(DefaultSourceFactory.java:74)] Creating >>>> instance of so >>>> urce source1, type netcat >>>> 2012-07-31 18:17:56,316 (conf-file-poller-0) [INFO - >>>> org.apache.flume.sink.Defau >>>> ltSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of >>>> sink: sin >>>> k1, type: hdfs >>>> 2012-07-31 18:17:56,392 (conf-file-poller-0) [DEBUG - >>>> org.apache.hadoop.conf.Con >>>> figuration.<init>(Configuration.java:227)] java.io.IOException: config() >>>> at >>>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227) >>>> at >>>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use >>>> rGroupInformation.java:184) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(Use >>>> rGroupInformation.java:236) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.j >>>> ava:516) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java >>>> :238) >>>> at >>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> loadSinks(PropertiesFileConfigurationProvider.java:373) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> load(PropertiesFileConfigurationProvider.java:223) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(A >>>> bstractFileConfigurationProvider.java:123) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$3 >>>> 00(AbstractFileConfigurationProvider.java:38) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatc >>>> herRunnable.run(AbstractFileConfigurationProvider.java:202) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47 >>>> 1) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java >>>> :351) >>>> at >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. >>>> java:1110) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor >>>> .java:603) >>>> at java.lang.Thread.run(Thread.java:722) >>>> >>>> 2012-07-31 18:17:56,464 (conf-file-poller-0) [DEBUG - >>>> org.apache.hadoop.security >>>> .Groups.getUserToGroupsMappingService(Groups.java:139)] Creating new >>>> Groups obj >>>> ect >>>> 2012-07-31 18:17:56,508 (conf-file-poller-0) [DEBUG - >>>> org.apache.hadoop.security >>>> .Groups.<init>(Groups.java:59)] Group mapping impl= >>>> org.apache.hadoop.security.Sh >>>> ellBasedUnixGroupsMapping; cacheTimeout=300000 >>>> 2012-07-31 18:17:56,551 (conf-file-poller-0) [DEBUG - >>>> org.apache.hadoop.conf.Con >>>> figuration.<init>(Configuration.java:227)] java.io.IOException: config() >>>> at >>>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227) >>>> at >>>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use >>>> rGroupInformation.java:184) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(Use >>>> rGroupInformation.java:236) >>>> at >>>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79 >>>> ) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI >>>> nformation.java:209) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use >>>> rGroupInformation.java:184) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(Use >>>> rGroupInformation.java:236) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.j >>>> ava:516) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java >>>> :238) >>>> at >>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> loadSinks(PropertiesFileConfigurationProvider.java:373) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> load(PropertiesFileConfigurationProvider.java:223) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(A >>>> bstractFileConfigurationProvider.java:123) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$3 >>>> 00(AbstractFileConfigurationProvider.java:38) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatc >>>> herRunnable.run(AbstractFileConfigurationProvider.java:202) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47 >>>> 1) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java >>>> :351) >>>> at >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. >>>> java:1110) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor >>>> .java:603) >>>> at java.lang.Thread.run(Thread.java:722) >>>> >>>> 2012-07-31 18:17:56,592 (conf-file-poller-0) >>>> Runnable.run(AbstractFileConfigura >>>> tionProvider.java:207)] Failed to start agent because [ERROR - >>>> org.apache.flume.conf.file >>>> .AbstractFileConfigurationProvider$FileWatchuse dependencies were not >>>> foun >>>> d in classpath. Error follows. >>>> java.lang.NoClassDefFoundError: >>>> org/apache/commons/configuration/Configuration >>>> at >>>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMet >>>> ricsSystem.java:37) >>>> at >>>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultM >>>> etricsSystem.java:34) >>>> at >>>> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentati >>>> on.java:51) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI >>>> nformation.java:216) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use >>>> rGroupInformation.java:184) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(Use >>>> rGroupInformation.java:236) >>>> at >>>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79 >>>> ) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI >>>> nformation.java:209) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use >>>> rGroupInformation.java:184) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(Use >>>> rGroupInformation.java:236) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.j >>>> ava:516) >>>> at >>>> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java >>>> :238) >>>> at >>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> loadSinks(PropertiesFileConfigurationProvider.java:373) >>>> at >>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider. >>>> load(PropertiesFileConfigurationProvider.java:223) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(A >>>> bstractFileConfigurationProvider.java:123) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$3 >>>> 00(AbstractFileConfigurationProvider.java:38) >>>> at >>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatc >>>> herRunnable.run(AbstractFileConfigurationProvider.java:202) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47 >>>> 1) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java >>>> :351) >>>> at >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. >>>> run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. >>>> java:1110) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor >>>> .java:603) >>>> at java.lang.Thread.run(Thread.java:722) >>>> Caused by: java.lang.ClassNotFoundException: >>>> org.apache.commons.configuration.Co >>>> nfiguration >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:423) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:356) >>>> ... 26 more >>>> 2012-07-31 18:18:26,601 (conf-file-poller-0) [DEBUG - >>>> org.apache.flume.conf.file >>>> >>>> .AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigura >>>> tionProvider.java:188)] Checking >>>> file:conf\flume-conf.properties.template for ch >>>> anges >>>> >>>> >>>> Please give me any solution if someone have. I dont understand what is >>>> mean by dependencies were not found. >>>> >>>> Thanks >>>> >>>> >>>> >>> >>> >>> -- >>> Apache MRUnit - Unit testing MapReduce - >>> http://incubator.apache.org/mrunit/ >>> >> >> > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/ -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
