Hi Mardan, How did you install Flume in SLES? Are you installing Flume straight from Apache, or from a distro?
Cheers, Will On Mon, Jul 23, 2012 at 4:46 PM, mardan Khan <[email protected]> wrote: > Hi, > > I am still getting missing class error. > > I have specify the full path. My flume is installed in usr/lib/flume-ng > and my configuration file is :/usr/lib/flume-ng/conf/flume.conf. > > you have mentioned that -c option with command, sorry i dont know exactly > how I can use, I tried but give me the error message. > > I am apply the following command: > > $: /usr/bin/flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume.conf > > > Could you please write me the exact command to override this problem.ch > > The configuration file exactly same you have posted, just change the hdfs > path and location of log file. > > > For remind the error message as: > > > 12/07/24 00:33:51 ERROR channel.ChannelProcessor: Builder class not found. > Exception follows. > > java.lang.ClassNotFoundException: > org.apache.flume.interceptor.HostInterceptor$Builder > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at java.lang.Class.forName0(Native Method) > > I am really straggling to run one command successful > > Thanks > > > > > > > > On Mon, Jul 23, 2012 at 9:09 AM, Mohammad Tariq <[email protected]>wrote: > >> Hi mardan, >> >> You need to use the -c option with tour command to specify the >> directory where your configuration file is kept. Just look at the the >> other thread of yours. >> >> Regards, >> Mohammad Tariq >> >> >> On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <[email protected]> >> wrote: >> > Dear Mohammad Tariq, >> > >> > Many thanks for your valuable information. >> > >> > For the testing purpose , i have installed flume on SuSE Linux system. >> when >> > i have type the command $/etc/init.d/flume-ng-agent start. Then i have >> > receive message that Starting Flume NG agent daemon (flume-ng-agent): >> > >> > I think this mean my flume agent is working properly. I have the >> following >> > changes in configuration file according to your example. The >> configuration >> > file as: >> > >> > >> > >> > agent.sources = seqGenSrc >> > agent.channels = memoryChannel >> > agent.sinks = loggerSink >> > >> > # For each one of the sources, the type is defined >> > agent.sources.seqGenSrc.type = seq >> > >> > # The channel can be defined as follows. >> > agent.sources.seqGenSrc.channels = memoryChannel >> > >> > # Each sink's type must be defined >> > agent.sinks.loggerSink.type = logger >> > >> > #Specify the channel the sink should use >> > agent.sinks.loggerSink.channel = memoryChannel >> > >> > # Each channel's type is defined. >> > agent.channels.memoryChannel.type = memory >> > >> > # Other config values specific to each type of channel(sink or source) >> > # can be defined as well >> > # In this case, it specifies the capacity of the memory channel >> > agent.channels.memoryChannel.capacity = 100 >> > >> > >> > agent1.sources = tail >> > agent1.channels = MemoryChannel-2 >> > agent1.sinks = HDFS >> > >> > agent1.sources.tail.type = exec >> > agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log >> > >> > agent1.sources.tail.channels = MemoryChannel-2 >> > >> > agent1.sources.tail.interceptors = hostint >> > agent1.sources.tail.interceptors.hostint.type = >> > org.apache.flume.interceptor.HostInterceptor$Builder >> > agent1.sources.tail.interceptors.hostint.preserverExisting = true >> > >> > agent1.sources.tail.interceptors.hostint.useIP = true >> > >> > agent1.sinks.HDFS.channel = MemoryChannel-2 >> > agent1.channels.MemoryChannel-2.type = memory >> > agent1.sinks.HDFS.type =hdfs >> > agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/ >> > >> > agent1.sinks.HDFS.hdfs.file.Type = DataStream >> > agent1.sinks.HDFS.hdfs.writeFormat = Text >> > >> > >> > >> > when i have type the following command >> > >> > $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf >> > >> > >> > >> > I got the following warring / error messages >> > >> > Warning: No configuration directory set! Use --conf <dir> to override. >> > Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS >> access >> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath >> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from >> classpath >> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from >> classpath >> > Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar >> from >> > classpath >> > Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE >> access >> > Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from >> classpath >> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from >> classpath >> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from >> > classpath >> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath >> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from >> classpath >> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from >> classpath >> > + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp >> > >> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera. >> > >> > >> ................................................................................ >> > >> > >> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle >> > supervisor 1 >> > 12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1 >> > 12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node >> manager >> > starting >> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >> > Configuration provider starting >> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting lifecycle >> > supervisor 10 >> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >> > Reloading configuration file:/etc/flume-ng/conf/flume.conf >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink >> > Agent: agent >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: >> > agent1 >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink >> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume >> > configuration contains configuration for agents: [agent, agent1] >> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >> > Creating channels >> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >> > created channel MemoryChannel-2 >> > 12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not >> found. >> > Exception follows. >> > java.lang.ClassNotFoundException: >> > org.apache.flume.interceptor.HostInterceptor$Builder >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> > at java.lang.Class.forName0(Native Method) >> > at java.lang.Class.forName(Class.java:169) >> > at >> > >> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103) >> > at >> > >> org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79) >> > at >> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >> > at >> > >> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337) >> > at >> > >> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) >> > at >> > >> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> > at >> > >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >> > at >> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > >> > >> > Could you please let me know why give me this message / class missing. >> > >> > Many thanks >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <[email protected]> >> wrote: >> >> >> >> Hello Mardan, >> >> >> >> In order to aggregate data into your Hadoop cluster you need >> >> to set up a Flume agent first. In order to do that you have to write a >> >> config file having desired properties. An example file would be >> >> somewhat like this : >> >> >> >> agent1.sources = tail >> >> agent1.channels = MemoryChannel-2 >> >> agent1.sinks = HDFS >> >> >> >> agent1.sources.tail.type = exec >> >> agent1.sources.tail.command = tail -F /var/log/apache2/access.log >> >> agent1.sources.tail.channels = MemoryChannel-2 >> >> >> >> agent1.sources.tail.interceptors = hostint >> >> agent1.sources.tail.interceptors.hostint.type = >> >> org.apache.flume.interceptor.HostInterceptor$Builder >> >> agent1.sources.tail.interceptors.hostint.preserveExisting = true >> >> agent1.sources.tail.interceptors.hostint.useIP = true >> >> >> >> agent1.sinks.HDFS.channel = MemoryChannel-2 >> >> agent1.sinks.HDFS.type = hdfs >> >> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host} >> >> agent1.sinks.HDFS.hdfs.file.Type = DataStream >> >> agent1.sinks.HDFS.hdfs.writeFormat = Text >> >> >> >> agent1.channels.MemoryChannel-2.type = memory >> >> >> >> You can visit this link as the starting point, if you want - >> >> >> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html >> >> >> >> And, it is quite possible to run Flume-1.x o windows. Here is a great >> >> post by Alex on how to do that - >> >> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html >> >> >> >> Hope it helps. >> >> >> >> Regards, >> >> Mohammad Tariq >> >> >> >> >> >> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <[email protected]> >> wrote: >> >> > Yeah, my cluster is always running. But i dont know how to setup the >> >> > flume >> >> > that directly stream the data to hadoop. I have must install the >> flume >> >> > agent >> >> > on window machine. As per my study the flume version-0.9.4 agent can >> >> > install >> >> > on window machine. Can we install flume version 1.x on window >> machine? >> >> > If any one have done, please let me guide. >> >> > >> >> > >> >> > >> >> > Many thanks >> >> > >> >> > >> >> > >> >> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <[email protected]> >> >> > wrote: >> >> >> >> >> >> NameNode and DataNode must be running if we need to write anything >> to >> >> >> the >> >> >> Hdfs. >> >> >> >> >> >> Regards, >> >> >> Mohammad Tariq >> >> >> >> >> >> >> >> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson < >> [email protected]> >> >> >> wrote: >> >> >> > You can have flume write to HDFS: however, do you have your hadoop >> >> >> > cluster running all the time? >> >> > >> >> > >> > >> > >> > >
