Followed the following links 1) https://ccp.cloudera.com/display/CDH4B2/CDH4+Installation#CDH4Installation-AddingaDebianRepository
2) https://ccp.cloudera.com/display/CDH4B2/Flume+Installation cheers On Tue, Jul 24, 2012 at 3:55 AM, Will McQueen <[email protected]> wrote: > Hi Mardan, > > How did you install Flume in SLES? Are you installing Flume straight from > Apache, or from a distro? > > Cheers, > Will > > > On Mon, Jul 23, 2012 at 4:46 PM, mardan Khan <[email protected]> wrote: > >> Hi, >> >> I am still getting missing class error. >> >> I have specify the full path. My flume is installed in usr/lib/flume-ng >> and my configuration file is :/usr/lib/flume-ng/conf/flume.conf. >> >> you have mentioned that -c option with command, sorry i dont know exactly >> how I can use, I tried but give me the error message. >> >> I am apply the following command: >> >> $: /usr/bin/flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume.conf >> >> >> Could you please write me the exact command to override this problem.ch >> >> The configuration file exactly same you have posted, just change the >> hdfs path and location of log file. >> >> >> For remind the error message as: >> >> >> 12/07/24 00:33:51 ERROR channel.ChannelProcessor: Builder class not >> found. Exception follows. >> >> java.lang.ClassNotFoundException: >> org.apache.flume.interceptor.HostInterceptor$Builder >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> at java.lang.Class.forName0(Native Method) >> >> I am really straggling to run one command successful >> >> Thanks >> >> >> >> >> >> >> >> On Mon, Jul 23, 2012 at 9:09 AM, Mohammad Tariq <[email protected]>wrote: >> >>> Hi mardan, >>> >>> You need to use the -c option with tour command to specify the >>> directory where your configuration file is kept. Just look at the the >>> other thread of yours. >>> >>> Regards, >>> Mohammad Tariq >>> >>> >>> On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <[email protected]> >>> wrote: >>> > Dear Mohammad Tariq, >>> > >>> > Many thanks for your valuable information. >>> > >>> > For the testing purpose , i have installed flume on SuSE Linux system. >>> when >>> > i have type the command $/etc/init.d/flume-ng-agent start. Then i have >>> > receive message that Starting Flume NG agent daemon (flume-ng-agent): >>> > >>> > I think this mean my flume agent is working properly. I have the >>> following >>> > changes in configuration file according to your example. The >>> configuration >>> > file as: >>> > >>> > >>> > >>> > agent.sources = seqGenSrc >>> > agent.channels = memoryChannel >>> > agent.sinks = loggerSink >>> > >>> > # For each one of the sources, the type is defined >>> > agent.sources.seqGenSrc.type = seq >>> > >>> > # The channel can be defined as follows. >>> > agent.sources.seqGenSrc.channels = memoryChannel >>> > >>> > # Each sink's type must be defined >>> > agent.sinks.loggerSink.type = logger >>> > >>> > #Specify the channel the sink should use >>> > agent.sinks.loggerSink.channel = memoryChannel >>> > >>> > # Each channel's type is defined. >>> > agent.channels.memoryChannel.type = memory >>> > >>> > # Other config values specific to each type of channel(sink or source) >>> > # can be defined as well >>> > # In this case, it specifies the capacity of the memory channel >>> > agent.channels.memoryChannel.capacity = 100 >>> > >>> > >>> > agent1.sources = tail >>> > agent1.channels = MemoryChannel-2 >>> > agent1.sinks = HDFS >>> > >>> > agent1.sources.tail.type = exec >>> > agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log >>> > >>> > agent1.sources.tail.channels = MemoryChannel-2 >>> > >>> > agent1.sources.tail.interceptors = hostint >>> > agent1.sources.tail.interceptors.hostint.type = >>> > org.apache.flume.interceptor.HostInterceptor$Builder >>> > agent1.sources.tail.interceptors.hostint.preserverExisting = true >>> > >>> > agent1.sources.tail.interceptors.hostint.useIP = true >>> > >>> > agent1.sinks.HDFS.channel = MemoryChannel-2 >>> > agent1.channels.MemoryChannel-2.type = memory >>> > agent1.sinks.HDFS.type =hdfs >>> > agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/ >>> > >>> > agent1.sinks.HDFS.hdfs.file.Type = DataStream >>> > agent1.sinks.HDFS.hdfs.writeFormat = Text >>> > >>> > >>> > >>> > when i have type the following command >>> > >>> > $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf >>> > >>> > >>> > >>> > I got the following warring / error messages >>> > >>> > Warning: No configuration directory set! Use --conf <dir> to override. >>> > Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS >>> access >>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath >>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from >>> classpath >>> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from >>> classpath >>> > Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar >>> from >>> > classpath >>> > Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE >>> access >>> > Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from >>> classpath >>> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from >>> classpath >>> > Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from >>> > classpath >>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath >>> > Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from >>> classpath >>> > Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from >>> classpath >>> > + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp >>> > >>> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera. >>> > >>> > >>> ................................................................................ >>> > >>> > >>> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting >>> lifecycle >>> > supervisor 1 >>> > 12/07/23 05:41:29 INFO node.FlumeNode: Flume node starting - agent1 >>> > 12/07/23 05:41:29 INFO nodemanager.DefaultLogicalNodeManager: Node >>> manager >>> > starting >>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >>> > Configuration provider starting >>> > 12/07/23 05:41:29 INFO lifecycle.LifecycleSupervisor: Starting >>> lifecycle >>> > supervisor 10 >>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >>> > Reloading configuration file:/etc/flume-ng/conf/flume.conf >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: loggerSink >>> > Agent: agent >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Added sinks: HDFS >>> Agent: >>> > agent1 >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:HDFS >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Processing:loggerSink >>> > 12/07/23 05:41:29 INFO conf.FlumeConfiguration: Post-validation flume >>> > configuration contains configuration for agents: [agent, agent1] >>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >>> > Creating channels >>> > 12/07/23 05:41:29 INFO properties.PropertiesFileConfigurationProvider: >>> > created channel MemoryChannel-2 >>> > 12/07/23 05:41:29 ERROR channel.ChannelProcessor: Builder class not >>> found. >>> > Exception follows. >>> > java.lang.ClassNotFoundException: >>> > org.apache.flume.interceptor.HostInterceptor$Builder >>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>> > at java.lang.Class.forName0(Native Method) >>> > at java.lang.Class.forName(Class.java:169) >>> > at >>> > >>> org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:103) >>> > at >>> > >>> org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:79) >>> > at >>> org.apache.flume.conf.Configurables.configure(Configurables.java:41) >>> > at >>> > >>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:337) >>> > at >>> > >>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222) >>> > at >>> > >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) >>> > at >>> > >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) >>> > at >>> > >>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) >>> > at >>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>> > at >>> > >>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>> > at >>> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>> > at >>> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >>> > at >>> > >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> > at java.lang.Thread.run(Thread.java:662) >>> > >>> > >>> > >>> > Could you please let me know why give me this message / class missing. >>> > >>> > Many thanks >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Sun, Jul 22, 2012 at 10:12 PM, Mohammad Tariq <[email protected]> >>> wrote: >>> >> >>> >> Hello Mardan, >>> >> >>> >> In order to aggregate data into your Hadoop cluster you need >>> >> to set up a Flume agent first. In order to do that you have to write a >>> >> config file having desired properties. An example file would be >>> >> somewhat like this : >>> >> >>> >> agent1.sources = tail >>> >> agent1.channels = MemoryChannel-2 >>> >> agent1.sinks = HDFS >>> >> >>> >> agent1.sources.tail.type = exec >>> >> agent1.sources.tail.command = tail -F /var/log/apache2/access.log >>> >> agent1.sources.tail.channels = MemoryChannel-2 >>> >> >>> >> agent1.sources.tail.interceptors = hostint >>> >> agent1.sources.tail.interceptors.hostint.type = >>> >> org.apache.flume.interceptor.HostInterceptor$Builder >>> >> agent1.sources.tail.interceptors.hostint.preserveExisting = true >>> >> agent1.sources.tail.interceptors.hostint.useIP = true >>> >> >>> >> agent1.sinks.HDFS.channel = MemoryChannel-2 >>> >> agent1.sinks.HDFS.type = hdfs >>> >> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host} >>> >> agent1.sinks.HDFS.hdfs.file.Type = DataStream >>> >> agent1.sinks.HDFS.hdfs.writeFormat = Text >>> >> >>> >> agent1.channels.MemoryChannel-2.type = memory >>> >> >>> >> You can visit this link as the starting point, if you want - >>> >> >>> http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html >>> >> >>> >> And, it is quite possible to run Flume-1.x o windows. Here is a great >>> >> post by Alex on how to do that - >>> >> http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html >>> >> >>> >> Hope it helps. >>> >> >>> >> Regards, >>> >> Mohammad Tariq >>> >> >>> >> >>> >> On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <[email protected]> >>> wrote: >>> >> > Yeah, my cluster is always running. But i dont know how to setup the >>> >> > flume >>> >> > that directly stream the data to hadoop. I have must install the >>> flume >>> >> > agent >>> >> > on window machine. As per my study the flume version-0.9.4 agent can >>> >> > install >>> >> > on window machine. Can we install flume version 1.x on window >>> machine? >>> >> > If any one have done, please let me guide. >>> >> > >>> >> > >>> >> > >>> >> > Many thanks >>> >> > >>> >> > >>> >> > >>> >> > On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <[email protected] >>> > >>> >> > wrote: >>> >> >> >>> >> >> NameNode and DataNode must be running if we need to write anything >>> to >>> >> >> the >>> >> >> Hdfs. >>> >> >> >>> >> >> Regards, >>> >> >> Mohammad Tariq >>> >> >> >>> >> >> >>> >> >> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson < >>> [email protected]> >>> >> >> wrote: >>> >> >> > You can have flume write to HDFS: however, do you have your >>> hadoop >>> >> >> > cluster running all the time? >>> >> > >>> >> > >>> > >>> > >>> >> >> >
