I created the /flume/messages directories, but still nothing is written with flume in those directories. please help me.
On Mon, Jun 16, 2014 at 10:15 AM, kishore alajangi < [email protected]> wrote: > Do I need to create the /flume/messages/ directories? > > > > On Mon, Jun 16, 2014 at 10:14 AM, kishore alajangi < > [email protected]> wrote: > >> checked, nothing is written in hdfs. >> >> >> On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <[email protected]> >> wrote: >> >>> That just means source has done its work and is waiting for more data to >>> read. Did you check hdfs to see if all data has been written? >>> >>> >>> >>> On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi < >>> [email protected]> wrote: >>> >>>> Hi Mohit and sharminder, >>>> >>>> Thanks for reply, after I called with -n tier, source is not directory >>>> error came, I changed the source to /tmp/ and hdfs.path to /flume/messages/ >>>> in config file, and run the command, the INFO i am getting now is "spooling >>>> directory source runner has shutdown" >>>> what could be the problem, please help me. >>>> >>>> >>>> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal < >>>> [email protected]> wrote: >>>> >>>>> Replace -n agent with -n tier1 >>>>> >>>>> >>>>> On Sunday, June 15, 2014, kishore alajangi <[email protected]> >>>>> wrote: >>>>> >>>>>> Dear Sharminder, >>>>>> >>>>>> Thanks for your reply, yes I am playing with flume, as you suggested >>>>>> i am using spool directory source, My configuration file looks like >>>>>> >>>>>> *tier1.sources = source1 >>>>>> tier1.channels = channel1 >>>>>> tier1.sinks = sink1 >>>>>> >>>>>> tier1.sources.source1.type = spooldir >>>>>> tier1.sources.source1.spoolDir = /var/log/messages >>>>>> tier1.sources.source1.channels = channel1 >>>>>> tier1.channels.channel1.type = memory >>>>>> >>>>>> tier1.sinks.sink1.type = hdfs >>>>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages >>>>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile >>>>>> tier1.sinks.sink1.hdfs.filePrefix = data >>>>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq >>>>>> >>>>>> # Roll based on the block size only >>>>>> tier1.sinks.sink1.hdfs.rollCount=0 >>>>>> tier1.sinks.sink1.hdfs.rollInterval=0 >>>>>> tier1.sinks.sink1.hdfs.rollSize = 120000000 >>>>>> # seconds to wait before closing the file. >>>>>> tier1.sinks.sink1.hdfs.idleTimeout = 60 >>>>>> tier1.sinks.sink1.channel = channel1 >>>>>> >>>>>> tier1.channels.channel1.capacity = 100000* >>>>>> tier1.sources.source1.deserializer.maxLineLength = 32768 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> the command I used is >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf >>>>>> -Dflume.root.logger=DEBUG,console -n agent >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> it gives warn after created sources, channels, sinks for tier1 agent is >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> no configuration found for this host:agent >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> any help? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> I want to copy my local data to hdfs using flume in a single machine >>>>>>>> which isrunning hadoop, How can I do that, please help me. >>>>>>>> >>>>>>>> What is this "local data" ? >>>>>>> >>>>>>> If it's just files, why not use the hadoop fs copy command instead? >>>>>>> If you want to play around with flume, take a look at the spool >>>>>>> directory >>>>>>> source or the exec source and you should be able to put something >>>>>>> together >>>>>>> that'll push data through flume to hadoop. >>>>>>> >>>>>>> -- >>>>>>> Sharninder >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Kishore. >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Kishore. >>>> >>> >>> >> >> >> -- >> Thanks, >> Kishore. >> > > > > -- > Thanks, > Kishore. > -- Thanks, Kishore.
