checked, nothing is written in hdfs.
On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <[email protected]> wrote: > That just means source has done its work and is waiting for more data to > read. Did you check hdfs to see if all data has been written? > > > > On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi < > [email protected]> wrote: > >> Hi Mohit and sharminder, >> >> Thanks for reply, after I called with -n tier, source is not directory >> error came, I changed the source to /tmp/ and hdfs.path to /flume/messages/ >> in config file, and run the command, the INFO i am getting now is "spooling >> directory source runner has shutdown" >> what could be the problem, please help me. >> >> >> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal <[email protected] >> > wrote: >> >>> Replace -n agent with -n tier1 >>> >>> >>> On Sunday, June 15, 2014, kishore alajangi <[email protected]> >>> wrote: >>> >>>> Dear Sharminder, >>>> >>>> Thanks for your reply, yes I am playing with flume, as you suggested i >>>> am using spool directory source, My configuration file looks like >>>> >>>> *tier1.sources = source1 >>>> tier1.channels = channel1 >>>> tier1.sinks = sink1 >>>> >>>> tier1.sources.source1.type = spooldir >>>> tier1.sources.source1.spoolDir = /var/log/messages >>>> tier1.sources.source1.channels = channel1 >>>> tier1.channels.channel1.type = memory >>>> >>>> tier1.sinks.sink1.type = hdfs >>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages >>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile >>>> tier1.sinks.sink1.hdfs.filePrefix = data >>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq >>>> >>>> # Roll based on the block size only >>>> tier1.sinks.sink1.hdfs.rollCount=0 >>>> tier1.sinks.sink1.hdfs.rollInterval=0 >>>> tier1.sinks.sink1.hdfs.rollSize = 120000000 >>>> # seconds to wait before closing the file. >>>> tier1.sinks.sink1.hdfs.idleTimeout = 60 >>>> tier1.sinks.sink1.channel = channel1 >>>> >>>> tier1.channels.channel1.capacity = 100000* >>>> tier1.sources.source1.deserializer.maxLineLength = 32768 >>>> >>>> >>>> >>>> >>>> the command I used is >>>> >>>> >>>> >>>> >>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf >>>> -Dflume.root.logger=DEBUG,console -n agent >>>> >>>> >>>> >>>> >>>> it gives warn after created sources, channels, sinks for tier1 agent is >>>> >>>> >>>> >>>> >>>> no configuration found for this host:agent >>>> >>>> >>>> >>>> >>>> any help? >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <[email protected]> >>>> wrote: >>>> >>>>> >>>>> I want to copy my local data to hdfs using flume in a single machine >>>>>> which isrunning hadoop, How can I do that, please help me. >>>>>> >>>>>> What is this "local data" ? >>>>> >>>>> If it's just files, why not use the hadoop fs copy command instead? If >>>>> you want to play around with flume, take a look at the spool directory >>>>> source or the exec source and you should be able to put something together >>>>> that'll push data through flume to hadoop. >>>>> >>>>> -- >>>>> Sharninder >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Kishore. >>>> >>> >> >> >> -- >> Thanks, >> Kishore. >> > > -- Thanks, Kishore.
