no luck Needham.
On Mon, Jun 16, 2014 at 11:45 AM, Needham, Guy < [email protected]> wrote: > Hi Kishore, > > the issue I think is > > > > > > *tier1.sinks.sink1.hdfs.rollInterval=0 tier1.sinks.sink1.hdfs.rollSize = > 120000000 # seconds to wait before closing the file. > tier1.sinks.sink1.hdfs.idleTimeout = 60 * > > Can you try getting rid of the idleTimeout and changing rollInterval to > 30, see if that helps? > > Regards, > > Guy Needham | Data Discovery > Virgin Media | Enterprise Data, Design & Management > Bartley Wood Business Park, Hook, Hampshire RG27 9UP > D 01256 75 3362 > > I welcome VSRE emails. Learn more at http://vsre.info/ > > > ------------------------------ > *From:* kishore alajangi [mailto:[email protected]] > *Sent:* 16 June 2014 08:04 > *To:* [email protected] > *Subject:* Re: copy to hdfs > > could anybody help me ? > > > On Mon, Jun 16, 2014 at 10:27 AM, kishore alajangi < > [email protected]> wrote: > >> Instead just mentioning hdsfs.path = /flume/messages/, do i need to >> mention something else? >> >> >> On Mon, Jun 16, 2014 at 10:25 AM, kishore alajangi < >> [email protected]> wrote: >> >>> I created the /flume/messages directories, but still nothing is >>> written with flume in those directories. please help me. >>> >>> >>> On Mon, Jun 16, 2014 at 10:15 AM, kishore alajangi < >>> [email protected]> wrote: >>> >>>> Do I need to create the /flume/messages/ directories? >>>> >>>> >>>> >>>> On Mon, Jun 16, 2014 at 10:14 AM, kishore alajangi < >>>> [email protected]> wrote: >>>> >>>>> checked, nothing is written in hdfs. >>>>> >>>>> >>>>> On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <[email protected]> >>>>> wrote: >>>>> >>>>>> That just means source has done its work and is waiting for more data >>>>>> to read. Did you check hdfs to see if all data has been written? >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Mohit and sharminder, >>>>>>> >>>>>>> Thanks for reply, after I called with -n tier, source is not >>>>>>> directory error came, I changed the source to /tmp/ and hdfs.path to >>>>>>> /flume/messages/ in config file, and run the command, the INFO i am >>>>>>> getting >>>>>>> now is "spooling directory source runner has shutdown" >>>>>>> what could be the problem, please help me. >>>>>>> >>>>>>> >>>>>>> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Replace -n agent with -n tier1 >>>>>>>> >>>>>>>> >>>>>>>> On Sunday, June 15, 2014, kishore alajangi < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Dear Sharminder, >>>>>>>>> >>>>>>>>> Thanks for your reply, yes I am playing with flume, as you >>>>>>>>> suggested i am using spool directory source, My configuration file >>>>>>>>> looks >>>>>>>>> like >>>>>>>>> >>>>>>>>> *tier1.sources = source1 >>>>>>>>> tier1.channels = channel1 >>>>>>>>> tier1.sinks = sink1 >>>>>>>>> >>>>>>>>> tier1.sources.source1.type = spooldir >>>>>>>>> tier1.sources.source1.spoolDir = /var/log/messages >>>>>>>>> tier1.sources.source1.channels = channel1 >>>>>>>>> tier1.channels.channel1.type = memory >>>>>>>>> >>>>>>>>> tier1.sinks.sink1.type = hdfs >>>>>>>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages >>>>>>>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile >>>>>>>>> tier1.sinks.sink1.hdfs.filePrefix = data >>>>>>>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq >>>>>>>>> >>>>>>>>> # Roll based on the block size only >>>>>>>>> tier1.sinks.sink1.hdfs.rollCount=0 >>>>>>>>> tier1.sinks.sink1.hdfs.rollInterval=0 >>>>>>>>> tier1.sinks.sink1.hdfs.rollSize = 120000000 >>>>>>>>> # seconds to wait before closing the file. >>>>>>>>> tier1.sinks.sink1.hdfs.idleTimeout = 60 >>>>>>>>> tier1.sinks.sink1.channel = channel1 >>>>>>>>> >>>>>>>>> tier1.channels.channel1.capacity = 100000* >>>>>>>>> tier1.sources.source1.deserializer.maxLineLength = 32768 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> the command I used is >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf >>>>>>>>> -Dflume.root.logger=DEBUG,console -n agent >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> it gives warn after created sources, channels, sinks for tier1 agent >>>>>>>>> is >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> no configuration found for this host:agent >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> any help? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I want to copy my local data to hdfs using flume in a single >>>>>>>>>>> machine which isrunning hadoop, How can I do that, please help me. >>>>>>>>>>> >>>>>>>>>>> What is this "local data" ? >>>>>>>>>> >>>>>>>>>> If it's just files, why not use the hadoop fs copy command >>>>>>>>>> instead? If you want to play around with flume, take a look at the >>>>>>>>>> spool >>>>>>>>>> directory source or the exec source and you should be able to put >>>>>>>>>> something >>>>>>>>>> together that'll push data through flume to hadoop. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sharninder >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks, >>>>>>>>> Kishore. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Kishore. >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Kishore. >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Kishore. >>>> >>> >>> >>> >>> -- >>> Thanks, >>> Kishore. >>> >> >> >> >> -- >> Thanks, >> Kishore. >> > > > > -- > Thanks, > Kishore. > > > -------------------------------------------------------------------- > Save Paper - Do you really need to print this e-mail? > > Visit www.virginmedia.com for more information, and more fun. > > This email and any attachments are or may be confidential and legally > privileged > and are sent solely for the attention of the addressee(s). If you have > received this > email in error, please delete it from your system: its use, disclosure or > copying is > unauthorised. Statements and opinions expressed in this email may not > represent > those of Virgin Media. Any representations or commitments in this email are > subject to contract. > > Registered office: Media House, Bartley Wood Business Park, Hook, > Hampshire, RG27 9UP > Registered in England and Wales with number 2591237 > -- Thanks, Kishore.
