Thank you Bejoy. Does file append is production stable?
Cheers! Manoj. On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <bejoy.had...@gmail.com> wrote: > ** > Hi Manoj > > You can load daily logs into a individual directories in hdfs and process > them daily. Keep those results in hdfs or hbase or dbs etc. Every day do > the processing, get the results and aggregate the same with the previously > aggregated results till date. > > Regards > Bejoy KS > > Sent from handheld, please excuse typos. > ------------------------------ > *From: * Manoj Babu <manoj...@gmail.com> > *Date: *Sun, 9 Sep 2012 21:28:54 +0530 > *To: *<mapreduce-user@hadoop.apache.org> > *ReplyTo: * mapreduce-user@hadoop.apache.org > *Subject: *Reg: parsing all files & file append > > Hi All, > > I have two questions, providing info on it will be helpful. > > 1, I am using hadoop to analyze and to find top n search term metric's > from logs. > If any new log file is added to HDFS then again we are running the job to > find the metrics. > Daily we will be getting log files and we are parsing the whole file and > getting the metric's. > All the log file's are parsed daily to get the latest metric's is there > any way is there any way to avoid this? > > 2, Does file append is production stable? > > Cheers! > Manoj. > >
