The goal to convert to avro is for us to use the splittable property so we can have some type of compressed data for huge log files so we can save some hdfs disk spaces.
On Fri, Jan 28, 2011 at 1:02 PM, Anand Padmanaban <[email protected]>wrote: > Meta question. I see avro is the means, what is the end goal? What do you > want to do with the data after converting it to avro? > > > -----Original Message----- > > From: felix gao [mailto:[email protected]] > > Sent: Friday, January 28, 2011 12:30 PM > > To: [email protected] > > Subject: Re: How to get started with examples on avro > > > > Thanks for the quick reply. I am interested in doing this through the > java implementation and I would like to do it in parallel that > > utilizes the mapreduce framework. > > > > > > On Fri, Jan 28, 2011 at 12:22 PM, Harsh J <[email protected]> > wrote: > > > > > > Based on the language you're targeting, have a look at its > test-cases > > available on the in the project's version control: > > http://svn.apache.org/repos/asf/avro/trunk/lang/ [You can check it > out > > via SVN, or via Git mirrors] > > > > Another good resource on the ends of Avro (Data and RPC) is by > phunt > > at http://github.com/phunt/avro-rpc-quickstart#readme > > > > I had written a python data-file centric snippet for Avro a while > ago > > at my blog; it may help if you're looking to get started with > Python > > (although it does not cover all aspects, which the functions in the > > available test cases for lang/python do): > > > http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/ > > > > > > On Sat, Jan 29, 2011 at 1:34 AM, felix gao <[email protected]> > wrote: > > > Hi all, > > > I am trying to convert a lot of our existing logs into avro > format in > > > hadoop. I am not sure if there are any examples to follow. > > > Thanks, > > > Felix > > > > > > > > > > -- > > Harsh J > > www.harshj.com > > > > > >
