The Colossal Pipe (https://github.com/ThinkBigAnalytics/colossal-pipe)
framework also supports working with Avro as its native format for Java
map-reduce, but it also lets you read in JSON or text files as input to
mappers, making it fairly easy to use for this kind of conversion job. E.g.,
the hea
Felix,
After you've figured out how to work it for your application, I do encourage
you to contribute
(https://cwiki.apache.org/AVRO/how-to-contribute.html) examples
to the open source project. We'll find a place for them!
-- Philip
On Fri, Jan 28, 2011 at 12:29 PM, felix gao wrote:
> Thanks
The goal to convert to avro is for us to use the splittable property so we
can have some type of compressed data for huge log files so we can save
some hdfs disk spaces.
On Fri, Jan 28, 2011 at 1:02 PM, Anand Padmanaban wrote:
> Meta question. I see avro is the means, what is the end goal? Wha
On Sat, Jan 29, 2011 at 1:59 AM, felix gao wrote:
> Thanks for the quick reply. I am interested in doing this through the java
> implementation and I would like to do it in parallel that utilizes the
> mapreduce framework.
That operation is pretty similar to writing a normal output data file.
Y
Meta question. I see avro is the means, what is the end goal? What do you
want to do with the data after converting it to avro?
> -Original Message-
> From: felix gao [mailto:gre1...@gmail.com]
> Sent: Friday, January 28, 2011 12:30 PM
> To: user@avro.apache.org
> Subject: Re: How to ge
Thanks for the quick reply. I am interested in doing this through the java
implementation and I would like to do it in parallel that utilizes the
mapreduce framework.
On Fri, Jan 28, 2011 at 12:22 PM, Harsh J wrote:
> Based on the language you're targeting, have a look at its test-cases
> avail
Based on the language you're targeting, have a look at its test-cases
available on the in the project's version control:
http://svn.apache.org/repos/asf/avro/trunk/lang/ [You can check it out
via SVN, or via Git mirrors]
Another good resource on the ends of Avro (Data and RPC) is by phunt
at http:
Hi all,
I am trying to convert a lot of our existing logs into avro format in
hadoop. I am not sure if there are any examples to follow.
Thanks,
Felix
