Something like this works and is how I create an RDD of specific records. val avroRdd = sc.newAPIHadoopFile("twitter.avro", classOf[AvroKeyInputFormat[twitter_schema]], classOf[AvroKey[twitter_schema]], classOf[NullWritable], conf) (From https://github.com/julianpeeters/avro-scala-macro-annotation-examples/blob/master/spark/src/main/scala/AvroSparkScala.scala) Keep in mind you'll need to use the kryo serializer as well.
From: Frank Austin Nothaft <fnoth...@berkeley.edu<mailto:fnoth...@berkeley.edu>> Date: Wednesday, November 5, 2014 at 5:06 PM To: Simone Franzini <captainfr...@gmail.com<mailto:captainfr...@gmail.com>> Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: AVRO specific records Hi Simone, Matt Massie put together a good tutorial on his blog<http://zenfractal.com/2013/08/21/a-powerful-big-data-trio/>. If you’re looking for more code using Avro, we use it pretty extensively in our genomics project. Our Avro schemas are here<https://github.com/bigdatagenomics/bdg-formats/blob/master/src/main/resources/avro/bdg.avdl>, and we have serialization code here<https://github.com/bigdatagenomics/adam/tree/master/adam-core/src/main/scala/org/bdgenomics/adam/serialization>. We use Parquet for storing the Avro records, but there is also an Avro HadoopInputFormat. Regards, Frank Austin Nothaft fnoth...@berkeley.edu<mailto:fnoth...@berkeley.edu> fnoth...@eecs.berkeley.edu<mailto:fnoth...@eecs.berkeley.edu> 202-340-0466 On Nov 5, 2014, at 1:25 PM, Simone Franzini <captainfr...@gmail.com<mailto:captainfr...@gmail.com>> wrote: How can I read/write AVRO specific records? I found several snippets using generic records, but nothing with specific records so far. Thanks, Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.