Hi Anoop,
I don't see the exception you mentioned in the link. I can use spark-avro
to read the sample file users.avro in spark successfully. Do you have the
details of the union issue ?
On Sat, Feb 27, 2016 at 10:05 AM, Anoop Shiralige wrote:
> Hi Jeff,
>
> Thank
Hi Jeff,
Thank you for looking into the post.
I had explored spark-avro option earlier. Since, we have union of multiple
complex data types in our avro schema we couldn't use it.
Couple of things I tried.
-
Avro Record is not supported by pickler, you need to create a custom
pickler for it. But I don't think it worth to do that. Actually you can
use package spark-avro to load avro data and then convert it to RDD if
necessary.
https://github.com/databricks/spark-avro
On Thu, Feb 11, 2016 at 10:38