> > At my company we use Avro heavily and it's not been fun when i've tried to > work with complex avro schemas and python. This may not be relevant to you > however...otherwise I found Python to be a great fit for Spark :) >
Have you tried using https://github.com/databricks/spark-avro ? It should help you avoid needing to work with avro directly by translating to/from DataFrames.
