Hi Ewan,
Different from AvroParquetWriter, in Spark SQL we uses StructType as the
intermediate schema format. So when converting Avro files to Parquet
files, we internally converts Avro schema to Spark SQL StructType first,
and then convert StructType to Parquet schema.
Cheng
On 5/19/15 4:42 PM, Ewan Leith wrote:
Hi all,
I might be missing something, but does the new Spark 1.3 sqlContext
save interface support using Avro as the schema structure when writing
Parquet files, in a similar way to AvroParquetWriter (which I’ve got
working)?
I've seen how you can load an avro file and save it as parquet
fromhttps://databricks.com/blog/2015/03/24/spark-sql-graduates-from-alpha-in-spark-1-3.html,
but not using the 2 together.
Thanks, and apologies if I've missed something obvious!
Ewan
