I am planning to try upgrading spark sql to a newer version of parquet, too.
I'll let you know if I make progress.
Thanks,
Michael
On Oct 8, 2014, at 12:17 PM, Michael Armbrust wrote:
> Thats a good question, I'm not sure if that will work. I will note that we
> are hoping to do some upgra
Thats a good question, I'm not sure if that will work. I will note that we
are hoping to do some upgrades of our parquet support in the near future.
On Tue, Oct 7, 2014 at 10:33 PM, Michael Allman
wrote:
> Hello,
>
> I was interested in testing Parquet V2 with Spark SQL, but noticed after
> som
Hello,
I was interested in testing Parquet V2 with Spark SQL, but noticed after some
investigation that the parquet writer that Spark SQL uses is fixed at V1 here:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala#L350.
An
