At the time 1.3.x was released, 1.6.0 hasn't been released yet. And we
didn't have enough time to upgrade and test Parquet 1.6.0 for Spark
1.4.0. But we've already upgraded Parquet to 1.7.0 (which is exactly the
same as 1.6.0 with package name renamed from com.twitter to
org.apache.parquet) on master branch recently.
Cheng
On 6/12/15 6:16 PM, Eric Eijkelenboom wrote:
Hi
What is the reason that Spark still comes with Parquet 1.6.0rc3? It
seems like newer Parquet versions are available (e.g. 1.6.0). This
would fix problems with ‘spark.sql.parquet.filterPushdown’, which
currently is disabled by default, because of a bug in Parquet 1.6.0rc3.
Thanks!
Eric
