Re: parquet support - some questions about code

2015-03-18 Thread Cheng Lian
Hey Gil, ParquetRelation2 is based on the external data sources API, which is a more modular and non-intrusive way to add external data sources to Spark SQL. We are planning to replace ParquetRelation with ParquetRelation2 entirely after the latter is more mature and stable. That's why you see

parquet support - some questions about code

2015-03-18 Thread Gil Vernik
Hi, I am trying to better understand the code for Parquet support. In particular i got lost trying to understand ParquetRelation and ParquetRelation2. Does ParquetRelation2 is the new code that should completely remove ParquetRelation? ( I think there is some remark in the code notifying this