Thanks Fokko, I will definitely take a look at this.
Cheers Andrew From: "Driesprong, Fokko" <[email protected]> Date: Friday, August 24, 2018 at 2:39 AM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Spark data quality bug when reading parquet files from hive metastore Hi Andrew, This blog gives an idea how to schema is resolved: https://blog.godatadriven.com/multiformat-spark-partition There is some optimisation going on when reading Parquet using Spark. Hope this helps. Cheers, Fokko Op wo 22 aug. 2018 om 23:59 schreef t4 <[email protected]<mailto:[email protected]>>: https://issues.apache.org/jira/browse/SPARK-23576 ? -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]<mailto:[email protected]>
