gatorsmile commented on issue #24346: [SPARK-27176][FOLLOW-UP][SQL] Upgrade Hive parquet to 1.10.1 URL: https://github.com/apache/spark/pull/24346#issuecomment-484296944 > Hive on vectorized reader is quite common AFAIK @felixcheung How about upgrading Parquet to 1.11 in Spark 3.1 release, if the new release of Parquet becomes table. In the current stage, let us try to use 1.10.1 first and disable the Hive's vectorized Parquet readers. [You know, our Spark native Parquet reader is also vectorized. I assume our reader is either faster or as fast as Hive vectorized reader.] If we do not hit any issue using 1.10.1, we can avoid shading our Parquet version in sql/core.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
