rdblue commented on issue #26804: [WIP][SPARK-26346][BUILD][SQL] Upgrade parquet to 1.11.0 URL: https://github.com/apache/spark/pull/26804#issuecomment-564678199 The main change is that Parquet 1.11.0 will now write column indexes near the Parquet footer for page-level skipping. Skipping is not turned on by default. There are also some changes with how logical types are tracked in metadata that allow storing extra information, like whether a timestamp is `timestamp with time zone` or `timestamp without time zone`. I don't know of anyone running 1.11.0 in production yet. I think @mccheah runs a branch close to Parquet master and may be running something like 1.11.0. I tend to agree with the cautious approach. Let's at least run the benchmarks to verify there is no perf regression from the changes. I'd also be fine delaying this until 3.1.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
