h-vetinari opened a new pull request #31685: URL: https://github.com/apache/spark/pull/31685
Trying to revive #23721 from @nandorKollar: > ## What changes were proposed in this pull request? > > A new, more flexible logical type API was introduced in parquet-mr 1.11.0 (based on the the Thrift field in parquet-format available for a while). This change migrates from the old (now deprecated) enum-based OriginalType API to this new logical type API. > > In addition to replacing the deprecated API calls, this PR also introduces support for reading the new subtypes for different timestamp semantics. > > Since parquet-mr 1.11.0 is not yet released, this is tested against a release candidate. Before merging, the additional repository should be deleted from pom.xml, which can only be done once parquet-mr 1.11.0 is released. > ## How was this patch tested? > > Unit tests were added to the PR. I intentionally left the conflicts in the merge commit, so that it becomes clear how I've chosen (on a best effort basis...) to resolve them - this is obviously WIP. Also, please note that this is my first PR for spark, so I'm probably in above my head, and happy to close this PR if desired (or take any advice). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
