dongjoon-hyun commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-819877731
@ggershinsky . You should split (1) `sql/core` module dependency from (2) the Spark distribution dependency. I didn't ask Apache Parquet dependency change or Apache Spark dependency change. I have the solution https://github.com/apache/spark/pull/32146#discussion_r612867557 already. Please move the test case to `hive` module simply. > @dongjoon-hyun @dbtsai I agree replacing codehaus jackson with the fasterxml one is the right thing to do in the next parquet version. Regarding the current situation - parquet 1.12.0 has been released, with the coudehaus runtime dependency. This jackson was leveraged by PME a few years back, and tested with Spark 2.4 and 3.0. All these Spark distros, and the latest 3.1.1, have the codhaus jar. The current master drops this dependency in the core, but maybe it can be kept for one more release, so PME is enabled in Spark 3.2.0? We will work on replacing the jackson in parquet, making sure it's properly tested (inc backwards compatibility with 1.1.2.0) etc, can take some time; the next parquet version would go into the next Spark version after 3.2.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
