pan3793 edited a comment on pull request #994: URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912791182
Stand on the Spark side, HUDI is a data source implementation, or just call it a plugin. Thus, for the transitive dependencies of HUDI, like Hadoop, Curator, Parquet, Avro, ORC, Hive, we must **RESPECT** the Spark bundled versions, as Kyuubi support multi Spark versions, we need to separate those deps version by different Spark versions using maven profiles. e.g. use parquet 1.10.0 for Spark 3.0/3.1, but use parquet 1.12.0 for Spark 3.2(coming soon). Besides on Spark, we should also respect the Kyuubi deps version, assume you are using IDEA, please open the `External Libraries`, there are lots of jars which has multi versions exist in the classpath, which means a high risk of class conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
