Good to see we have a plan to improve this situation! I’m +1 to option 2. It may add some maintenance overhead, but it’s more intuitive. I see lots of projects have their own shaded repo, e.g. Flink[1] and Presto[2].
[1] https://github.com/apache/flink-shaded [2] https://github.com/prestodb/presto-hive-apache Best, Paul Lam > 2022年5月18日 13:45，Cheng Pan <pan3...@gmail.com> 写道： > > Hi Kyuubi developers, > > Recently, we notice that many new contributors are confused about > Kyuubi's development and debugging process, because they can not run > `mvn test` or run test on IDE(e.g. Jetbrains IDEA) directly like most > maven based Java projects, instead, they should run `mvn install` > before doing that. > > The root cause is Kyuubi Hive JDBC shaded client. We introduce this > module because the upstream Hive JDBC client has lots of transitive > dependencies which may pollute the user classpath and cause class > conflict. The Kyuubi Hive JDBC shaded client shaded and relocated all > Hive transitive classes to make it friendly to downstream projects and > users. > > The problem is, Maven and IDEA[1] do not work well with shaded > modules, the current workaround requires the developer to run `mvn > install` before testing. > > I have 2 ideas to solve the problem. > > 1. Some Kyuubi modules use Kyuubi Hive JDBC shaded client for testing, > we let them depend on the latest released version of Kyuubi Hive JDBC > shaded client rather than the project module, and introduce a profile > to switch back on CI environment for verification. > > 2. Separate the Kyuubi Hive JDBC shaded client to an independent Git > repo, and make it have an independent release cycle with the Kyuubi > main project. > > Further, now the Kyuubi Hive JDBC shaded client has room to improve, > i.e. it's easy to deserialize the metadata and record from thrift > format to Java format directly, we don't need to use Hive serde to do > such things, then we can drop those Hive dependencies rather than > shade them; as Kyuubi proposes to support etcd as service discover > server, we also need to support that in Kyuubi Hive JDBC client. > > What do you guys think of that? > > [1] https://youtrack.jetbrains.com/issue/IDEA-126596 > > Thanks, > Cheng Pan
