Thanks for the feedback, as we all agree to move the Kyuubi Hive JDBC shaded client to a separated repo, the next things are: 1. Any suggestions for repo name? e.g. `incubating-kyuubi-hive-jdbc` 2. We may need to shade some modules in the future as Paul exampled, should we put them together or separately?
For question 2, I need to clarify that as far as I know, we have at least 2 kinds of artifacts to be shaded and separated 1) Popular 3rd-party library, e.g. guava; 2) Kyuubi self-components, which should be light(no-deps) and used by other components for verification. e.g. Kyuubi Hive JDBC shaded client, Kyuubi Spark TPC-DS Connector Thanks, Cheng Pan On Thu, May 19, 2022 at 5:30 PM hongdd <jn_...@163.com> wrote: > > +1 for option 2 > > > Thanks, > hongdd > > > ---- Replied Message ---- > | From | Paul Lam<paullin3...@gmail.com> | > | Date | 05/19/2022 14:42 | > | To | <dev@kyuubi.apache.org> | > | Subject | Re: [DISCUSS] Kyuubi Hive JDBC shaded client and development > experience | > Good to see we have a plan to improve this situation! > > I’m +1 to option 2. It may add some maintenance overhead, but it’s more > intuitive. > I see lots of projects have their own shaded repo, e.g. Flink[1] and > Presto[2]. > > [1] https://github.com/apache/flink-shaded > [2] https://github.com/prestodb/presto-hive-apache > > Best, > Paul Lam > > 2022年5月18日 13:45，Cheng Pan <pan3...@gmail.com> 写道： > > Hi Kyuubi developers, > > Recently, we notice that many new contributors are confused about > Kyuubi's development and debugging process, because they can not run > `mvn test` or run test on IDE(e.g. Jetbrains IDEA) directly like most > maven based Java projects, instead, they should run `mvn install` > before doing that. > > The root cause is Kyuubi Hive JDBC shaded client. We introduce this > module because the upstream Hive JDBC client has lots of transitive > dependencies which may pollute the user classpath and cause class > conflict. The Kyuubi Hive JDBC shaded client shaded and relocated all > Hive transitive classes to make it friendly to downstream projects and > users. > > The problem is, Maven and IDEA[1] do not work well with shaded > modules, the current workaround requires the developer to run `mvn > install` before testing. > > I have 2 ideas to solve the problem. > > 1. Some Kyuubi modules use Kyuubi Hive JDBC shaded client for testing, > we let them depend on the latest released version of Kyuubi Hive JDBC > shaded client rather than the project module, and introduce a profile > to switch back on CI environment for verification. > > 2. Separate the Kyuubi Hive JDBC shaded client to an independent Git > repo, and make it have an independent release cycle with the Kyuubi > main project. > > Further, now the Kyuubi Hive JDBC shaded client has room to improve, > i.e. it's easy to deserialize the metadata and record from thrift > format to Java format directly, we don't need to use Hive serde to do > such things, then we can drop those Hive dependencies rather than > shade them; as Kyuubi proposes to support etcd as service discover > server, we also need to support that in Kyuubi Hive JDBC client. > > What do you guys think of that? > > [1] https://youtrack.jetbrains.com/issue/IDEA-126596 > > Thanks, > Cheng Pan >
