Hi Kyuubi developers,

Recently, we notice that many new contributors are confused about
Kyuubi's development and debugging process, because they can not run
`mvn test` or run test on IDE(e.g. Jetbrains IDEA) directly like most
maven based Java projects, instead, they should run `mvn install`
before doing that.

The root cause is Kyuubi Hive JDBC shaded client. We introduce this
module because the upstream Hive JDBC client has lots of transitive
dependencies which may pollute the user classpath and cause class
conflict. The Kyuubi Hive JDBC shaded client shaded and relocated all
Hive transitive classes to make it friendly to downstream projects and
users.

The problem is, Maven and IDEA[1] do not work well with shaded
modules, the current workaround requires the developer to run `mvn
install` before testing.

I have 2 ideas to solve the problem.

1. Some Kyuubi modules use Kyuubi Hive JDBC shaded client for testing,
we let them depend on the latest released version of Kyuubi Hive JDBC
shaded client rather than the project module, and introduce a profile
to switch back on CI environment for verification.

2. Separate the Kyuubi Hive JDBC shaded client to an independent Git
repo, and make it have an independent release cycle with the Kyuubi
main project.

Further, now the Kyuubi Hive JDBC shaded client has room to improve,
i.e. it's easy to deserialize the metadata and record from thrift
format to Java format directly, we don't need to use Hive serde to do
such things, then we can drop those Hive dependencies rather than
shade them; as Kyuubi proposes to support etcd as service discover
server, we also need to support that in Kyuubi Hive JDBC client.

What do you guys think of that?

[1] https://youtrack.jetbrains.com/issue/IDEA-126596

Thanks,
Cheng Pan

Reply via email to