Hi Pan, You completely misunderstood what I meant.
kyuubi-shaded-hive-service-rpc, kyuubi-hive-service-rpc-shaded I just wanted to know why these two kinds of naming policies can make both end-users and kyuubi developers happy? Why is even this necessary? Is it said that if an end-users use kyuubi-hive-jdbc-shaded as a JDBC Driver, he/she could not use kyuubi-shaded-hive-service-rpc as as a thrift client? Cheng Pan <pan3...@gmail.com> 于2023年11月30日周四 11:25写道： > > > >> The artifacts end with “-shaded” are for end users. > > > > This looks like an explanation for mistakes we've already made. > > I don’t think they're mistakes, only facts. In practice, there are various > ways to name this type of jar, we just picked one of them. > > There are some examples: > > - https://mvnrepository.com/artifact/com.datastax.oss/java-driver-core-shaded > - > https://mvnrepository.com/artifact/org.apache.zeppelin/zeppelin-interpreter-shaded > - > https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-3.3 > - https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-api > - https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-runtime > - > https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl-assembly > > Currently, Kyuubi has two such modules > > - kyuubi-hive-jdbc-shaded > - kyuubi-spark-authz-shaded > > The main purpose of this kind of artifact is to assemble all transitive > dependencies into one jar, usually with proper third-party class relocation > but WITHOUT changing any public API, e.g. for the JDBC driver, the public API > is JDBC API, it also means that the user can migrate from `A` to `A-shaded` > smoothly if only public API is consumed. > > > I've never been heard of such policy being discussed, documented. > > There hasn't been any specific topic of discussion about name policy before. > It's impossible to discuss everything in detail. If someone thinks something > matters, then we should discuss it. > > > Furthermore, since we've already posted them publicly in the maven > > repository, > > how can we decide that they won't be used by end-users? > > Technically, we can not stop the user from using such shaded API directly. > Personally, I think it’s a quite common practice, especially in Apache > projects, thus it’s obvious to me, for example: > > - > https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-netty > - > https://mvnrepository.com/artifact/org.apache.hadoop.thirdparty/hadoop-shaded-guava > - https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-jackson > - https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-bundled-guava > > Also, I just follow the most popular name pattern > `<project_name>-shaded-<thrid_party_name>` to name Kyuubi shaded artifacts. > > > Did we explicitly remove the public APIs from the original artifact？ > > > No, in most cases, just simple relocation and repackaging of the original > jar, with class overwriting if necessary. > > Thanks, > Cheng Pan > > > > On Nov 29, 2023, at 11:13, Kent Yao <y...@apache.org> wrote: > > > >> The artifacts end with “-shaded” are for end users. > > > > This looks like an explanation for mistakes we've already made. > > > > I've never been heard of such policy being discussed, documented. > > > > Furthermore, since we've already posted them publicly in the maven > > repository, > > how can we decide that they won't be used by end-users? Did we explicitly > > remove > > the public APIs from the original artifact？ > > > > Kent > > > > Cheng Pan <pan3...@gmail.com> 于2023年11月29日周三 10:25写道： > >> > >> Kent > >> > >> They are for different purposes. > >> > >> The artifacts start with "kyuubi-shaded” is for the Kyuubi project's > >> internal usage, see [1] and [2] > >> > >> The artifacts end with “-shaded” are for end users. > >> > >> [1] https://github.com/apache/kyuubi-shaded > >> [2] > >> https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-shaded-zookeeper-34 > >> > >> Thanks, > >> Cheng Pan > >> > >> > >>> On Nov 28, 2023, at 18:36, Kent Yao <y...@apache.org> wrote: > >>> > >>> -1 for the arbitrary naming > >>> > >>> > >>> FYI, > >>> https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-hive-jdbc-shaded > >>> > >>> XiDuo You <ulyssesyo...@gmail.com> 于2023年11月28日周二 18:03写道： > >>>> > >>>> +1, looks fine to me > >>>> > >>>> Zhen Wang <wangz...@apache.org> 于2023年11月28日周二 12:53写道： > >>>>> > >>>>> +1, I like this idea, it avoids the possibility of engine thrift class > >>>>> conflicts in different environments. > >>>>> > >>>>> > >>>>> Kind Regards, > >>>>> Zhen Wang > >>>>> > >>>>> Cheng Pan <pan3...@gmail.com> 于2023年11月28日周二 01:00写道： > >>>>>> > >>>>>> Hi Kyuubi developers, > >>>>>> > >>>>>> Recently, I've been looking at the code of the Hive engine, one of my > >>>>>> goals id to make it support multiple versions of Hive runtime, > >>>>>> including > >>>>>> - 3.1.3, which is the latest stable version of Apache Hive, already > >>>>>> supported > >>>>>> - 2.3.9, which is the latest stable version of Apache Hive 2.x, is > >>>>>> adopted widely, including Apache Spark, Apache Flink > >>>>>> - 2.1.1-cdh6.3.2, the latest free version of CDH, has lots of users > >>>>>> > >>>>>> When I tried to run the Hive engine built against Hive 3.x with Hive > >>>>>> 2.3.9/2.1.1-cdh6.3.2 runtime, I encountered some thrift class conflict > >>>>>> issues which were hard to resolve, thus I propose to create a > >>>>>> pre-shaded hive-service-rpc to tackles such issue. > >>>>>> > >>>>>> I have created two PRs[1][2] to demonstrate and verify my idea, it > >>>>>> also introduces additional benefits, e.g. speed up the engine > >>>>>> packaging by reducing relocation classes. > >>>>>> > >>>>>> Looking forward to the community feedback and PR reviews. > >>>>>> > >>>>>> Once the idea is accepted, and [1] gets merged, I will start the > >>>>>> kyuubi-shaded release voting. > >>>>>> > >>>>>> [1] https://github.com/apache/kyuubi-shaded/pull/20 > >>>>>> [2] https://github.com/apache/kyuubi/pull/5783 > >>>>>> > >>>>>> Thanks, > >>>>>> Cheng Pan > >> >
