————— 2022-5-23 ————— Heisenberg 22:51 @liangqilang_bosszhipin_Zhu Hui Brother Hui, I have compiled a simple document for your reference, I hope it will be helpful. After you improve the account, you can also help to review the document to see if there is anything that needs to be modified.
Heisenberg 22:53 In the future, the original md file will be put into the project document @peacewong@WDS Brother Ping will help you when you have time to see if there is any wrong description. peacewong@WDS 22:56 Brother Longping, it is too strong, you can mention it under the web site. It is easy to reply if you have any questions, just modify the new engine implementation document directly. Heisenberg 22:56 that line Heisenberg 22:56 website I remember it was another project, right? peacewong@WDS 22:57 Yes, it is the project on the official website. Heisenberg 22:58 Ok, use it first, if there is a problem, I will improve it peacewong@WDS 22:58 The document basically covers the basic process, it is recommended to use the jdbc engine already supported by Linkis to explain peacewong@WDS 22:59 @Andy@WDS This can also be delivered to the public account Heisenberg 23:01 Well, this was originally intended as a supplement for users, just to provide an example, in addition to implementing the code of the engine itself, there are some extra things to do peacewong@WDS 23:02 Hahaha, it can be used as an official document [呲 tooth] Heisenberg 23:02 How can this be improved? peacewong@WDS 23:04 Brother Longping, it is basically the same as the jdbc engine. The engine installation section can add links to the official engine plugin installation documentation. Heisenberg 23:04 Uh-huh Heisenberg 23:05 In fact, I have always wanted to implement a kyuubi engine, especially I want to make a progress pull interface at the kyuubi layer, and then integrate it on the Scripts side, so that the experience with the kyuubi engine will be better [break tears] Heisenberg 23:06 I will improve this tomorrow and put it on the web peacewong@WDS 23:07 Yes, look at version 1.1.4 and add it. Or the extension above the jdbc engine can also be Heisenberg 23:08 The extension on JDBC is actually supported now. After jdbc kerberos accesses kyuubi, there is no problem. I will look at this later. peacewong@WDS 23:08 Okay, Brother Longping [handshake] Heisenberg 23:08 It's mainly about the progress of pulling this layer, you have to study it [cover face] W 23:17 Does the jdbc engine extend the hive driver? You can connect hiveserver2 or kyuubi Heisenberg 23:18 use hive-jdbc Heisenberg 23:19 Can directly access kyuubi hiveserver2 sparkthriftserver Heisenberg 23:20 No extension general jdbc interface Heisenberg 23:20 Access is also supported in the kerberos environment W 23:20 HiveStatement#setInPlaceUpdateStream progress should be obtained through this W 23:21 you can try Heisenberg 23:21 I don't know how to add in kyuubi [cover face] Heisenberg 23:21 Let me see W 23:21 The same is the hive protocol Heisenberg 23:22 Mainly it needs to be packaged in KyuubiStatement, which is not very familiar [laughs through tears] W 23:23 It should be possible to use hive driver directly W 23:23 Then update the progress through this in the jdbc engine W 23:24 This is also compatible with hiveserver2 peacewong@WDS 23:24 It would be better if it was compatible Heisenberg 23:26 In kyuubi-jdbc, List<String> getQueryLog(boolean incremental, int fetchSize) has been implemented in the main KyuubiStatement. Now I have implemented the pull real-time log, and then I want to add an interface for calculating the progress @wForget-iQiyi-Wang Zhen Heisenberg 23:26 I see this too Heisenberg 23:26 Mmmm optimize user experience 😁 W 23:27 kyuubi statement also has this method setInPlaceUpdateStream W 23:27 Implement an InPlaceUpdateStream to update the progress peacewong@WDS 23:27 Can pull logs and progress user experience is good Heisenberg 23:28 Support beeline? Heisenberg 23:28 Only W 23:28 Just implement a new one Heisenberg 23:28 Hmm, I'll look into it W 23:29 Create a new type to implement the InPlaceUpdateStream interface, it is better to use hive-jdbc to be more general Heisenberg 23:30 Hmm, I'll think about how to achieve this Heisenberg 23:34 Brother Ping thought about the implementation details of the new engine. I will change it to the presto engine. In fact, it is the same as kyuubi. It just happened to supplement the development documentation of the 1.2.0 presto engine for Brother Zhen. Heisenberg 23:34 😁 peacewong@WDS 23:34 Hahaha, you can Heisenberg 23:34 👌 ————— 2022-5-24 ————— Heisenberg 09:17 @peacewong@WDS Brother Ping, I have considered and revised the additional details added by the engine, and recorded it from the standpoint of the official website document, which feels quite complete. 😁 Heisenberg 09:17 Image 1 (can be viewed in the attachment) peacewong@WDS 09:18 I feel that the official website can be optimized Heisenberg 09:19 merge? peacewong@WDS 09:24 Well, it's better to combine them. Brother Longping, be careful with this Heisenberg 09:28 OK ————— 2022-5-23 ————— 海森堡 22:51 @liangqilang_boss直聘_朱辉 辉哥，整了个简单的文档，供你参考，希望有帮助，你那边完善号之后，也帮忙review下文档，看哪里有需要修改的 海森堡 22:53 后续会把md原文件放到项目文档中 @peacewong@WDS 平哥有空也帮忙看下，看看有没有描述错误的地方 peacewong@WDS 22:56 龙平兄，太强了，可以提到web site下。有问题了也方便回复，直接修改新引擎实现文档就好。 海森堡 22:56 那行 海森堡 22:56 website 我记得是另一个项目对吧 peacewong@WDS 22:57 对的，就是官网那个项目。 海森堡 22:58 好的，那先用着，有问题 我再完善 peacewong@WDS 22:58 文档基本涵盖了基本流程了，建议用Linkis已经支持的jdbc引擎来做说明[呲牙][呲牙] peacewong@WDS 22:59 @Andy@WDS 这个还可以投递到公众号 海森堡 23:01 嗯嗯，这个原想的只是用作用户的一个补充，只是提供一个例子，除了实现引擎本身的代码之外，还有一些额外的东西要做 peacewong@WDS 23:02 哈哈哈，可以作为官方文档的[呲牙] 海森堡 23:02 这个再怎么完善下？ peacewong@WDS 23:04 龙平兄，以jdbc引擎来说明基本就差不多了。引擎安装部分可以加上链接官方的引擎插件安装文档。 海森堡 23:04 嗯嗯 海森堡 23:05 其实一直想实现一个kyuubi引擎的，尤其是想在kyuubi那层做一个进度拉取的接口，然后集成在Scripts这端，这样使用起kyuubi引擎来体验会更好[破涕为笑] 海森堡 23:06 这个我明天再完善下，放到web上 peacewong@WDS 23:07 可以呀，看1.1.4版本加上下。或者jdbc引擎上面扩展也可 海森堡 23:08 JDBC上做扩展这个其实现在就支持了，jdbc kerberos之后访问kyuubi就没问题啦，这个后面我再看看 peacewong@WDS 23:08 好的，龙平兄[握手] 海森堡 23:08 主要是进度拉取这层，得研究下[捂脸] W 23:17 jdbc 引擎扩展一下 hive driver ？ 可以连 hiveserver2 也可以连 kyuubi 海森堡 23:18 使用hive-jdbc 海森堡 23:19 可以直接访问kyuubi hiveserver2 sparkthriftserver 海森堡 23:20 没做扩展 通用jdbc接口 海森堡 23:20 kerberos环境下也支持访问的 W 23:20 HiveStatement#setInPlaceUpdateStream 进度应该是通过这个去获取 W 23:21 可以试试 海森堡 23:21 kyuubi中我不知道咋加[捂脸] 海森堡 23:21 我看看 W 23:21 一样的 都是 hive 协议 海森堡 23:22 主要是需要在KyuubiStatement做包装，这个不太熟悉[破涕为笑] W 23:23 直接用 hive driver 连应该也可以 W 23:23 然后在 jdbc 引擎里面通过这个去更新进度 W 23:24 这样还兼容 hiveserver2 peacewong@WDS 23:24 能兼容的话就更好了 海森堡 23:26 在kyuubi-jdbc中呢，主要KyuubiStatement中已经实现了List<String> getQueryLog(boolean incremental, int fetchSize)，我现在实现了拉实时日志，然后还想加一个计算进度的接口 @wForget-爱奇艺-王震 海森堡 23:26 这个我也看看 海森堡 23:26 嗯嗯 优化用户体验 😁 W 23:27 kyuubi statement 里面也有这个方法 setInPlaceUpdateStream W 23:27 实现一个 InPlaceUpdateStream，去更新进度 peacewong@WDS 23:27 能拉日志和进度用户体验好 海森堡 23:28 支持beeline？ 海森堡 23:28 只 W 23:28 实现一个新的就行 海森堡 23:28 嗯嗯 我再研究下 W 23:29 新建一个类型实现 InPlaceUpdateStream 接口，最好使用 hive-jdbc 通用一些 海森堡 23:30 嗯嗯，我再琢磨琢磨这块怎么实现 海森堡 23:34 平哥 想了想 新引擎实现细节 我改成以presto引擎吧 其实跟kyuubi是一样的 刚好为震哥补充1.2.0 presto引擎的开发文档 海森堡 23:34 😁 peacewong@WDS 23:34 哈哈哈，也可以的 海森堡 23:34 👌 ————— 2022-5-24 ————— 海森堡 09:17 @peacewong@WDS 平哥，又考虑并修改了下，引擎新增的额外细节部分，以官网文档的立场来记录，这样感觉也挺完整的。😁 海森堡 09:17 图片1（可在附件中查看） peacewong@WDS 09:18 感觉可以优化下官网的 海森堡 09:19 合并起来？ peacewong@WDS 09:24 嗯啦，就合并起来好点。龙平兄这个仔细点 海森堡 09:28 好的 --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@linkis.apache.org For additional commands, e-mail: dev-h...@linkis.apache.org
