Hi Yuxia, FLINK-30667 <https://issues.apache.org/jira/browse/FLINK-30667> aims to address hive-connector dependency on table-planner @internal class . Here is a bit of rationale.
ParserImpl in table-planner and HiveParser in hive connector should be able to evolve separately after future externalization; Parser interface on other hand should keep consistent when community plan for other dialect support. This would help decouple one of the internal class dependencies you mentioned in FLINK-26603 <https://issues.apache.org/jira/browse/FLINK-26603>. QueryOperation is a simple operator class shared between table-planner as well as hive-connector. Keep a copy instead of announcing public in table-planner could decouple hive connectors from table-planner change or future proof when considering more customized operations. PlannerContext acts as a utility class providing reference to a group of components. we could make it publicEvolving. Let me know what you think. Chen On Tue, Jan 10, 2023 at 8:34 PM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > Hi, Chen. > Cool! Of course not. Much appreciated that you can help continue this work. > If you have any question about it, please let me know. > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "Chen Qin" <qinnc...@gmail.com> > 收件人: "dev" <dev@flink.apache.org> > 发送时间: 星期三, 2023年 1 月 11日 上午 11:24:48 > 主题: Re: [Discussion] externalize Hive connector > > Yuxia, > > thanks for details context, I see with your point on maintainability. If > your don’t mind, I would like to help continue the work you mentioned. > > Chen > > On Tue, Jan 10, 2023 at 03:47 yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > > > Hi, Chen. > > Appreciated your efforts. > > I would like to shared my thonghts about it. > > > > I do think it's not a good timing to externalize Hive connector right > now. > > As you know, the Hive connector relies some internal methods in > > flink-table-planner. The Flink devs can > > modify them with more freedom as they are internal methods with providing > > any compatibility gurantee. > > If externalize it now, it may well happen that some methods relied by > Hive > > connector have been removed/renamed from flink-table-planner, which will > > then cause the Hive connector fail. > > It'll bring much burden to the connector maintainer, the maintainer will > > need to take much care of these changes and do the adaption. As I see, > > it'll always end with only some users report this issue, did the > maintainer > > notice that. It'll be definitely a bad user experience. > > > > It's a histrical technical debt that Hive connector relies the internal > > methods in flink-table-planner, and we have created FLINK-26603[1] to > > resolve it. > > I'm intended to finish it in Flink 1.17, but may well fail to catch up > > with Flink 1.17. Feel sorry for that for I'm busy with other stuff. If > you > > have interest about it, appreciated that you can also take part in this > > jira. > > > > Of course you can just create a seperate Hive connector now, and then > > cherry pick changes from the Flink repo before the Hive connector can be > > removed from Flink repo. But I'm afraid that it may still take much > > efforts. And IIUC, the external Hive connector can't be released as > offical > > Hive connector unless we resolve the problem I refered above. > > So, why not wait any thing is ready before externalize the Hive > connector? > > > > Best regards, > > Yuxia > > > > ----- 原始邮件 ----- > > 发件人: "Chen Qin" <qinnc...@gmail.com> > > 收件人: "dev" <dev@flink.apache.org> > > 发送时间: 星期二, 2023年 1 月 10日 上午 11:47:02 > > 主题: Re: [Discussion] externalize Hive connector > > > > Hi Martijin, > > > > Thank you for sharing your thoughts. In my opinion, FLINK-26603 is no > > longer blocking FLINK-30064. > > That being said, we could consider keeping FLINK-26603 as follow-up work > > and breaking it down into smaller tasks as we proceed. > > > > [PR] https://github.com/apache/flink-connector-hive/pull/3 > > > > Chen > > > > On Mon, Jan 9, 2023 at 7:45 AM Martijn Visser <martijnvis...@apache.org> > > wrote: > > > > > Hi Chen, > > > > > > Thanks for bringing this up! I think it would be great if the Hive > > > connector is externalized. We've already previously established [1] > that > > it > > > should be externalized. I believe the only reason why this hasn't been > > done > > > yet is because it's blocked by > > > https://issues.apache.org/jira/browse/FLINK-26603. Is that still the > > case? > > > > > > Best regards, > > > > > > Martijn > > > > > > [1] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > > > > > > On Mon, Jan 9, 2023 at 4:22 PM Chen Qin <qinnc...@gmail.com> wrote: > > > > > > > Hi there, > > > > > > > > Following community guidance Externalized+Connector+development > > > > < > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > > >, > > > We > > > > would like to initiate discussion on moving connector/hive to > > > > apache/flink-connector-hive > > > > <https://github.com/apache/flink-connector-hive>. > > > > > > > > Currently proposed changes includes > > > > > > > > - cleanup dependencies introduced from hive/yarn dependencies with > > > > latest package version stated in properties section in POM file > > > > - add FlinkPlannerCalciteShim to handle PlannerCalcite API > signature > > > > changes from 1.16 v.s 1.17-SNAPSHOT > > > > - add PackageITTests and ProductionArchitectureTests > > > > - [bonus] adding docker e2e tests with list of supported Hive/HMS > > > > versions > > > > > > > > Risk associated with this change includes not being able to release > > until > > > > 1.17 release, so we would have to keep cherry-pick changes from > > > > flink/connectors/hive for a period of time. > > > > > > > > Looking forward to hearing community feedback. > > > > > > > > Chen > > > > > > > > > >
