Hi Jianyun, I've assigned you to the respective tickets.
Cheers, Till On Thu, Jan 14, 2021 at 9:07 PM 赵 建云 <zhaojianyu...@outlook.com> wrote: > Hi Arvid, > > Thank you very much for being a mentor to the Flink community, I will > finish submitting all the content as soon as possible. > > The PR submission will be supported by me and weijie Guo( > wei...@streamnative.io). > I have submitted PRs for SourceFunction > <https://github.com/apache/flink/pull/14622> and SinkFunction > <https://github.com/apache/flink/pull/14623> now, but I see from > flink-bot that I need to ticket and assign tasks in jira first. I need your > help here to handle it. > > > Thanks for your help! > > Jianyun Zhao > > 2021年1月6日 下午6:50，Arvid Heise <ar...@ververica.com> 写道： > > Hi Till, > > 1) Who from the Flink community will mentor this effort and could take > > responsibility for it? > > > I'd be happy to mentor the transition. It remains to be seen who is doing > mainly the maintenance in the long run. If all fails, I can also take that > over but I was hoping that the respective connector authors would also > become Flink contributors/committers in the long run. > > 2) How can Pulsar be tested together with Flink (is there a Pulsar mini > > cluster for IT tests)? > > > We use testcontainer for running a Pulsar standalone cluster to do the > integration tests. If it is not suitable for Flink project, we are happy to > adopt any technologies the Flink community prefers. > > > We currently try to establish testcontainers for some external systems, so > this sounds perfect to me. > > 3) Which Pulsar versions will/can be supported? > > > > Pulsar has a pretty good backward compatibility between versions. In our > experience, people have been running the connector with Pulsar clusters > from 2.5.x to 2.7.x in production. > > > I must admit that I didn't think too much about versions yet. Pulsar has a > fast development pace, so I'm assuming that we also need to cut support for > older versions rather quickly. We should avoid supporting several major > versions like we do with other connectors. Maybe we can phase out old > versions into flink-packages.org? > > If the Pulsar community is willing to help with some of these tasks, then > > this should be much easier but we need buy in from someone in the Flink > community to help with this effort. > > > I hope that Sijie and I addressed your concerns. > > Best, > > Arvid > > > On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <s...@streamnative.io> wrote: > > Hi Till, > > Thank you for your email! Please find my comments inline. > > On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <trohrm...@apache.org> > wrote: > > Hi Jianyun, > > Thanks a lot for reviving this discussion. I think it would be great to > have a well working Pulsar connector for Flink. Before diving into the > detailed plan on how to do it technically, I think we should discuss where > exactly the connector should live. From the Flink community's perspective > every additional connector adds a considerable maintenance burden and, > thus, needs to be carefully considered. > > 1) Who from the Flink community will mentor this effort and could take > responsibility for it? > > > We had a conversation with Stephen and Arvid about this. I think Arvid > Heise was willing to mentor this effort. > > > 2) How can Pulsar be tested together with Flink (is there a Pulsar mini > cluster for IT tests)? > > > We use testcontainer for running a Pulsar standalone cluster to do the > integration tests. If it is not suitable for Flink project, we are happy to > adopt any technologies the Flink community prefers. > > > 2.1) Do we need additional e2e tests? > > > The Pulsar-Flink connector repository contains pretty good test coverage. > If it is not sufficient, we are happy to add more to follow Flink's best > practices. > > > 3) Which Pulsar versions will/can be supported? > > > Pulsar has a pretty good backward compatibility between versions. In our > experience, people have been running the connector with Pulsar clusters > from 2.5.x to 2.7.x in production. > > > > If the Pulsar community is willing to help with some of these tasks, then > this should be much easier but we need buy in from someone in the Flink > community to help with this effort. > > > I am one of the Pulsar PMC members. Some of our Pulsar PMC members and > committers are committed to helping maintain this Pulsar Flink connector. > Hope this helps eliminate some concerns here. > > > The Flink community might be a bit unresponsive in the next couple of > days because of Christmas and New Year. Beginning of January the discussion > should see a couple of additional comments. > > Cheers, > Till > > On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zhaojianyu...@outlook.com> wrote: > > We are now done with the Flink Pulsar connector. It is ready to be > merged. > > Now our plan is > > 1. merge Source and Sink based on Function, because it has been > verified by users and is reliable. > 2. Next is to merge the DynamicTable feature. table implementation > depends on Step 1. > 3. merge Pulsar Catalog > 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface< > > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-27%253A%2BRefactor%2BSource%2BInterface&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293093069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=z9qIAy47xgEXOJpPDLuj%2FXi%2B72mzym8mcO9%2FJg5HEZA%3D&reserved=0 > > > of PulsarSource > 5. merge PulsarSink based on FLIP-143: Unified Sink API< > > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-143%253A%2BUnified%2BSink%2BAPI&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uBvfbkA0%2Bi0z2evYt%2FcSAqpV64Dn6ZaqLzLgREQ%2BKbo%3D&reserved=0 > > > 6. merge upsert-pulsar > > About wiki, I need to update the FLIP-72: Introduce Pulsar Connector< > > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-72%253A%2BIntroduce%2BPulsar%2BConnector&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KyUsD5EzruT0TxHlJogV1V2FQb%2B42kP7I0eUd9Eb1LI%3D&reserved=0 > >, > but I don't have access to it. > > StreamNative/Pulsar Flink Connector: > > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstreamnative%2Fpulsar-flink&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2oWFw2BaNYSn9kR5ZUIHWLAHUzpGoIJSeOW%2FAdw7SIY%3D&reserved=0 > > > Looking forward to your feedback~ > > Jianyun Zhao > > > > -- > > Arvid Heise | Senior Java Developer > > < > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ververica.com%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aLxoBwbE%2BRhC8pkQkq%2BkFwmEsq08LLRxPUgGb6C7CxM%3D&reserved=0 > > > > Follow us @VervericaData > > -- > > Join Flink Forward < > https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fflink-forward.org%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MBd4VnU4DVqoDEy3iT2voj2QzqsLg2Ouf0TOuBO2%2BaE%3D&reserved=0> > - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng > > >
