Thanks Chao for created this STIP. I have some questions hope be solved. 1. It seems that the main purpose of this proposal is to solve the current problem of multiple enumerators coexisting in Flink, rather than using a single enumerator to complete data distribution. But if we complete it according to the current design plan, what impact will it have on the Spark side? Can the Spark side still run normally? Or do we have any solutions that can be implemented to solve the problem of multiple enumerators running at the same time that Spark currently encounters? I think we should start coding only after we have a clear and implementable solution.
2. The logic of sleeping for 2 seconds will affect all engines and is not a reasonable solution. Why should we replace existing stable logic with new unstable logic? I think there should be other solutions to prevent repeated calls to pollNext, or repeated reading of split. ________________________ Jia Fan > 2023年12月7日 17:10，Lucifer Tyrant <tyrantluci...@apache.org> 写道： > > Hi all, > > https://cwiki.apache.org/confluence/display/SEATUNNEL/STIP6+-+Refactor+Flink+Source+Translation+Layer+using+new+source+flink+API+based+FLIP-27, > this is STIP please review. > > Chao.Tian > > Jia Fan <fanjiaemi...@qq.com.invalid <mailto:fanjiaemi...@qq.com.invalid>> > 于2023年12月4日周一 12:00写道： >> >> After communicating with Lucifer through WeChat, we thought we needed a STIP >> to explain the current problems we encountered, as well as the detailed >> logic and process of our solution. >> >> ________________________ >> >> Jia Fan >> >> >> >>> 2023年12月4日 11:05，Jia Fan <fanjiaemi...@qq.com> 写道： >>> >>> Thanks Lucifer for do this. I did some review on this PR, the change only >>> affects few source connector and flink translation layer. So I think it’s >>> fine if we can make sure this PR be merged ASAP. >>> >>> ________________________ >>> >>> Jia Fan >>> >>> >>> >>>> 2023年12月2日 16:34，JUN GAO <gaojun2...@apache.org> 写道： >>>> >>>> @hailin0 <mailto:wanghai...@apache.org> @Jia Fan >>>> <mailto:fan...@apache.org> What do you think about this? >>>> >>>> Lucifer Tyrant <tyrantluci...@apache.org <mailto:tyrantluci...@apache.org> >>>> <mailto:tyrantluci...@apache.org>> 于2023年11月29日周三 22:51写道： >>>>> Hi developers, >>>>> >>>>> After several major iterations in the community, the Source/Link v2 >>>>> API has stabilized, and the community has recently been promoting the >>>>> development of multi table read/write functionality. However, when I >>>>> was refactoring the Flink translation layer[1], I found that the >>>>> current implementation of the Source connector did not strictly follow >>>>> the pull push model originally designed. Therefore, I propose to pause >>>>> the merging of multi table read PR and adjust the implementation of >>>>> all Source connectors before proceeding with the related >>>>> functionality. >>>>> >>>>> What do you think? If you have any questions, please let me know! >>>>> >>>>> [1] https://github.com/apache/seatunnel/pull/5100 >>>>> >>>>> Chao Tian >>>> >>>> >>>> -- >>>> Best Regards >>>> ------------ >>>> EricJoy2048 >>>> >>>> gaojun2...@gmail.com <mailto:gaojun2...@gmail.com> >>>> <mailto:gaojun2...@gmail.com>
