Thanks for your votes and feedbacks. I have discussed with @Zhu Zhu offline and also on the design doc.
It seems that we have reached consensus on the design. I would bring up the VOTE if there is no other feedbacks. Thanks, Dian > 在 2019年11月22日，下午2:51，Hequn Cheng <chenghe...@gmail.com> 写道： > > Thanks a lot for putting this together, Dian! Definitely +1 for this! > It is great to make sure that the resources used by the Python process are > managed properly by Flink’s resource management framework. > > Also, thanks to the guys that working on the unified memory management > framework. > > Best, Hequn > > > On Mon, Nov 18, 2019 at 5:23 PM Yangze Guo <karma...@gmail.com> wrote: > >> Thanks for driving this discussion, Dian! >> >> +1 for this proposal. It will help to reduce container failure due to >> the memory overuse. >> Some comments left in the design doc. >> >> Best, >> Yangze Guo >> >> On Mon, Nov 18, 2019 at 4:06 PM Xintong Song <tonysong...@gmail.com> >> wrote: >>> >>> Sorry for the late reply. >>> >>> +1 for the general proposal. >>> >>> And one remainder, to use UNKNOWN resource requirement, we need to make >>> sure optimizer knowns which operators use off-heap managed memory, and >>> compute and set a fraction to the operators. See FLIP-53[1] for more >>> details, and I would suggest you to double check with @Zhu Zhu who works >> on >>> this part. >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> [1] >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management >>> >>> On Tue, Nov 12, 2019 at 11:53 AM Dian Fu <dian0511...@gmail.com> wrote: >>> >>>> Hi Jincheng, >>>> >>>> Thanks for the reply and also looking forward to the feedback from the >>>> community. >>>> >>>> Thanks, >>>> Dian >>>> >>>>> 在 2019年11月11日，下午2:34，jincheng sun <sunjincheng...@gmail.com> 写道： >>>>> >>>>> Hi all, >>>>> >>>>> +1， Thanks for bring up this discussion Dian! >>>>> >>>>> The Resource Management is very important for PyFlink UDF. So, It's >> great >>>>> if anyone can add more comments or inputs in the design doc or >> feedback >>>> in >>>>> ML. :) >>>>> >>>>> Best, >>>>> Jincheng >>>>> >>>>> Dian Fu <dian0511...@gmail.com> 于2019年11月5日周二 上午11:32写道： >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> In FLIP-58[1] it will add the support of Python user-defined >> stateless >>>>>> function for Python Table API. It will launch a separate Python >> process >>>> for >>>>>> Python user-defined function execution. The resources used by the >> Python >>>>>> process should be managed properly by Flink’s resource management >>>>>> framework. FLIP-49[2] has proposed a unified memory management >> framework >>>>>> and PyFlink user-defined function resource management should be >> based on >>>>>> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about >>>> this. I >>>>>> draft a design doc[3] and want to start a discussion about PyFlink >>>>>> user-defined function resource management. >>>>>> >>>>>> Welcome any comments on the design doc or giving us feedback on the >> ML >>>>>> directly. >>>>>> >>>>>> Regards, >>>>>> Dian >>>>>> >>>>>> [1] >>>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table >>>>>> [2] >>>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors >>>>>> [3] >>>>>> >>>> >> https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m >>>> >>>> >>
