Hi Dian:
Thanks for your driving. I have some questions: - Where should these configurations belong? You have mentioned tableApi/SQL, so should in TableConfig? - If just in table/sql, whether it should be called: table.python.****, because in table, all config options are called table.***. - What should table module do? So in CommonPythonCalc, we should read options from table config, and set resources to OneInputTransformation? - Are all buffer.memory off-heap memory? I took a look to AbstractPythonScalarFunctionOperator, there is a forwardedInputQueue, is this one a heap queue? So we need heap memory too? Hope to get your reply. Best, Jingsong Lee On Tue, Nov 26, 2019 at 12:17 PM Dian Fu <dian0511...@gmail.com> wrote: > Thanks for your votes and feedbacks. I have discussed with @Zhu Zhu > offline and also on the design doc. > > It seems that we have reached consensus on the design. I would bring up > the VOTE if there is no other feedbacks. > > Thanks, > Dian > > > 在 2019年11月22日，下午2:51，Hequn Cheng <chenghe...@gmail.com> 写道： > > > > Thanks a lot for putting this together, Dian! Definitely +1 for this! > > It is great to make sure that the resources used by the Python process > are > > managed properly by Flink’s resource management framework. > > > > Also, thanks to the guys that working on the unified memory management > > framework. > > > > Best, Hequn > > > > > > On Mon, Nov 18, 2019 at 5:23 PM Yangze Guo <karma...@gmail.com> wrote: > > > >> Thanks for driving this discussion, Dian! > >> > >> +1 for this proposal. It will help to reduce container failure due to > >> the memory overuse. > >> Some comments left in the design doc. > >> > >> Best, > >> Yangze Guo > >> > >> On Mon, Nov 18, 2019 at 4:06 PM Xintong Song <tonysong...@gmail.com> > >> wrote: > >>> > >>> Sorry for the late reply. > >>> > >>> +1 for the general proposal. > >>> > >>> And one remainder, to use UNKNOWN resource requirement, we need to make > >>> sure optimizer knowns which operators use off-heap managed memory, and > >>> compute and set a fraction to the operators. See FLIP-53[1] for more > >>> details, and I would suggest you to double check with @Zhu Zhu who > works > >> on > >>> this part. > >>> > >>> Thank you~ > >>> > >>> Xintong Song > >>> > >>> > >>> [1] > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > >>> > >>> On Tue, Nov 12, 2019 at 11:53 AM Dian Fu <dian0511...@gmail.com> > wrote: > >>> > >>>> Hi Jincheng, > >>>> > >>>> Thanks for the reply and also looking forward to the feedback from the > >>>> community. > >>>> > >>>> Thanks, > >>>> Dian > >>>> > >>>>> 在 2019年11月11日，下午2:34，jincheng sun <sunjincheng...@gmail.com> 写道： > >>>>> > >>>>> Hi all, > >>>>> > >>>>> +1， Thanks for bring up this discussion Dian! > >>>>> > >>>>> The Resource Management is very important for PyFlink UDF. So, It's > >> great > >>>>> if anyone can add more comments or inputs in the design doc or > >> feedback > >>>> in > >>>>> ML. :) > >>>>> > >>>>> Best, > >>>>> Jincheng > >>>>> > >>>>> Dian Fu <dian0511...@gmail.com> 于2019年11月5日周二 上午11:32写道： > >>>>> > >>>>>> Hi everyone, > >>>>>> > >>>>>> In FLIP-58[1] it will add the support of Python user-defined > >> stateless > >>>>>> function for Python Table API. It will launch a separate Python > >> process > >>>> for > >>>>>> Python user-defined function execution. The resources used by the > >> Python > >>>>>> process should be managed properly by Flink’s resource management > >>>>>> framework. FLIP-49[2] has proposed a unified memory management > >> framework > >>>>>> and PyFlink user-defined function resource management should be > >> based on > >>>>>> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about > >>>> this. I > >>>>>> draft a design doc[3] and want to start a discussion about PyFlink > >>>>>> user-defined function resource management. > >>>>>> > >>>>>> Welcome any comments on the design doc or giving us feedback on the > >> ML > >>>>>> directly. > >>>>>> > >>>>>> Regards, > >>>>>> Dian > >>>>>> > >>>>>> [1] > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > >>>>>> [2] > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >>>>>> [3] > >>>>>> > >>>> > >> > https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m > >>>> > >>>> > >> > > -- Best, Jingsong Lee
