Thanks a lot for putting this together, Dian! Definitely +1 for this!
It is great to make sure that the resources used by the Python process are
managed properly by Flink’s resource management framework.

Also, thanks to the guys that working on the unified memory management
framework.

Best, Hequn


On Mon, Nov 18, 2019 at 5:23 PM Yangze Guo <karma...@gmail.com> wrote:

> Thanks for driving this discussion, Dian!
>
> +1 for this proposal. It will help to reduce container failure due to
> the memory overuse.
> Some comments left in the design doc.
>
> Best,
> Yangze Guo
>
> On Mon, Nov 18, 2019 at 4:06 PM Xintong Song <tonysong...@gmail.com>
> wrote:
> >
> > Sorry for the late reply.
> >
> > +1 for the general proposal.
> >
> > And one remainder, to use UNKNOWN resource requirement, we need to make
> > sure optimizer knowns which operators use off-heap managed memory, and
> > compute and set a fraction to the operators. See FLIP-53[1] for more
> > details, and I would suggest you to double check with @Zhu Zhu who works
> on
> > this part.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> >
> > On Tue, Nov 12, 2019 at 11:53 AM Dian Fu <dian0511...@gmail.com> wrote:
> >
> > > Hi Jincheng,
> > >
> > > Thanks for the reply and also looking forward to the feedback from the
> > > community.
> > >
> > > Thanks,
> > > Dian
> > >
> > > > 在 2019年11月11日,下午2:34,jincheng sun <sunjincheng...@gmail.com> 写道:
> > > >
> > > > Hi all,
> > > >
> > > > +1, Thanks for bring up this discussion Dian!
> > > >
> > > > The Resource Management is very important for PyFlink UDF. So, It's
> great
> > > > if anyone can add more comments or inputs in the design doc or
> feedback
> > > in
> > > > ML. :)
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > > Dian Fu <dian0511...@gmail.com> 于2019年11月5日周二 上午11:32写道:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> In FLIP-58[1] it will add the support of Python user-defined
> stateless
> > > >> function for Python Table API. It will launch a separate Python
> process
> > > for
> > > >> Python user-defined function execution. The resources used by the
> Python
> > > >> process should be managed properly by Flink’s resource management
> > > >> framework. FLIP-49[2] has proposed a unified memory management
> framework
> > > >> and PyFlink user-defined function resource management should be
> based on
> > > >> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about
> > > this. I
> > > >> draft a design doc[3] and want to start a discussion about PyFlink
> > > >> user-defined function resource management.
> > > >>
> > > >> Welcome any comments on the design doc or giving us feedback on the
> ML
> > > >> directly.
> > > >>
> > > >> Regards,
> > > >> Dian
> > > >>
> > > >> [1]
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > > >> [2]
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > > >> [3]
> > > >>
> > >
> https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m
> > >
> > >
>

Reply via email to