Hi Xingbo, Thanks for creating this FLIP. Big +1 for it!
I have some question about the Thread Mode: 1. It seems that we dynamically load an embedded Python and user dependencies in the TM process. Can they be uninstalled cleanly after the task finished? i.e. Can we use the Thread Mode in session mode and Pyflink shell? 2. Does one TM have only one embedded Python running at the same time? If all the Python operator in the TM share the same PVM, will there be a loss in performance? 3. How do we load the relevant c library if the python.executable is provided by users? May there be a risk of version conflicts? Best, Wei > 2021年12月29日 上午11:56，Xingbo Huang <hxbks...@gmail.com> 写道： > > Hi everyone, > > I would like to start a discussion thread on "Support PyFlink Runtime > Execution in Thread Mode" > > We have provided PyFlink Runtime framework to support Python user-defined > functions since Flink 1.10. The PyFlink Runtime framework is called Process > Mode, which depends on an inter-process communication architecture based on > the Apache Beam Portability framework. Although starting a dedicated > process to execute Python user-defined functions could have better resource > isolation, it will bring greater resource and performance overhead. > > In order to overcome the resource and performance problems on Process Mode, > we will propose a new execution mode which executes Python user-defined > functions in the same thread instead of a separate process. > > I have drafted the FLIP-206[1]. Please feel free to reply to this email > thread. Looking forward to your feedback! > > Best, > Xingbo > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-206%3A+Support+PyFlink+Runtime+Execution+in+Thread+Mode
