Hi Shengkai, Thanks so much for the response and the details.
Enhancements you proposed make perfect sense. Was thinking along the same lines. I am interested in contributing to this. Have a rough design and a POC implementation around this. >From my initial digging, we would need to extend PythonTableFunctionOperator to encapsulate the Model Context with it. I will refine it and open a FLIP, so we can discuss more details. >> *Introduce a pluggable Model Provider interface* Sounds good. Also, to start with, we could also expose a very generic python model provider, where users can plugin their python implementation. Users can bring in their own generic Python implementation of PyPredictFunction. >> *Add built-in model providers for PyTorch and TensorFlow in the Flink repository* Yes, I agree, this is very much needed. Also, i see https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/dev/python/table/udfs/vectorized_python_udfs/ but there is no implementation of Vectorized Table function available currently. Providing that seems promising to use resources efficiently. >>*Optimize resource usage via cached files and batched inference* On Mon, Sep 22, 2025 at 7:56 PM Shengkai Fang <[email protected]> wrote: > Hi, Swapna. > > Supporting local models has always been a key milestone on our journey > toward Flink AI. After an offline discussion with @DianFu, we found that > the current API is not well-suited for PyFlink, as PyFlink prefers to > provide its own Python operators. > > To better accommodate this requirement and improve extensibility, we > propose the following enhancements: > > *Introduce a pluggable Model Provider interface* > Extend the model management framework by adding a standardized interface > for model providers, enabling custom implementations — such as a > PythonModelProvider — to supply native Python operators (e.g., > PythonTableFunctionOperator). > > *Add built-in model providers for PyTorch and TensorFlow in the Flink > repository* > Implement and integrate first-party model providers for popular frameworks > like PyTorch and TensorFlow. These providers will implement the new > interface and be maintained within the main Flink codebase. > > *Optimize resource usage via cached files and batched inference* > Leverage `pipeline.cached-files` to manage model weight files efficiently, > avoiding redundant downloads across tasks. Additionally, support batched > inference to maximize hardware utilization (e.g., GPU/TPU throughput) and > improve overall performance. > > Best, > Shengkai > > > Swapna Marru <[email protected]> 于2025年9月22日周一 23:36写道： > > > Hi Devs, > > > > > > I am interested in learning more about MODEL, ML_PREDICT, and ML_EVALUATE > > functionalities added in the following FLIP. > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-437%3A+Support+ML+Models+in+Flink+SQL > > > > > > I see the original FLIP has extensibility to local model providers in > > Flink. > > > > > > Is there a way to do pluggable local model providers in Python? Like, > say, > > generate embeddings using Sentence transformer models running locally in > > Flink. > > > > > > An option could be to introduce a Model Provider factory implementation > in > > Java that internally uses a predict function in Python . But I see this > > puts in a lot of work related to Java to Python communication/translation > > inside the provider. > > > > > > Something like PythonRuntimeProvider along with PredictRuntimeProvider / > > AsyncRuntimeProvider which can handle Java -> Python translations out of > > the box would be helpful to de-duplicate that effort. > > > > > > Can you please point to, if there are any discussions related to this > > already ? Or any other ways to achieve the same? Please share your > > thoughts. > > > > > > -Thanks, > > > > Swapna Marru > > >
