Hi Shengkai, Yeah right, the current design relies on SPI to load model providers.
With GenericPythonModelProvider(in Java), my idea is to provide an implementation of Modelprovider in java registered through SPI itself. And have a GenericPyPredictFunction(PythonTableFunction), that can run the user provided implementation of predict function. This with the thought that high level interfaces still would allow any other specialized model provider implementations like PyTorch to have custom/complex option parsing or model sharing configs implemented in java. But for this, it needs a Java (modelfactory impl)+ Python (PredictFunction impl). -Thanks, M.Swapna On Tue, Sep 23, 2025 at 7:48 PM Shengkai Fang <[email protected]> wrote: > Hi, Swapna. > > Thanks for the quick response. > > >> we could also expose a very generic python model provider, where users > can plugin their python implementation. > > As I understand, the current design relies on SPI (Service Provider > Interface) to load model providers, which requires users to provide a JAR > containing a custom Factory implementation. This raises the question of how > Python users could contribute their own model providers within this > framework. I’d be interested in exploring how we might support such > extensibility for Python-based implementations. > > >> I will refine it and open a FLIP, so we can discuss more details. > > Looking forward to hearing from you. > > Best, > Shengkai > > > Swapna Marru <[email protected]> 于2025年9月24日周三 04:44写道： > > > Hi Shengkai, > > > > Thanks so much for the response and the details. > > > > Enhancements you proposed make perfect sense. > > > > Was thinking along the same lines. I am interested in contributing to > this. > > Have a rough design and a POC implementation around this. > > From my initial digging, we would need to extend > > PythonTableFunctionOperator to encapsulate the Model Context with it. > > I will refine it and open a FLIP, so we can discuss more details. > > >> *Introduce a pluggable Model Provider interface* > > > > Sounds good. Also, to start with, we could also expose a very generic > > python model provider, where users can plugin > > their python implementation. Users can bring in their own generic Python > > implementation of PyPredictFunction. > > >> *Add built-in model providers for PyTorch and TensorFlow in the Flink > > repository* > > > > Yes, I agree, this is very much needed. Also, i see > > > > > https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/dev/python/table/udfs/vectorized_python_udfs/ > > but there is no implementation of Vectorized Table function available > > currently. Providing that seems promising to use resources efficiently. > > >>*Optimize resource usage via cached files and batched inference* > > > > On Mon, Sep 22, 2025 at 7:56 PM Shengkai Fang <[email protected]> wrote: > > > > > Hi, Swapna. > > > > > > Supporting local models has always been a key milestone on our journey > > > toward Flink AI. After an offline discussion with @DianFu, we found > that > > > the current API is not well-suited for PyFlink, as PyFlink prefers to > > > provide its own Python operators. > > > > > > To better accommodate this requirement and improve extensibility, we > > > propose the following enhancements: > > > > > > *Introduce a pluggable Model Provider interface* > > > Extend the model management framework by adding a standardized > interface > > > for model providers, enabling custom implementations — such as a > > > PythonModelProvider — to supply native Python operators (e.g., > > > PythonTableFunctionOperator). > > > > > > *Add built-in model providers for PyTorch and TensorFlow in the Flink > > > repository* > > > Implement and integrate first-party model providers for popular > > frameworks > > > like PyTorch and TensorFlow. These providers will implement the new > > > interface and be maintained within the main Flink codebase. > > > > > > *Optimize resource usage via cached files and batched inference* > > > Leverage `pipeline.cached-files` to manage model weight files > > efficiently, > > > avoiding redundant downloads across tasks. Additionally, support > batched > > > inference to maximize hardware utilization (e.g., GPU/TPU throughput) > and > > > improve overall performance. > > > > > > Best, > > > Shengkai > > > > > > > > > Swapna Marru <[email protected]> 于2025年9月22日周一 23:36写道： > > > > > > > Hi Devs, > > > > > > > > > > > > I am interested in learning more about MODEL, ML_PREDICT, and > > ML_EVALUATE > > > > functionalities added in the following FLIP. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-437%3A+Support+ML+Models+in+Flink+SQL > > > > > > > > > > > > I see the original FLIP has extensibility to local model providers in > > > > Flink. > > > > > > > > > > > > Is there a way to do pluggable local model providers in Python? Like, > > > say, > > > > generate embeddings using Sentence transformer models running locally > > in > > > > Flink. > > > > > > > > > > > > An option could be to introduce a Model Provider factory > implementation > > > in > > > > Java that internally uses a predict function in Python . But I see > this > > > > puts in a lot of work related to Java to Python > > communication/translation > > > > inside the provider. > > > > > > > > > > > > Something like PythonRuntimeProvider along with > PredictRuntimeProvider > > / > > > > AsyncRuntimeProvider which can handle Java -> Python translations out > > of > > > > the box would be helpful to de-duplicate that effort. > > > > > > > > > > > > Can you please point to, if there are any discussions related to this > > > > already ? Or any other ways to achieve the same? Please share your > > > > thoughts. > > > > > > > > > > > > -Thanks, > > > > > > > > Swapna Marru > > > > > > > > > >
