Hi Niklas, We dropped the Flink ML lib in 1.9 and plan to replace it with a new machine learning library for traditional machine learning algorithms. And that library will be based on FLIP-39. The plan was pushed back a little bit because we plan to deprecate DataSet API and but haven't got the batch iteration support in DataStream API yet. So at this point we don't have an ML lib implementation in Flink.
That being said, we are working with the community to add some ML related features to Flink. At this point, we have the following two projects available from Alibaba that will likely be contributed to Flink. You may also take a look at them. Alink - A machine learning library. https://github.com/alibaba/alink Flink-AI-Extended - A project helps running TF / PyTorch on top of Flink. https://github.com/alibaba/flink-ai-extended Thanks, Jiangjie (Becket) Qin On Fri, Nov 20, 2020 at 3:43 AM Arvid Heise <ar...@ververica.com> wrote: > Hi Niklas, > > indeed some efforts on the machine learning libraries are pushed back in > favor of getting proper PyTorch and Tensorflow support through PyFlink. > > Native implementations in Flink have been done so far in the DataSet API, > which is going to deprecated in the next few releases in favor of the > unified DataStream API with bounded streams. I expect efforts for native > implementations to be picked up once DataSet is fully replaced to avoid > doubling the work. One of the most important features that is lacking is > proper iteration support in DataStream. > > On Thu, Nov 19, 2020 at 1:34 PM Niklas Wilcke <niklas.wil...@uniberg.com> > wrote: > >> Hi Flink-Community, >> >> I'm digging through the history of FlinkML and FLIP-39 [0]. What I >> understood so far is that FlinkML has been removed in 1.9, because it got >> unmaintained. >> I'm not really able to find out whether FLIP-39 and providing a >> replacement for FlinkML is currently worked on. The Umbrella Jira Ticket >> FLINK-12470 [1] looks stale to me. >> Was there maybe a change of strategy in the meantime? Is the focus >> currently on PyFlink to provide ML-Solutions (FLIP-96 [2])? >> It would be really interesting to get some insights about the future and >> roadmap of ML in the Flink ecosystem. Thank you very much! >> >> Kind Regards, >> Niklas >> >> [0] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs >> [1] https://issues.apache.org/jira/browse/FLINK-12470 >> [2] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API > > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
