Hi, thanks Martijn for bringing it up for discussion. I think we could make the discussion a little bit clearer by splitting it into two questions:
1. should Flink drop Gelly? 2. should Flink drop the graph computing? The answer of the first question could be yes, since there have been no changes for years. +1 for dropping Gelly. But for the second question, I would suggest answering it with no or with strategic yes and will definitely support it again in the near future, because there are many use cases that could be solved by streaming/batch + graph in a more elegant way. Afaik fintech companies have a lot of those use cases[1]. It would be great if we could find a way to drop Gelly but keep the graph computing ability within Flink's ecosystem. Best regards Jing [1] https://california18.com/the-ant-graph-calculation-is-upgraded-to-tugraph-and-it-won-the-2021-world-internet-leading-scientific-and-technological-achievement-award/2117052021/ On Tue, Jan 4, 2022 at 2:02 PM Zhipeng Zhang <zhangzhipe...@gmail.com> wrote: > Hi Martijin, > > Thanks for the feedback. I am not proposing to bundle the new graph > library with Alink. I am +1 for dropping the DataSet-based Gelly library, > but we probably need a new graph library in Flink for the possible > migration. > > We haven't decided what to do yet and probably need more discussion. There > are some possible solutions: > 1. We include a new DataStream-based graph library in FlinkML[1], given > that graphs and machine learning algorithms are more often used together > [2][3][4]. To achieve this, we could reuse the `AlgoOperator` interface in > FlinkML. > 2. We include a new DataStream-based graph library as a separate > module/repo. This is consistent with existing libraries like Spark [5]. > > What do you think? > > > [1] https://github.com/apache/flink-ml > [2] https://arxiv.org/abs/1403.6652 > [3] https://arxiv.org/abs/1503.03578 > [4] https://github.com/apache/spark > > Best, > Zhipeng > > Martijn Visser <mart...@ververica.com> 于2022年1月4日周二 15:27写道： > >> Hi Zhipeng, >> >> Good that you've reached out, I wasn't aware that Gelly is being used in >> Alink. Are you proposing to write a new graph library as a successor of >> Gelly and bundle that with Alink? >> >> Best regards, >> >> Martijn >> >> On Tue, 4 Jan 2022 at 02:57, Zhipeng Zhang <zhangzhipe...@gmail.com> >> wrote: >> >>> Hi everyone, >>> >>> Thanks for starting the discussion :) >>> >>> We (Alink team [1]) are actually using part of the Gelly library to >>> support graph algorithms (connected component, single source shortest path, >>> etc.) for users in Alibaba Inc. >>> >>> As DataSet API is going to be dropped, shall we also provide a new graph >>> library based on DataStream runtime (similar as we did for machine >>> learning)? >>> >>> [1] https://github.com/Alibaba/alink >>> >>> David Anderson <dander...@apache.org> 于2022年1月4日周二 00:01写道： >>> >>>> Most of the inquiries I've had about Gelly in recent memory have been >>>> from folks looking for a streaming solution, and it's only been a handful. >>>> >>>> +1 for dropping Gelly >>>> >>>> David >>>> >>>> On Mon, Jan 3, 2022 at 2:41 PM Till Rohrmann <trohrm...@apache.org> >>>> wrote: >>>> >>>>> I haven't seen any changes or requests to/for Gelly in ages. Hence, I >>>>> would assume that it is not really used and can be removed. >>>>> >>>>> +1 for dropping Gelly. >>>>> >>>>> Cheers, >>>>> Till >>>>> >>>>> On Mon, Jan 3, 2022 at 2:20 PM Martijn Visser <mart...@ververica.com> >>>>> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> Flink is bundled with Gelly, a Graph API library [1]. This has been >>>>>> marked as approaching end-of-life for quite some time [2]. >>>>>> >>>>>> Gelly is built on top of Flink's DataSet API, which is deprecated and >>>>>> slowly being phased out [3]. It only works on batch jobs. Based on the >>>>>> activity in the Dev and User mailing lists, I don't see a lot of >>>>>> questions >>>>>> popping up regarding the usage of Gelly. Removing Gelly would reduce CI >>>>>> time and resources because we won't need to run tests for this anymore. >>>>>> >>>>>> I'm cross-posting this to the User mailing list to see if there are >>>>>> any users of Gelly at the moment. >>>>>> >>>>>> Let me know your thoughts. >>>>>> >>>>>> Martijn Visser | Product Manager >>>>>> >>>>>> mart...@ververica.com >>>>>> >>>>>> >>>>>> [1] >>>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/libs/gelly/overview/ >>>>>> >>>>>> [2] https://flink.apache.org/roadmap.html >>>>>> >>>>>> [3] https://lists.apache.org/thread/b2y3xx3thbcbtzdphoct5wvzwogs9sqz >>>>>> >>>>>> <https://www.ververica.com/> >>>>>> >>>>>> >>>>>> Follow us @VervericaData >>>>>> >>>>>> -- >>>>>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>>>>> Conference >>>>>> >>>>>> Stream Processing | Event Driven | Real Time >>>>>> >>>>>> >>> >>> -- >>> best, >>> Zhipeng >>> >>> > > -- > best, > Zhipeng > >
