If your graphs fit in memory (at least after an initial partitioning),
you could use any external library for graph processing within a single
node in a Flink ProcessFunction.
Flink is a general data processor that allows to have arbitrary logic
where user code is allowed.
Regards,
Timo
On 10.09.21 15:13, Dipanjan Mazumder wrote:
Good point what is the better option for graph processing with flink..
any suggestions
On Friday, September 10, 2021, 04:52:30 PM GMT+5:30, Martijn Visser
<mart...@ververica.com> wrote:
Hi,
Please keep in mind that Gelly is approaching end-of-life [1]
Regards,
Martijn
[1] https://flink.apache.org/roadmap.html
<https://flink.apache.org/roadmap.html>
On Fri, 10 Sept 2021 at 09:09, Dipanjan Mazumder <java...@yahoo.com
<mailto:java...@yahoo.com>> wrote:
Hi Jing,
Thanks for the input another question i had was can Gelly be
used for processing the graph that flink receives through kafka and
then using Gelly i decompose the graph into its nodes and edges and
then process them individually through substreams and then write the
final output of processing the graph somewhere.
I saw Gelly is for batch processing but had this question if it
supports above , it will solve my entire use case.
Regards
Dipanjan
On Friday, September 10, 2021, 09:50:08 AM GMT+5:30, JING ZHANG
<beyond1...@gmail.com <mailto:beyond1...@gmail.com>> wrote:
Hi Dipanjan,
Base your description, I think Flink could handle this user case.
Don't worry that Flink can't handle this kind of data scale because
Flink is a distributed engine. As long as the problem of data skew
is carefully avoided, the input throughput can be handled through
appropriate resources.
Best,
JING ZHANG
Dipanjan Mazumder <java...@yahoo.com <mailto:java...@yahoo.com>> 于
2021年9月10日周五 上午11:11写道：
Hi,
I am working on a usecase and thinking of using flink for
the same. The use case is i will be having many large resource
graphs , i need to parse that graph for each node and edge and
evaluate each one of them against some suddhi rules , right now
the implementation for evaluating individual entities with flink
and siddhi are in place , but i am in dilemma whether i should
do the graph processing as well in flink or not.
So this is what i am planning to do
From kafka will fetch the graph , decompose the graph into
nodes and edges , fetch additional meradata for each node and
edge from different Rest API’s and then pass the individual
nodes and edges which are resources to different substreams
which are already inplace and rules will work on individual
substreams to process individual nodes and edges and finally
they will spit the rule output into a stream. I will collate all
of them based on the graph id from that stream using another
operator and send the final result to an outputstream.
This is what i am thinking , now need input from all of you
whether this is a fair usecase to do with flink , will flink be
able to handle this level of processing at scale and volume or not.
Any help input will ease my understanding and will help me go
ahead with this idea.
Regard
dipanjan
