Re: How to analyze space usage of Flink algorithms

Thank you for your reply, I'm afraid i still don't understand it, the part i don't understand is how to actually analyze it. It's ok if i can just analyze the system instead of the actual job, but how would i actually do that? I don't have any function in my program that extends the richfunction

Re: How to analyze space usage of Flink algorithms

Hey Fabian, Thanks for the quick reply, I was looking through the flink metrics [1] but i couldn't find anything in there how to analyze the environment from start to finish, only for functions that extend the richmapfunction [1]

How to analyze space usage of Flink algorithms

Currently i'm doing some analysis for some algorithms that i use in Flink, I'm interested in the Space and time it takes to execute them. For the Time i used getNetRuntime() in the executionenvironment, but I have no idea how to analyse the amount of space an algorithm uses. Space can mean

Gelly simple program but: java.lang.RuntimeException: Memory ran out

doing wrong [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/RuntimeException-Gelly-API-Memory-ran-out-Compaction-failed-td866.html [2] http://paste.thezomg.com/19947/14805185/ [3] http://konect.uni-koblenz.de/networks/munmun_twitter_social [4] https://github.com/otherwise777

Re: Executing graph algorithms on Gelly that are larger then memmory

Small addition, i'm currently running the programs via my IDE intelij -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Executing-graph-algorithms-on-Gelly-that-are-larger-then-memmory-tp10358p10359.html Sent from the Apache Flink User Mailing

Executing graph algorithms on Gelly that are larger then memmory

? [1] https://github.com/otherwise777/Temporal_Graph_library/ -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Executing-graph-algorithms-on-Gelly-that-are-larger-then-memmory-tp10358.html Sent from the Apache Flink User Mailing List archive

Re: Type of TypeVariable 'K' in 'class <> could not be determined

The one that's currently in my github will give you the error, In my other file i made a really ugly workaround by adding the element in an ArrayList as a single item. -- View this message in context:

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Sorry i already pushed a new update, But in testclass.java if you change line 266 to: tempgraphdoubles.run(new SingleSourceShortestTemporalPathEAT3(maxIterations)).print(); And then run the testclass.java You should get the error -- View this message in context:

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hey Vasia, I made this simple mapper to illustrate the problem, the file i'm working on is here: https://github.com/otherwise777/Temporal_Graph_library/blob/master/src/main/java/Tgraphs/SingleSourceShortestTemporalPathEATBetweenness.java <https://github.com/otherwise777/Temporal_Graph_libr

Re: Type of TypeVariable 'K' in 'class <> could not be determined

Hello timo, the whole project is on github: https://github.com/otherwise777/Temporal_Graph_library <https://github.com/otherwise777/Temporal_Graph_library> The Tgraphalgorithm is here: https://github.com/otherwise777/Temporal_Graph_library/blob/master/src/main/java/T

Type of TypeVariable 'K' in 'class <> could not be determined

I get this error: *Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class Tgraphs.SingleSourceShortestTemporalPathEAT3$InitVerticesMapper' could not be determined. This is most likely a type erasure problem. The type extraction

Re: 33 segments problem with configuration set

Hello Vasia, thank you for your fast reply, I am aware that determining the betweenness is very demanding, however i still want to give a try at it to a certain extent in Flink, not using Flink is currently not an option since my project is partly about Flink. I will rethink my login, i guess

Re: 33 segments problem with configuration set

Some additional information i just realized, it crashes on this line of code: collectionDataSet.print(); I tried placing it inside of the loop, it crashes at the 7th iteration now -- View this message in context:

33 segments problem with configuration set

Hello Community, I'm trying to make a function to determine the betweenness of the Vertices in a Graph. I'm using Gelly for this and a custom shortestpath function This is my input graph: http://prntscr.com/d7y51y What i've done is use collect() on the vertice

Re: Retrieving values from a dataset of datasets

It seems what i tried did indeed not work. Can you explain me why that doesn't work though? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Retrieving-values-from-a-dataset-of-datasets-tp10108p10128.html Sent from the Apache Flink User

Retrieving values from a dataset of datasets

Hey There, I'm trying to calculate the betweenness in a graph with Flink and Gelly, the way I tried this was by calculating the shortest path from every node to the rest of the nodes. This results in a Dataset of vertices which all have Datasets of their own with all the other vertices and their

Re: Retrieving a single element from a DataSet

Hey there, I don't really understand what Broadcast does, does it in a way export the elements from a DataSet in a Collection? Because then it might be what i'm looking for. when implementing algorithms in Flink Gelly i keep getting stuck on what i cannot do with DataSets. For example, if i want

Re: Retrieving a single element from a DataSet

Cool, thnx for that, I tried searching for it in teh github but couldn't find it, do you have the url by any chance? I'm going to try to implement such an algorithm for temporal graphs -- View this message in context:

Re: Looping over a DataSet and accesing another DataSet

I just found out that I am able to use arrays in tuple values, nvm about that question -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Looping-over-a-DataSet-and-accesing-another-DataSet-tp9778p9850.html Sent from the Apache Flink User

Re: Looping over a DataSet and accesing another DataSet

I did mean the iteratino yes, I currently solved the problem by rewriting the algorithm in gelly's GathersumApply model, thnx for the tips I had another question regarding the original message, about appending items to a list, how would I do that? Because afaik it's not possible to add a list or

Re: Looping over a DataSet and accesing another DataSet

Thank you for your reply, this is new information for me, Regarding the algorithm, i gave it a better look and i don't think it will work with joining. When looping over the Edge set (u,v) we need to be able to write and read A[u] and A[v]. If i join them it will create a new instances of that

Re: Looping over a DataSet and accesing another DataSet

Thank you for your reply and explanation, I think there is one issue with your method though, you said that i should make a join with the the key value pair A on v and the Edge set (u,v), this would work, however i not only need to access A[v] in one iteration but also A[u], so if i join on v

Looping over a DataSet and accesing another DataSet

Currently i'm trying to implement this algorithm [1] which requires me to loop over one DataSet (the edges) and access another DataSet (the vertices), for this loop i use a Mapping (i'm not sure if this is the correct way of looping over a DataSet) but i don't know how to access the elements of

Re: Retrieving a single element from a DataSet

That is indeed not the nice way to do it because it will create an executionplan just to get that value, but it does work, so thnx for that A more concrete example for what i want, In gelly you have the SingleSourceShortestPaths algorith which requires the sourceVertexId, now i want to execute a

Retrieving a single element from a DataSet

I'm currently making a shortest path algorithm in Gelly using DataSets, here's a piece of the code i've started with: public DataSet> ShortestPathsEAT(K startingnode) { DataSet> results = this.getVertices().distinct().map(new MapFunction,

Re: Flink error: Too few memory segments provided

thank you so much, it worked immediately. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-error-Too-few-memory-segments-provided-tp9657p9669.html Sent from the Apache Flink User Mailing List archive. mailing list archive at

Re: Flink error: Too few memory segments provided

I tried increasing the taskmanager.network.numberOfBuffers to 4k and later to 8k, i'm not sure if my configuration file is even read, it's stored inside my IDE as follows: http://prntscr.com/cx0vrx i build the flink program from the IDE and run it. I created several

Flink error: Too few memory segments provided

I got this error in Gelly, which is a result of flink (i believe) Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at

Re: question about making a temporal Graph with Gelly

Hello Greg, So far i've added a Tuple3 in the value field of an edge and that seems to work. However in the end i want to make a library on top of Gelly that supports temporal graphs all together. For that i want to add a temporal edge class to use in the graph but i didn't succeed in doing that,