Re: Strongly Connected Components

2016-11-13 Thread Nicholas Chammas
016 at 7:58 PM, Daniel Darabos < > daniel.dara...@lynxanalytics.com> wrote: > > Hi Shreya, > GraphFrames just calls the GraphX strongly connected components code. ( > https://github.com/graphframes/graphframes/blob/release-0.2.0/src/main/scala/org/graphframes/lib/StronglyCo

Re: Strongly Connected Components

2016-11-12 Thread Koert Kuipers
daniel.dara...@lynxanalytics.com> wrote: > >> Hi Shreya, >> GraphFrames just calls the GraphX strongly connected components code. ( >> https://github.com/graphframes/graphframes/blob/release-0. >> 2.0/src/main/scala/org/graphframes/lib/StronglyConnec >> tedCom

Re: Strongly Connected Components

2016-11-12 Thread Koert Kuipers
the exact same algo as is implemented in graphx i think. or is it not? On Fri, Nov 11, 2016 at 7:58 PM, Daniel Darabos < daniel.dara...@lynxanalytics.com> wrote: > Hi Shreya, > GraphFrames just calls the GraphX strongly connected components code. ( > https://github.com/graphframes/gr

RE: Strongly Connected Components

2016-11-11 Thread Shreya Agarwal
g; Denny Lee <denny@microsoft.com> Subject: Re: Strongly Connected Components Hi Shreya, GraphFrames just calls the GraphX strongly connected components code. (https://github.com/graphframes/graphframes/blob/release-0.2.0/src/main/scala/org/graphframes/lib/StronglyConnectedCo

Re: Strongly Connected Components

2016-11-11 Thread Daniel Darabos
Hi Shreya, GraphFrames just calls the GraphX strongly connected components code. ( https://github.com/graphframes/graphframes/blob/release-0.2.0/src/main/scala/org/graphframes/lib/StronglyConnectedComponents.scala#L51 ) For choosing the number of iterations: If the number of iterations is less

RE: Strongly Connected Components

2016-11-11 Thread Shreya Agarwal
ct: RE: Strongly Connected Components Yesterday's run died sometime during the night, without any errors. Today, I am running it using GraphFrames instead. It is still spawning new tasks, so there is progress. From: Felix Cheung [mailto:felixcheun...@hotmail.com] Sent: Thursday, November 10, 2016 7

RE: Strongly Connected Components

2016-11-10 Thread Shreya Agarwal
; Shreya Agarwal <shrey...@microsoft.com> Subject: Re: Strongly Connected Components It is possible it is dead. Could you check the Spark UI to see if there is any progress? _ From: Shreya Agarwal <shrey...@microsoft.com<mailto:shrey...@microsoft.com>&

Re: Strongly Connected Components

2016-11-10 Thread Felix Cheung
It is possible it is dead. Could you check the Spark UI to see if there is any progress? _ From: Shreya Agarwal <shrey...@microsoft.com<mailto:shrey...@microsoft.com>> Sent: Thursday, November 10, 2016 12:45 AM Subject: RE: Strongly Connected Componen

RE: Strongly Connected Components

2016-11-10 Thread Shreya Agarwal
Bump. Anyone? Its been running for 10 hours now. No results. From: Shreya Agarwal Sent: Tuesday, November 8, 2016 9:05 PM To: user@spark.apache.org Subject: Strongly Connected Components Hi, I am running this on a graph with >5B edges and >3B edges and have 2 questions - 1.

Strongly Connected Components

2016-11-08 Thread Shreya Agarwal
Hi, I am running this on a graph with >5B edges and >3B edges and have 2 questions - 1. What is the optimal number of iterations? 2. I am running it for 1 iteration right now on a beefy 100 node cluster, with 300 executors each having 30GB RAM and 5 cores. I have persisted the graph to