Hi,
Daniel -- can you talk about how long each "sub stage" takes for the 1.4 hour
run? For instance:
1. Loading the graphRDD from Hadoop.
2. The out()
3. The count()
4. The saveNewHadoopAPI of results.
Thanks,
Marko.
http://markorodriguez.com
On Apr 14, 2015, at 9:27 AM, Daniel Kuppitz <[email protected]> wrote:
> Hi everyone,
>
> over the last couple of days Marko and I tried find the best settings for
> Spark(GraphComputer) [1] when it's used to execute TP3 OLAP computations.
> Here's what we've used:
> a 4 node (1 master, 3 slaves) cluster running Spark server (24 CPU cores, 62
> GB RAM)
> TinkerPop3 M8
> Friendster dataset (2.5 billion edges, ~24 GB) [2]
> ScriptInputFormat to load the data
> g.V().out().count() to run the actual benchmark
>
> There were 2 settings which had a significant impact on our benchmark results:
> spark.executor.memory
> spark.storage.memoryFraction
>
> What we've learned is that you get the best performance if
> the memory (spark.executor.memory * number of slaves) is slightly more then
> the total size of your input dataset and
> spark.storage.memoryFraction is set to 0.1 (faster job completion, but higher
> avg. GC time) or 0.2 (lower avg. GC time, but slower job completion)
>
> The following table summarizes our results:
>
> Memory | Fraction | Time | Succeeded | GC Time (%)
> ==========+==============+==========+===========+============
> 38g | 0.1 | 2.3h | Yes | 1.89
> 38g | 0.2 | 2.5h | Yes | 1.24
> 38g | 0.3 | 2.4h | Yes | 1.26
> 38g | 0.4 | 3.1h | Yes | 1.27
> 38g | 0.5 | 5.3h | No | 5.32
> ----------+--------------+----------+-----------+------------
> 10g | 0.1 | 1.4h | Yes | 2.92
> 10g | 0.2 | 1.4h | Yes | 2.73
> 10g | 0.3 | 1.5h | Yes | 2.76
> 10g | 0.4 | 1.8h | Yes | 2.52
> 10g | 0.5 | 6.5h | Yes | 2.07
> ----------+--------------+----------+-----------+------------
> 5g | 0.1 | 2.0h | Yes | 2.54
> 5g | 0.2 | 2.3h | Yes | 2.03
> 5g | 0.3 | 2.6h | Yes | 2.06
> 5g | 0.4 | 3.8h | Yes | 1.81
> 5g | 0.5 | 2.5h | No | ----
>
> As you can see, a memory fraction >= 0.5 seems to be a bad idea; 2 jobs were
> not able to finish successfully (they either became unresponsive or ran out
> of memory). However, I'm stoked about the best job's (highlighted)
> performance. 1.4 hours to process 2.5 billion edges -- that's approximately
> 500.000 edges per second or 500 edges per millisecond (!).
>
> If you're interested in running this benchmark on your own hardware, drop me
> a line, I can share the configuration file(s) and the Groovy script that
> we've used to parse the Friendster input files.
>
> [1]
> http://tinkerpop.incubator.apache.org/docs/3.0.0-SNAPSHOT/#sparkgraphcomputer
> [2] https://archive.org/details/friendster-dataset-201107
>
> Cheers,
> Daniel
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gremlin-users/CA%2Bf9seXo87EakoLR_50q0RoAa8hArZ6kBh4SVwHQ18Y5b6dWyQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
