Hi everyone, over the last couple of days Marko and I tried find the best settings for Spark(GraphComputer) [1] when it's used to execute TP3 OLAP computations. Here's what we've used:
- a 4 node (1 master, 3 slaves) cluster running Spark server (24 CPU cores, 62 GB RAM) - TinkerPop3 M8 - Friendster dataset (2.5 billion edges, ~24 GB) [2] - ScriptInputFormat to load the data - g.V().out().count() to run the actual benchmark There were 2 settings which had a significant impact on our benchmark results: - spark.executor.memory - spark.storage.memoryFraction What we've learned is that you get the best performance if 1. the memory (spark.executor.memory * number of slaves) is slightly more then the total size of your input dataset and 2. spark.storage.memoryFraction is set to 0.1 (faster job completion, but higher avg. GC time) or 0.2 (lower avg. GC time, but slower job completion) The following table summarizes our results: Memory | Fraction | Time | Succeeded | GC Time (%) ==========+==============+==========+===========+============ 38g | 0.1 | 2.3h | Yes | 1.89 38g | 0.2 | 2.5h | Yes | 1.24 38g | 0.3 | 2.4h | Yes | 1.26 38g | 0.4 | 3.1h | Yes | 1.27 38g | 0.5 | 5.3h | No | 5.32 ----------+--------------+----------+-----------+------------ 10g | 0.1 | 1.4h | Yes | 2.92 10g | 0.2 | 1.4h | Yes | 2.73 10g | 0.3 | 1.5h | Yes | 2.76 10g | 0.4 | 1.8h | Yes | 2.52 10g | 0.5 | 6.5h | Yes | 2.07 ----------+--------------+----------+-----------+------------ 5g | 0.1 | 2.0h | Yes | 2.54 5g | 0.2 | 2.3h | Yes | 2.03 5g | 0.3 | 2.6h | Yes | 2.06 5g | 0.4 | 3.8h | Yes | 1.81 5g | 0.5 | 2.5h | No | ---- As you can see, a memory fraction >= 0.5 seems to be a bad idea; 2 jobs were not able to finish successfully (they either became unresponsive or ran out of memory). However, I'm stoked about the best job's (highlighted) performance. 1.4 hours to process 2.5 billion edges -- that's approximately 500.000 edges per second or 500 edges per millisecond (!). If you're interested in running this benchmark on your own hardware, drop me a line, I can share the configuration file(s) and the Groovy script that we've used to parse the Friendster input files. [1] http://tinkerpop.incubator.apache.org/docs/3.0.0-SNAPSHOT/#sparkgraphcomputer [2] https://archive.org/details/friendster-dataset-201107 Cheers, Daniel
