Github user maropu commented on the pull request:
https://github.com/apache/spark/pull/9478#issuecomment-160057949
@andrewor14 ISTM that this pr has a little effect on performance even
in case of many partitions involved in shuffle.
Test settings:
- 10 test runs except for the first run
- 4 workers in single c4.8xlarge (each worker has 12g mem)
- spark.driver.memory=1g
- spark.executor.memory=12g
- spark.shuffle.bypassNetworkAccess=true
- spark.sql.shuffle.partitions=1000
- total shuffle size = around 1.5g
- query executed:
```
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val schema = StructType(
Seq(
StructField(s"col0", IntegerType, true),
StructField(s"col1", StringType, true)
)
)
val rdd =
sc.parallelize((1 to 36), 36).flatMap { j => (1 to 5000000).map ( i =>
Row(i, s"${i}"))
}
sqlContext.createDataFrame(rdd, schema).registerTempTable("shuffle")
sqlContext.sql("cache table shuffle")
def timer[R](block: => R): R = {
val t0 = System.nanoTime()
val result = block
val t1 = System.nanoTime()
println("Elapsed time: " + ((t1 - t0 + 0.0) / 1000000000.0)+ "s")
result
}
(0 until 12).map { i =>
timer { sql(s"""select col0, col1 from shuffle cluster by
col0""").queryExecution.executedPlan(1).execute().foreach(_ => {}) }
}
```
- w/o bypassing
Elapsed time: 9.344261745s
Elapsed time: 10.039675768s
Elapsed time: 11.644405718s
Elapsed time: 9.600332115s
Elapsed time: 9.745275262s
Elapsed time: 11.469732737s
Elapsed time: 11.495929827s
Elapsed time: 9.036307879s
Elapsed time: 9.159725732s
Elapsed time: 9.142030108s
- w/ bypassing
Elapsed time: 8.529972448s
Elapsed time: 9.302905215s
Elapsed time: 9.933504313s
Elapsed time: 7.681615877s
Elapsed time: 9.16150229s
Elapsed time: 9.655643012s
Elapsed time: 8.948393461s
Elapsed time: 8.048391038s
Elapsed time: 8.780075475s
Elapsed time: 9.826335759s
I think that this patch might avoid issues caused by netty (i.e., gc
pressures and network troubles) in a corner case though,
there is no big performance gain in this quick benchmarks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
