I have tested on another pc which has 8 CPU cores.
But it hangs when defaultParallelismLevel > 4, e.g.
sparkConf.setMaster("local[*]")
local[1] ~ local[3] work well.
4 is the mysterious boundary.
It seems that I am not the only one encountered this problem:
https://issues.apache.org/jira/browse/S
Given the following code which just reads from s3, then saves files to s3
val inputFileName: String = "s3n://input/file/path"
val outputFileName: String = "s3n://output/file/path"
val conf = new
SparkConf().setAppName(this.getClass.getName).setMaster("local[4]")
val
