Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/3009#issuecomment-63385969
@JoshRosen I think something like this will do the trick to trigger a
FetchFailedException:
test("hash shuffle manager recovers when local shuffle files get
deleted") {
val conf = new SparkConf(false)
conf.set("spark.shuffle.manager", "hash")
sc = new SparkContext("local", "test", conf)
val rdd = sc.parallelize(1 to 10, 2).map((_, 1)).reduceByKey(_+_)
rdd.count()
// Delete one of the local shuffle blocks.
sc.env.blockManager.diskBlockManager.getFile(new ShuffleBlockId(0, 0,
0)).delete()
rdd.count()
}
You'll need to change this a little bit: I wrote this to demonstrate a
(still outstanding) bug where the wrong exception gets thrown for lost local
blocks (SPARK-4085); until that gets fixed, you'll need to fix this to use
local-cluster rather than local mode.
The new thing for stages looks fine to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
