GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/4105
[SPARK-5316] [CORE] DAGScheduler may make shuffleToMapStage leak if
getParentStages failes
DAGScheduler may make shuffleToMapStage leak if getParentStages failes.
If getParentStages has exception for example input path does not exist,
DAGScheduler would fail to handle job submission, while shuffleToMapStage may
be put some records when getParentStages. However these records in
shuffleToMapStage aren't going to be cleaned.
A simple job as follows:
```
val inputFile1 = ... // Input path does not exist when this job submits
val inputFile2 = ...
val outputFile = ...
val conf = new SparkConf()
val sc = new SparkContext(conf)
val rdd1 = sc.textFile(inputFile1)
.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _, 1)
val rdd2 = sc.textFile(inputFile2)
.flatMap(line => line.split(","))
.map(word => (word, 1))
.reduceByKey(_ + _, 1)
try {
val rdd3 = new PairRDDFunctions(rdd1).join(rdd2, 1)
rdd3.saveAsTextFile(outputFile)
} catch {
case e : Exception =>
logError(e)
}
// print the information of DAGScheduler's shuffleToMapStage to check
// whether it still has uncleaned records.
...
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YanTangZhai/spark SPARK-5316
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4105.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4105
----
commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6
Author: YanTangZhai <[email protected]>
Date: 2014-08-06T13:07:08Z
Merge pull request #1 from apache/master
update
commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95
Author: YanTangZhai <[email protected]>
Date: 2014-08-20T13:14:08Z
Merge pull request #3 from apache/master
Update
commit 8a0010691b669495b4c327cf83124cabb7da1405
Author: YanTangZhai <[email protected]>
Date: 2014-09-12T06:54:58Z
Merge pull request #6 from apache/master
Update
commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3
Author: YanTangZhai <[email protected]>
Date: 2014-09-16T12:03:22Z
Merge pull request #7 from apache/master
Update
commit 76d40277d51f709247df1d3734093bf2c047737d
Author: YanTangZhai <[email protected]>
Date: 2014-10-20T12:52:22Z
Merge pull request #8 from apache/master
update
commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a
Author: YanTangZhai <[email protected]>
Date: 2014-11-04T09:00:31Z
Merge pull request #9 from apache/master
Update
commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a
Author: YanTangZhai <[email protected]>
Date: 2014-11-11T03:18:24Z
Merge pull request #10 from apache/master
Update
commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588
Author: YanTangZhai <[email protected]>
Date: 2014-12-01T11:23:56Z
Merge pull request #11 from apache/master
Update
commit 718afebe364bd54ac33be425e24183eb1c76b5d3
Author: YanTangZhai <[email protected]>
Date: 2014-12-05T11:08:31Z
Merge pull request #12 from apache/master
update
commit e4c2c0a18bdc78cc17823cbc2adf3926944e6bc5
Author: YanTangZhai <[email protected]>
Date: 2014-12-24T03:15:22Z
Merge pull request #15 from apache/master
update
commit d4bca32bf4b06d3694a5de3cf5b69bac606dda39
Author: YanTangZhai <[email protected]>
Date: 2014-12-31T03:50:26Z
Merge pull request #19 from apache/master
Update
commit 5041b3574dc89cd1e8a8d46590d2aba4c050de92
Author: YanTangZhai <[email protected]>
Date: 2015-01-12T12:33:20Z
Merge pull request #24 from apache/master
update
commit e2880f919dd54b43e0c53657a0f2d02880f47aa3
Author: YanTangZhai <[email protected]>
Date: 2015-01-19T09:14:27Z
Merge pull request #27 from apache/master
Update
commit 50291ca23192b3f05f572a60f68fcae0b66d5ffd
Author: yantangzhai <[email protected]>
Date: 2015-01-19T11:12:16Z
[SPARK-5316] [CORE] DAGScheduler may make shuffleToMapStage leak if
getParentStages failes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
