GitHub user ming616 opened a pull request:
https://github.com/apache/spark/pull/16206
Branch 2.0
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-2.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16206
commit fffcec90b65047c3031c2b96679401f8fbef6337
Author: Shixiong Zhu
Date: 2016-09-14T20:33:51Z
[SPARK-17463][CORE] Make CollectionAccumulator and SetAccumulator's value
can be read thread-safely
## What changes were proposed in this pull request?
Make CollectionAccumulator and SetAccumulator's value can be read
thread-safely to fix the ConcurrentModificationException reported in
[JIRA](https://issues.apache.org/jira/browse/SPARK-17463).
## How was this patch tested?
Existing tests.
Author: Shixiong Zhu
Closes #15063 from zsxwing/SPARK-17463.
(cherry picked from commit e33bfaed3b160fbc617c878067af17477a0044f5)
Signed-off-by: Josh Rosen
commit bb2bdb44032d2e71832b3e0e771590fb2225e4f3
Author: Xing SHI
Date: 2016-09-14T20:46:46Z
[SPARK-17465][SPARK CORE] Inappropriate memory management in
`org.apache.spark.storage.MemoryStore` may lead to memory leak
The expression like `if (memoryMap(taskAttemptId) == 0)
memoryMap.remove(taskAttemptId)` in method `releaseUnrollMemoryForThisTask` and
`releasePendingUnrollMemoryForThisTask` should be called after release memory
operation, whatever `memoryToRelease` is > 0 or not.
If the memory of a task has been set to 0 when calling a
`releaseUnrollMemoryForThisTask` or a `releasePendingUnrollMemoryForThisTask`
method, the key in the memory map corresponding to that task will never be
removed from the hash map.
See the details in
[SPARK-17465](https://issues.apache.org/jira/browse/SPARK-17465).
Author: Xing SHI
Closes #15022 from saturday-shi/SPARK-17465.
commit 5c2bc8360019fb08e2e62e50bb261f7ce19b231e
Author: codlife <1004910...@qq.com>
Date: 2016-09-15T08:38:13Z
[SPARK-17521] Error when I use sparkContext.makeRDD(Seq())
## What changes were proposed in this pull request?
when i use sc.makeRDD below
```
val data3 = sc.makeRDD(Seq())
println(data3.partitions.length)
```
I got an error:
Exception in thread "main" java.lang.IllegalArgumentException: Positive
number of slices required
We can fix this bug just modify the last line ,do a check of seq.size
```
def makeRDD[T: ClassTag](seq: Seq[(T, Seq[String])]): RDD[T] = withScope {
assertNotStopped()
val indexToPrefs = seq.zipWithIndex.map(t => (t._2, t._1._2)).toMap
new ParallelCollectionRDD[T](this, seq.map(_._1), math.max(seq.size,
defaultParallelism), indexToPrefs)
}
```
## How was this patch tested?
manual tests
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Author: codlife <1004910...@qq.com>
Author: codlife
Closes #15077 from codlife/master.
(cherry picked from commit 647ee05e5815bde361662a9286ac602c44b4d4e6)
Signed-off-by: Sean Owen
commit a09c258c9a97e701fa7650cc0651e3c6a7a1cab9
Author: junyangq
Date: 2016-09-15T17:00:36Z
[SPARK-17317][SPARKR] Add SparkR vignette to branch 2.0
## What changes were proposed in this pull request?
This PR adds SparkR vignette to branch 2.0, which works as a friendly
guidance going through the functionality provided by SparkR.
## How was this patch tested?
R unit test.
Author: junyangq
Author: Shivaram Venkataraman
Author: Junyang Qian
Closes #15100 from junyangq/SPARKR-vignette-2.0.
commit e77a437d292ecda66163a895427d62e4f72e2a25
Author: Josh Rosen
Date: 2016-09-15T18:22:58Z
[SPARK-17547] Ensure temp shuffle data file is cleaned up after error
SPARK-8029 (#9610) modified shuffle writers to first stage their data to a
temporary file in the same directory as the final destination file and then to
atomically rename this temporary file at the end of the write job. However,
this change introduced the potential for the temporary output file to be leaked
if an exception occurs during the write because the shuff