Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/1722#discussion_r15725724
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala
---
@@ -30,8 +30,19 @@ class ExternalAppendOnlyMapSuite extends FunSuite with
LocalSparkContext {
private def mergeValue(buffer: ArrayBuffer[Int], i: Int) = buffer += i
private def mergeCombiners(buf1: ArrayBuffer[Int], buf2:
ArrayBuffer[Int]) = buf1 ++= buf2
+ private def createSparkConf(loadDefaults: Boolean): SparkConf = {
+ val conf = new SparkConf(loadDefaults)
+ // Make the Java serializer write a reset instruction (TC_RESET) after
each object to test
+ // for a bug we had with bytes written past the last object in a batch
(SPARK-2792)
+ conf.set("spark.serializer.objectStreamReset", "0")
+ conf.set("spark.serializer",
"org.apache.spark.serializer.JavaSerializer")
+ // Ensure that we actually have multiple batches per spill file
+ conf.set("spark.shuffle.spill.batchSize", "10")
--- End diff --
Actually I should have made it 0 - to ensure reset after each object (with
the corresponding java serializer change to enable it) is because we want to
ensure that there is a TC_RESET after object write and before close. Else the
bug is not exposed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
