Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6420#discussion_r31100498
--- Diff:
core/src/test/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleSuite.scala ---
@@ -41,6 +41,20 @@ class UnsafeShuffleSuite extends ShuffleSuite with
BeforeAndAfterAll {
conf.set("spark.shuffle.memoryFraction", "0.5")
}
+ override def multipleAttemptConfs: Seq[(String,SparkConf)] = {
+ val kryoAndNoSpillMove = conf.clone()
+ .set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
+ .set("spark.shuffle.unsafe.testing.allowSpillMove", "false")
+ val noCompression = kryoAndNoSpillMove.clone()
+ .set("spark.shuffle.compress", "false")
+ .set("spark.file.transferTo","false")
+ Seq(
+ "slow merge path" -> kryoAndNoSpillMove,
+ "filestream based fast merge" -> noCompression
+ )
--- End diff --
there should be another case here for the `transferTo` path as well, but
that path fails with the msg about "Please check your kernel version to see if
it is 2.6.32". I don't understand the intricacies of when the channel's
position is wrong from the discussion here
https://issues.apache.org/jira/browse/SPARK-3948 so I just left it as is.
(which is another reason to consider an approach where the same files are
written to, so append vs. overwrite doesn't matter)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
