Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/9214#discussion_r42904139
--- Diff:
core/src/test/scala/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriterSuite.scala
---
@@ -121,12 +140,16 @@ class BypassMergeSortShuffleWriterSuite extends
SparkFunSuite with BeforeAndAfte
taskContext,
conf
)
- writer.write(Iterator.empty)
+ val files = writer.write(Iterator.empty)
writer.stop( /* success = */ true)
+ assert(ShuffleOutputCoordinator.commitOutputs(0, 0, files))
assert(writer.getPartitionLengths.sum === 0)
- assert(outputFile.exists())
- assert(outputFile.length() === 0)
- assert(temporaryFilesCreated.isEmpty)
+ if (outputFile.exists()) {
+ assert(outputFile.length() === 0)
+ assert(temporaryFilesCreated.size === 2)
+ } else {
+ assert(temporaryFilesCreated.size === 1)
+ }
--- End diff --
@JoshRosen I'm pretty sure that `BypassMergeSortShuffleWriter` actually
wasn't writing any data file if there was an empty iterator. This test was
passing just b/c the test setup created the file w/ `File.createTempFile`. So
the change here brings it into line with the implementation (either write no
file at all, or write a zero-length file) but given my comment on
non-determinism I will change this to always write a zero length file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
