yucai created SPARK-12582:
-----------------------------
Summary: IndexShuffleBlockResolverSuite fails in windows
Key: SPARK-12582
URL: https://issues.apache.org/jira/browse/SPARK-12582
Project: Spark
Issue Type: Bug
Components: Shuffle
Reporter: yucai
IndexShuffleBlockResolverSuite fails in my windows develop machine.
{code}
[info] IndexShuffleBlockResolverSuite:
[info] - commit shuffle files multiple times *** FAILED *** (388 milliseconds)
[info] Array(10, 0, 20) equaled Array(10, 0, 20)
(IndexShuffleBlockResolverSuite.scala:108)
[info] org.scalatest.exceptions.TestFailedException:
.....
.....
[info] Exception encountered when attempting to run a suite with class name:
org.apache.spark.shuffle.sort.IndexShuffleB
lockResolverSuite *** ABORTED *** (2 seconds, 234 milliseconds)
[info] java.io.IOException: Failed to delete:
C:\Users\yyu29\Documents\codes.next\spark\target\tmp\spark-0e81a15a-e712
-4b1c-a089-f421db149e65
[info] at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:940)
[info] at
org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
60)
[info] at
org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:205)
[info] at
org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
36)
[info] at
org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:220)
[info] at
org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
36)
{code}
Root cause is when "afterEach" wants to clean up data, some files are still
open. For example:
{code}
// The dataFile should be the previous one
val in = new FileInputStream(dataFile)
val firstByte = new Array[Byte](1)
in.read(firstByte)
assert(firstByte(0) === 0)
{code}
Lack of "in.close()".
In Linux, it is not a problem, you can still delete a file even it is open, but
this does not work in windows, which will report "resource is busy".
Another issue is this IndexShuffleBlockResolverSuite.scala is a scala file but
it is placed in "test/java".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
