Github user frreiss commented on the issue:
https://github.com/apache/spark/pull/15027
When I comment out line 155 in HDFSMetadataLog.scala on this branch (`if
(fileManager.exists(crcPath)) fileManager.delete(crcPath)`) and run the test
case attached to this PR, the test case fails:
```
[freiss@fuzzy]:~/spark/from_git/spark-17475$ build/sbt -Phadoop-2.7
-Pscala-2.11 "test-only
org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite"
...
[info] - HDFSMetadataLog: purge *** FAILED *** (135 milliseconds)
[info]
Array(/Users/freiss/spark/from_git/spark-17475/target/tmp/spark-682aa8da-3f04-494f-846f-13c97d3e5538/..29ef67f7-1712-4350-8552-1f8bc6424d0b.tmp.crc,
/Users/freiss/spark/from_git/spark-17475/target/tmp/spark-682aa8da-3f04-494f-846f-13c97d3e5538/..ab9bafcb-bdf5-4411-9a9b-60d293d653a6.tmp.crc,
/Users/freiss/spark/from_git/spark-17475/target/tmp/spark-682aa8da-3f04-494f-846f-13c97d3e5538/..f79fbc34-58c5-4856-a40d-84eef49c8b9e.tmp.crc,
/Users/freiss/spark/from_git/spark-17475/target/tmp/spark-682aa8da-3f04-494f-846f-13c97d3e5538/2)
had size 4 instead of expected size 1 (HDFSMetadataLogSuite.scala:126)
[info] org.scalatest.exceptions.TestFailedException:
[info] at
org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
...
```
Note how the test case failure message contains a list of orphan .crc files
left in HDFSMetadataLog's temp directory. So, while the filesystem code appears
on the surface to be immune to this problem, the problem is clearly happening
in the context of the unit tests. Determining exactly what is going on will
require more in-depth investigation. Depending on the true root cause, it's
possible that this problem also occurs in some distributed settings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
