Github user superbobry commented on a diff in the pull request:
https://github.com/apache/spark/pull/19458#discussion_r146491977
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -100,7 +100,17 @@ private[spark] class DiskBlockManager(conf: SparkConf,
deleteFilesOnStop: Boolea
/** List all the blocks currently stored on disk by the disk manager. */
def getAllBlocks(): Seq[BlockId] = {
- getAllFiles().map(f => BlockId(f.getName))
+ getAllFiles().flatMap { f =>
+ try {
+ Some(BlockId(f.getName))
+ } catch {
+ case _: UnrecognizedBlockId =>
+ // This does not handle a special-case of a temporary file
+ // created by [[SortShuffleWriter]].
+ log.warn(s"Encountered an unexpected file in a managed
directory: $f")
--- End diff --
I am not convinced we should log either. I've added this line because it
was suggested by @jerryshao.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
