Github user superbobry commented on the issue:
https://github.com/apache/spark/pull/19458
> Instead of filtering out temp blocks, why not adding parsing rule for
`TempLocalBlockId` and `TempShuffleBlockId`?
Because this fixes the issue out 2 out of 3 possible temp files. The
unhandled case is produced by `Utils.tempFileWith`.
> Since `DiskBlockManager#getAllFiles` doesn't filter out temp
shuffle/local files, is it better to keep the same behavior for
`DiskBlockManager#getAllBlocks`?
I agree that it makes sense to keep those in sync, therefore I prefer to
introduce `Block.isValid` and use it in `getAllFiles`.
> Also it would better for upstream code to decide whether to filter out
temp files/blocks.
Possibly, but in any case `getAllBlocks` should not throw, since temp
blocks are an implementation detail.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
