On 16 Jan 2017, at 12:51, Rostyslav Sotnychenko <r.sotnyche...@gmail.com<mailto:r.sotnyche...@gmail.com>> wrote:
Thanks all! I was using another DFS instead of HDFS, which was logging an error when fs.delete got called on non-existing path. really? Whose DFS, if you don't mind me asking? I'm surprised they logged that delete() of a missing path, as it's not entirely uncommon to happen during cleanup In Spark 2.0.1 which I was using previously, everything was working fine because existence of an additional check that was made prior to deleting. However that check got removed in 2.1 (SPARK-16736<https://issues.apache.org/jira/browse/SPARK-16736>, commit<https://github.com/apache/spark/pull/14371/files#diff-b050df3f55b82065803d6e83453b9706>), so I started seeing an error from my DFS. Its not a problem in any way (i.e. it does not affect Spark job in any way), so everything is fine. I just wanted to make sure its not a Spark issue. Thanks, Rostyslav No, not a problem. Really that whole idea of having delete() return true/false is pretty useless: nobody really knows what it means when it returns false. It should just have been void() wth an exception thrown if something actually failed. That's essentially what they all do, though I've never seen any which complains about the situation. mkdirs(), now there's one to fear. Not even the java.io<http://java.io> API clearly defines what "false" coming back from there means, as it can mean both "ther'es a directory there, so I didnt' do any work", or "there's a file/symlnk/mount point/device there which is a probably a serious problem"
