I think Rostyslav is using a DFS which logs at warn/error if you try to delete a directory that isn't there, so is seeing warning messages that nobody else does
Rostyslav —like I said, i'd be curious as to which DFS/object store you are working with, as it is behaving slightly differently from everyone else's. It sounds like I may need to have a quick chat with them about the merits of running all the Hadoop FS specification tests, if they aren't already. Something like a warning printed on delete(nonexistent path) hints that there may be other differences, which makes me worry about what their rename() does. That's the thing that atomic and speculative work depends on, so everyone needs to understand what's expected there. On 14 Jan 2017, at 19:42, Marcelo Vanzin <van...@cloudera.com<mailto:van...@cloudera.com>> wrote: scala> org.apache.hadoop.fs.FileSystem.getLocal(sc.hadoopConfiguration) res0: org.apache.hadoop.fs.LocalFileSystem = org.apache.hadoop.fs.LocalFileSystem@3f84970b scala> res0.delete(new org.apache.hadoop.fs.Path("/tmp/does-not-exist"), true) res3: Boolean = false Does that explain your confusion? On Sat, Jan 14, 2017 at 11:37 AM, Marcelo Vanzin <van...@cloudera.com<mailto:van...@cloudera.com>> wrote: Are you actually seeing a problem or just questioning the code? I have never seen a situation where there's a failure because of that part of the current code. On Fri, Jan 13, 2017 at 3:24 AM, Rostyslav Sotnychenko <r.sotnyche...@gmail.com<mailto:r.sotnyche...@gmail.com>> wrote: Hi all! I am a bit confused why Spark AM and Client are both trying to delete Staging Directory. https://github.com/apache/spark/blob/branch-2.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110 https://github.com/apache/spark/blob/branch-2.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L233 As you can see, in case if a job was running on YARN in Cluster deployment mode, both AM and Client will try to delete Staging directory if job succeeded and eventually one of them will fail to do this, because the other one already deleted the directory. Shouldn't we add some check to Client? Thanks, Rostyslav -- Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>
