> I think Rostyslav is using a DFS which logs at warn/error if you try to
delete a directory that isn't there, so is seeing warning messages that
nobody else does
Yep, you are correct.
> Rostyslav —like I said, i'd be curious as to which DFS/object store you
are working with
Unfortunately, I am not able to share this information :(
> It sounds like I may need to have a quick chat with them about the merits
of running all the Hadoop FS specification tests
I already send the link to Hadoop docs you mentioned to appropriate people,
so the difference could be fixed.
Thanks,
Rostyslav
On Tue, Jan 17, 2017 at 1:58 PM, Steve Loughran <ste...@hortonworks.com>
wrote:
> I think Rostyslav is using a DFS which logs at warn/error if you try to
> delete a directory that isn't there, so is seeing warning messages that
> nobody else does
>
> Rostyslav —like I said, i'd be curious as to which DFS/object store you
> are working with, as it is behaving slightly differently from everyone
> else's.
>
> It sounds like I may need to have a quick chat with them about the merits
> of running all the Hadoop FS specification tests, if they aren't already.
> Something like a warning printed on delete(nonexistent path) hints that
> there may be other differences, which makes me worry about what their
> rename() does. That's the thing that atomic and speculative work depends
> on, so everyone needs to understand what's expected there.
>
>
>
> On 14 Jan 2017, at 19:42, Marcelo Vanzin <van...@cloudera.com> wrote:
>
> scala> org.apache.hadoop.fs.FileSystem.getLocal(sc.hadoopConfiguration)
> res0: org.apache.hadoop.fs.LocalFileSystem =
> org.apache.hadoop.fs.LocalFileSystem@3f84970b
>
> scala> res0.delete(new org.apache.hadoop.fs.Path("/tmp/does-not-exist"),
> true)
> res3: Boolean = false
>
> Does that explain your confusion?
>
>
> On Sat, Jan 14, 2017 at 11:37 AM, Marcelo Vanzin <van...@cloudera.com>
> wrote:
>
> Are you actually seeing a problem or just questioning the code?
>
> I have never seen a situation where there's a failure because of that
> part of the current code.
>
> On Fri, Jan 13, 2017 at 3:24 AM, Rostyslav Sotnychenko
> <r.sotnyche...@gmail.com> wrote:
>
> Hi all!
>
> I am a bit confused why Spark AM and Client are both trying to delete
> Staging Directory.
>
> https://github.com/apache/spark/blob/branch-2.1/yarn/
> src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110
> https://github.com/apache/spark/blob/branch-2.1/yarn/
> src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L233
>
> As you can see, in case if a job was running on YARN in Cluster deployment
> mode, both AM and Client will try to delete Staging directory if job
> succeeded and eventually one of them will fail to do this, because the
> other
> one already deleted the directory.
>
> Shouldn't we add some check to Client?
>
>
> Thanks,
> Rostyslav
>
>
>
>
> --
> Marcelo
>
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>
