Re: Can we access files on Cluster mode

Thank you . I guess I have to use common mount or s3 to access those files. On Sun, Jun 25, 2017 at 4:42 AM Mich Talebzadeh wrote: > Thanks. In my experience certain distros like Cloudera only support yarn > client mode so AFAIK the driver stays on the Edge node.

Re: Can we access files on Cluster mode

Thanks. In my experience certain distros like Cloudera only support yarn client mode so AFAIK the driver stays on the Edge node. Happy to be corrected :) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Can we access files on Cluster mode

Hi Mich, If the driver starts on the edge node with cluster mode, then I don't see the difference between client and cluster deploy mode. In cluster mode, it is the responsibility of the resource manager (yarn, etc) to decide where to run the driver (at least for spark 1.6 this is what I have

Re: Can we access files on Cluster mode

Hi Anastasios. Are you implying that in Yarn cluster mode even if you submit your Spark application on an Edge node the driver can start on any node. I was under the impression that the driver starts from the Edge node? and the executors can be on any node in the cluster (where Spark agents are

Re: Can we access files on Cluster mode

Just to note that in cluster mode the spark driver might run on any node of the cluster, hence you need to make sure that the file exists on *all* nodes. Push the file on all nodes or use client deploy-mode. Best, Anastasios Am 24.06.2017 23:24 schrieb "Holden Karau" : >

Re: Can we access files on Cluster mode

addFile is supposed to not depend on a shared FS unless the semantics have changed recently. On Sat, Jun 24, 2017 at 11:55 AM varma dantuluri wrote: > Hi Sudhir, > > I believe you have to use a shared file system that is accused by all > nodes. > > > On Jun 24, 2017, at

Re: Can we access files on Cluster mode

Hi Sudhir, I believe you have to use a shared file system that is accused by all nodes. > On Jun 24, 2017, at 1:30 PM, sudhir k wrote: > > > I am new to Spark and i need some guidance on how to fetch files from --files > option on Spark-Submit. > > I read on some