addFile is supposed to not depend on a shared FS unless the semantics have changed recently.
On Sat, Jun 24, 2017 at 11:55 AM varma dantuluri <dvsnva...@gmail.com> wrote: > Hi Sudhir, > > I believe you have to use a shared file system that is accused by all > nodes. > > > On Jun 24, 2017, at 1:30 PM, sudhir k <k.sudhi...@gmail.com> wrote: > > > I am new to Spark and i need some guidance on how to fetch files from > --files option on Spark-Submit. > > I read on some forums that we can fetch the files from > Spark.getFiles(fileName) and can use it in our code and all nodes should > read it. > > But i am facing some issue > > Below is the command i am using > > spark-submit --deploy-mode cluster --class com.check.Driver --files > /home/sql/first.sql test.jar 20170619 > > so when i use SparkFiles.get(first.sql) , i should be able to read the > file Path but it is throwing File not Found exception. > > I tried SpackContext.addFile(/home/sql/first.sql) and then > SparkFiles.get(first.sql) but still the same error. > > Its working on the stand alone mode but not on cluster mode. Any help is > appreciated.. Using Spark 2.1.0 and Scala 2.11 > > Thanks. > > > Regards, > Sudhir K > > > > -- > Regards, > Sudhir K > > > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
