This is an interesting one. I have never tried to add --files ...
spark-submit --master yarn --deploy-mode client --files /etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml Rather, under $SPARK_HOME/conf, I create soft links to the needed XML files as below /d4T/hduser/spark-3.1.1-bin-hadoop3.2/conf> ls -lhaF | grep ^l lrwxrwxrwx 1 hduser hadoop 50 Mar 3 08:08 core-site.xml -> /home/hduser/hadoop-3.1.0/etc/hadoop/core-site.xml lrwxrwxrwx 1 hduser hadoop 45 Mar 3 08:07 hbase-site.xml -> /data6/hduser/hbase-1.2.6/conf/hbase-site.xml lrwxrwxrwx 1 hduser hadoop 50 Mar 3 08:08 hdfs-site.xml -> /home/hduser/hadoop-3.1.0/etc/hadoop/hdfs-site.xml lrwxrwxrwx 1 hduser hadoop 43 Mar 3 08:07 hive-site.xml -> /data6/hduser/hive-3.0.0/conf/hive-site.xml This works HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Sat, 15 May 2021 at 18:32, KhajaAsmath Mohammed <mdkhajaasm...@gmail.com> wrote: > Thanks everyone. I was able to resolve this. > > Here is what I did. Just passed conf file using —files option. > > Mistake that I did was reading the json conf file before creating spark > session . Reading if after creating spark session helped it. Thanks once > again for your valuable suggestions > > Thanks, > Asmath > > On May 15, 2021, at 8:12 AM, Sean Owen <sro...@gmail.com> wrote: > > > If code running on the executors need some local file like a config file, > then it does have to be passed this way. That much is normal. > > On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta <gourav.sengu...@gmail.com> > wrote: > >> Hi, >> >> once again lets start with the requirement. Why are you trying to pass >> xml and json files to SPARK instead of reading them in SPARK? >> Generally when people pass on files they are python or jar files. >> >> Regards, >> Gourav >> >
