Accidentally to get it working, though don't thoroughly understand why (So far
as I know, it's to configure in allowing executor refers to the conf file after
copying to executors' working dir). Basically it's a combination of parameters
--conf, --files, and --driver-class-path, instead of any single parameter.
spark-submit --class pkg.to.MyApp --master local[*] --conf
"spark.executor.extraClassPath=-Dconfig.file=" --files
--driver-class-path ""
--conf requires to pass the conf file name e.g. myfile.conf along with spark
executor class path as directive.
--files passes the conf file associated from the context root e.g. executing
under dir , under which it contains folders such as conf, logs,
work and so on. The conf file i.e. myfile.conf is located under conf folder.
--driver-class-path points to the conf directory with absolute path.
‐‐‐ Original Message ‐‐‐
On August 17, 2018 3:00 AM, yujhe.li wrote:
> So can you read the file on executor side?
> I think the file passed by --files my.app.conf would be added under
> classpath, and you can use it directly.
>
>
>
>
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
>
>
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
