Ellison Anne Williams created PIRK-62:
-----------------------------------------
Summary: Local Properties Files Not Picked Up by Responder Using
spark-submit
Key: PIRK-62
URL: https://issues.apache.org/jira/browse/PIRK-62
Project: PIRK
Issue Type: Bug
Reporter: Ellison Anne Williams
Assignee: Ellison Anne Williams
When launching the Responder using spark-submit, the local property file given
by the '--localPropFile' argument (within the localFS of the launching machine)
is not picked up. This is due to the fact the localFS is not available to the
instance launched by spark-submit.
This can be fixed in one of two ways:
(1) Place the 'local' properties files in a directory in hdfs and have
everything that ends in '.properties' be read and parsed into the system
properties
or
(2) Leave the local properties file on the localFS and use the SparkLauncher to
launch all spark jobs from the Responder (the local prop file will be parsed
and the properties will be passed in as args to the instance).
I am leaning towards (1) as we have had some trouble with the SparkLauncher
correctly picking up 'spark-home' (the directory containing the spark-submit
script) on various cloud platforms (AWS and GCP).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
