[
https://issues.apache.org/jira/browse/PIRK-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ellison Anne Williams resolved PIRK-62.
---------------------------------------
Resolution: Fixed
> Local Properties Files Not Picked Up by Responder Using spark-submit
> --------------------------------------------------------------------
>
> Key: PIRK-62
> URL: https://issues.apache.org/jira/browse/PIRK-62
> Project: PIRK
> Issue Type: Bug
> Reporter: Ellison Anne Williams
> Assignee: Ellison Anne Williams
> Fix For: 0.1.1
>
>
> When launching the Responder using spark-submit, the local property file
> given by the '--localPropFile' argument (within the localFS of the launching
> machine) is not picked up. This is due to the fact the localFS is not
> available to the instance launched by spark-submit.
> This can be fixed in one of two ways:
> (1) Place the 'local' properties files in a directory in hdfs and have
> everything that ends in '.properties' be read and parsed into the system
> properties
> or
> (2) Leave the local properties file on the localFS and use the SparkLauncher
> to launch all spark jobs from the Responder (the local prop file will be
> parsed and the properties will be passed in as args to the instance).
> I am leaning towards (1) as we have had some trouble with the SparkLauncher
> correctly picking up 'spark-home' (the directory containing the spark-submit
> script) on various cloud platforms (AWS and GCP).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
