akshatb1 opened a new pull request #331: URL: https://github.com/apache/incubator-livy/pull/331
## What changes were proposed in this pull request? Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.). In this PR, modifying the behavior to merge all the configurations into a temporary properties file and adding that as --properties-file opt in spark-submit command JIRA: https://issues.apache.org/jira/browse/LIVY-870 ## How was this patch tested? Verified Spark job submission in a local Yarn cluster. Checked in the logs that all configurations are merged to a properties file which is added as an opt in spark-submit command. Please see the logs below. ``` 21/10/12 12:01:15 INFO SparkProcessBuilder: Merged Spark config and generated file: D:\Data\Livy0.8\sparkConfig7641984478423208311.properties 21/10/12 12:01:15 INFO SparkProcessBuilder: Running 'D:/Spark/\bin\spark-submit.cmd' '--name' 'SparkBatchJobTest-14' '--class' 'WordCountJob' '--properties-file' 'D:\Data\Livy0.8\sparkConfig7641984478423208311.properties' 'file:///D:/Spark/examples/jars/default_artifact.jar' 'D:/Test.txt' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
