tszerszen commented on a change in pull request #13743:
URL: https://github.com/apache/beam/pull/13743#discussion_r563283595
##########
File path:
runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
##########
@@ -274,6 +472,15 @@ public static void main(String[] args) throws Exception {
"The job to run. This must correspond to a subdirectory of the
jar's BEAM-PIPELINE "
+ "directory. *Only needs to be specified if the jar contains
multiple pipelines.*")
private String baseJobName = null;
+
+ @Option(
+ name = "--spark-history-dir",
+ usage = "Spark history dir to store logs (e.g. /tmp/spark-events/)")
+ private String sparkHistoryDir = "/tmp/spark-events/";
Review comment:
@ibzib it only affects Spark History Server, another flaw is for now it
only shows start and end time with accumulator value to download, native Spark
History Server logs are not supported yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
