[
https://issues.apache.org/jira/browse/BEAM-7587?focusedWorklogId=455135&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-455135
]
ASF GitHub Bot logged work on BEAM-7587:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jul/20 22:59
Start Date: 06/Jul/20 22:59
Worklog Time Spent: 10m
Work Description: ibzib commented on a change in pull request #12157:
URL: https://github.com/apache/beam/pull/12157#discussion_r450521542
##########
File path: runners/spark/job-server/build.gradle
##########
@@ -82,13 +82,21 @@ runShadow {
jvmArgs +=
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
}
-def portableValidatesRunnerTask(String name) {
+def portableValidatesRunnerTask(String name, Boolean streaming) {
+ def pipelineOptions = [
+ // Limit resource consumption via parallelism
+ "--parallelism=2",
Review comment:
This option only applies to Flink, it has no meaning in Spark.
https://github.com/apache/beam/blob/103facbf80c2e4786e60367161ec59352196d04a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java#L74
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 455135)
Time Spent: 1h 10m (was: 1h)
> Spark portable runner: Streaming mode
> -------------------------------------
>
> Key: BEAM-7587
> URL: https://issues.apache.org/jira/browse/BEAM-7587
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Anna Qin
> Priority: P2
> Labels: portability-spark
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> So far all work on the Spark portable runner has been in batch mode. This is
> intended as an uber-issue for tracking progress on adding support for
> streaming.
> -It might be advantageous to wait for the structured streaming (non-portable)
> runner to be completed (to some reasonable extent) before undertaking this,
> rather than using the DStream API.- Since work on the structured streaming
> runner is blocked by SPARK-26655, we should implement this using DStreams
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
