[
https://issues.apache.org/jira/browse/BEAM-14334?focusedWorklogId=771177&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-771177
]
ASF GitHub Bot logged work on BEAM-14334:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/May/22 07:38
Start Date: 17/May/22 07:38
Worklog Time Spent: 10m
Work Description: mosche commented on code in PR #17662:
URL: https://github.com/apache/beam/pull/17662#discussion_r874471181
##########
runners/spark/spark_runner.gradle:
##########
@@ -218,29 +218,17 @@ def validatesRunnerBatch =
tasks.register("validatesRunnerBatch", Test) {
group = "Verification"
// Disable gradle cache
outputs.upToDateWhen { false }
- def pipelineOptions = JsonOutput.toJson([
- "--runner=TestSparkRunner",
- "--streaming=false",
- "--enableSparkMetricSinks=false",
- ])
- systemProperty "beamTestPipelineOptions", pipelineOptions
- systemProperty "beam.spark.test.reuseSparkContext", "true"
- systemProperty "spark.ui.enabled", "false"
- systemProperty "spark.ui.showConsoleProgress", "false"
+ systemProperties sparkTestProperties(["--enableSparkMetricSinks":"false"])
classpath = configurations.validatesRunner
testClassesDirs = files(
project(":sdks:java:core").sourceSets.test.output.classesDirs,
project(":runners:core-java").sourceSets.test.output.classesDirs,
)
- testClassesDirs += files(project.sourceSets.test.output.classesDirs)
- // Only one SparkContext may be running in a JVM (SPARK-2243)
- forkEvery 1
maxParallelForks 4
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- includeCategories 'org.apache.beam.runners.spark.UsesCheckpointRecovery'
Review Comment:
as said above, tests of this custom category are normal unit tests and are
already run during `test`. there's no runner validation for such a category
Issue Time Tracking
-------------------
Worklog Id: (was: 771177)
Time Spent: 10.5h (was: 10h 20m)
> Avoid using forkEvery in Spark runner tests
> -------------------------------------------
>
> Key: BEAM-14334
> URL: https://issues.apache.org/jira/browse/BEAM-14334
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark, testing
> Reporter: Moritz Mack
> Assignee: Moritz Mack
> Priority: P2
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Usage of *{color:#FF0000}forkEvery 1{color}* is typically a strong sign of
> poor quality / bad code and should be avoided:
> * It significantly impacts performance when running tests.
> * And it often hides resource leaks, either in code or worse in the runner
> itself.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
