mosche commented on code in PR #22620:
URL: https://github.com/apache/beam/pull/22620#discussion_r968579281
##########
runners/spark/spark_runner.gradle:
##########
@@ -260,21 +265,60 @@ def validatesRunnerStreaming =
tasks.register("validatesRunnerStreaming", Test)
group = "Verification"
// Disable gradle cache
outputs.upToDateWhen { false }
- def pipelineOptions = JsonOutput.toJson([
- "--runner=TestSparkRunner",
- "--forceStreaming=true",
- "--enableSparkMetricSinks=true",
- ])
- systemProperty "beamTestPipelineOptions", pipelineOptions
+ systemProperties sparkTestProperties(["--enableSparkMetricSinks": "true",
"--forceStreaming": "true"])
classpath = configurations.validatesRunner
- testClassesDirs += files(project.sourceSets.test.output.classesDirs)
+ testClassesDirs += files(
+ project(":sdks:java:core").sourceSets.test.output.classesDirs,
Review Comment:
Yes, in fact VR test were never run/supported for Spark in streaming mode. I
just stumbled on this accidentally when I started looking into bugs related to
onWindowExpiration :(
Instead there was some custom tests in the module that try to mimic the VR
test, but they only cover a very small part (and also run as unit tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
