[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=250873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250873
]
ASF GitHub Bot logged work on BEAM-7305:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/May/19 10:00
Start Date: 30/May/19 10:00
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8699: [BEAM-7305]
Fix issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r288930784
##########
File path: runners/jet-experimental/build.gradle
##########
@@ -42,38 +44,64 @@ dependencies {
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+ shadowTest project(path: ":runners:core-construction-java", configuration:
"shadowTest")
shadowTest library.java.hamcrest_core
shadowTest library.java.junit
shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+ needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+ needsRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+ needsRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
+ needsRunner project(path: project.path, configuration: "shadowTest")
+
validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+ validatesRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadowTest")
}
task validatesRunnerBatch(type: Test) {
group = "Verification"
- systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
- "--runner=TestJetRunner",
- "--jetGroupName=jet",
- "--jetLocalParallelism=2"
- ])
+ systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+ exclude '**/ParDoLifecycleTest.class' //some of these tests aren't
playing well with our test runner's parallelism...
Review comment:
I honestly think that in this particular case the tests are the incorrect
thing. I think the problem is that the tests make an assumption which doesn't
hold in the case of our Runner. They assume that by the time the FIRST
`@Teardown` method of a DoFn instance get called, all other instances of the
same DoFn have also finished their processing, hence an Exception should
already have been thrown. I think this assumption is unnecessarily restrictive,
it should be enough to check that by the time the LAST `@Teardown` is being
called the Exception has been thrown. I've made changes towards this idea, now
the test pass, but pls. let me know if this is not ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 250873)
Time Spent: 7h 40m (was: 7.5h)
> Add first version of Hazelcast Jet Runner
> -----------------------------------------
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
> Reporter: Maximilian Michels
> Assignee: Jozsef Bartok
> Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
