[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=317736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317736 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 24/Sep/19 18:16 Start Date: 24/Sep/19 18:16 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9651: [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories URL: https://github.com/apache/beam/pull/9651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317736) Time Spent: 12h 20m (was: 12h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 12h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=317687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317687 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 24/Sep/19 17:21 Start Date: 24/Sep/19 17:21 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9651: [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories URL: https://github.com/apache/beam/pull/9651#issuecomment-534660416 Run JavaBeamZetaSQL PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317687) Time Spent: 12h 10m (was: 12h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 12h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=317418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317418 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 24/Sep/19 11:49 Start Date: 24/Sep/19 11:49 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9651: [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories URL: https://github.com/apache/beam/pull/9651#issuecomment-534522175 R: @iemejia R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 317418) Time Spent: 12h (was: 11h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 12h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=317416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317416 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 24/Sep/19 11:48 Start Date: 24/Sep/19 11:48 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #9651: [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories URL: https://github.com/apache/beam/pull/9651 While the Jet Runner for Beam is most definitely an experimental Runner, this fact should be pointed out by documentation, not by putting the "experimental" word into directory and module names for it. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=311081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311081 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 11/Sep/19 23:10 Start Date: 11/Sep/19 23:10 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9471: [BEAM-7305] Improve Jet Runner related documentation URL: https://github.com/apache/beam/pull/9471 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 311081) Time Spent: 11h 40m (was: 11.5h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 11h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=307925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307925
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 06/Sep/19 15:06
Start Date: 06/Sep/19 15:06
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #9471: [BEAM-7305]
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471#discussion_r321779610
##
File path: website/src/get-started/wordcount-example.md
##
@@ -390,6 +390,12 @@ $ mvn package -Pnemo-runner && java -cp
target/word-count-beam-bundled-0.1.jar o
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar
org.apache.beam.examples.WordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml
--output=counts
Review comment:
Sure, done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307925)
Time Spent: 11h 20m (was: 11h 10m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=307926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307926
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 06/Sep/19 15:06
Start Date: 06/Sep/19 15:06
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #9471: [BEAM-7305]
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471#discussion_r321779675
##
File path: website/src/get-started/wordcount-example.md
##
@@ -739,6 +755,12 @@ $ mvn package -Pnemo-runner && java -cp
target/word-count-beam-bundled-0.1.jar o
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar
org.apache.beam.examples.DebuggingWordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml
--output=counts
Review comment:
Good point, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307926)
Time Spent: 11.5h (was: 11h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=307882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307882
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 06/Sep/19 14:16
Start Date: 06/Sep/19 14:16
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on pull request #9471: [BEAM-7305]
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471#discussion_r321749773
##
File path: website/src/get-started/wordcount-example.md
##
@@ -1088,6 +1120,12 @@ $ mvn package -Pnemo-runner && java -cp
target/word-count-beam-bundled-0.1.jar o
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar
org.apache.beam.examples.WindowedWordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml
--output=counts
Review comment:
```suggestion
--runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml
--output=counts
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307882)
Time Spent: 11h 10m (was: 11h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=307881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307881
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 06/Sep/19 14:16
Start Date: 06/Sep/19 14:16
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on pull request #9471: [BEAM-7305]
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471#discussion_r321749449
##
File path: website/src/get-started/wordcount-example.md
##
@@ -739,6 +755,12 @@ $ mvn package -Pnemo-runner && java -cp
target/word-count-beam-bundled-0.1.jar o
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar
org.apache.beam.examples.DebuggingWordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml
--output=counts
Review comment:
```suggestion
--runner=JetRunner --jetLocalMode=3 --output=counts
```
It looks like most of the runners are running against the default inputFile
(the Shakespeare corpus) which demonstrates the PAsserts succeeding in the
example.
Nemo is running against `pom.xml` which demonstrates failing the PAssert in
the example.
I suggest following the majority!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307881)
Time Spent: 11h 10m (was: 11h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=307880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307880
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 06/Sep/19 14:16
Start Date: 06/Sep/19 14:16
Worklog Time Spent: 10m
Work Description: RyanSkraba commented on pull request #9471: [BEAM-7305]
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471#discussion_r321748337
##
File path: website/src/get-started/wordcount-example.md
##
@@ -390,6 +390,12 @@ $ mvn package -Pnemo-runner && java -cp
target/word-count-beam-bundled-0.1.jar o
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar
org.apache.beam.examples.WordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml
--output=counts
Review comment:
```suggestion
--runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml
--output=counts
```
Not a big deal -- `$pwd` doesn't work for my shell (unless I set it of
course), and `` `pwd` `` is consistent with the others.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307880)
Time Spent: 11h 10m (was: 11h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=306310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306310 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 04/Sep/19 13:03 Start Date: 04/Sep/19 13:03 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9471: [BEAM-7305] Improve Jet Runner related documentation URL: https://github.com/apache/beam/pull/9471#issuecomment-527890833 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 306310) Time Spent: 11h (was: 10h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 11h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=306309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306309 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 04/Sep/19 13:03 Start Date: 04/Sep/19 13:03 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #9471: [BEAM-7305] Improve Jet Runner related documentation URL: https://github.com/apache/beam/pull/9471 Small fix-ups of Jet Runner related documentation. Removed obsolete comment, added missing instructions, added Runner to GitHub md file. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=289655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289655 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 06/Aug/19 13:36 Start Date: 06/Aug/19 13:36 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289655) Time Spent: 10h 40m (was: 10.5h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=22=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-22 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 05/Aug/19 11:43 Start Date: 05/Aug/19 11:43 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245#issuecomment-518200658 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 22) Time Spent: 10.5h (was: 10h 20m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=21=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-21 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 05/Aug/19 11:43 Start Date: 05/Aug/19 11:43 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245#issuecomment-518200307 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 21) Time Spent: 10h 20m (was: 10h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 10h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=20=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-20 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 05/Aug/19 11:42 Start Date: 05/Aug/19 11:42 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245#issuecomment-518200307 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 20) Time Spent: 10h 10m (was: 10h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=288878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288878 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 05/Aug/19 11:42 Start Date: 05/Aug/19 11:42 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245#issuecomment-518200307 @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 288878) Time Spent: 10h (was: 9h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=288876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288876 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 05/Aug/19 11:41 Start Date: 05/Aug/19 11:41 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #9245: [BEAM-7305] Adapt Jet Runner page to runner being released now URL: https://github.com/apache/beam/pull/9245 Instruction for fetching the Examples project were obsolete and redundant, fixed it. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=284291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284291 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 29/Jul/19 13:38 Start Date: 29/Jul/19 13:38 Worklog Time Spent: 10m Work Description: mxm commented on pull request #9177: [BEAM-7305] Set job name in Jet Runner URL: https://github.com/apache/beam/pull/9177 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 284291) Time Spent: 9h 40m (was: 9.5h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=284038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284038 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 29/Jul/19 06:27 Start Date: 29/Jul/19 06:27 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #9177: [BEAM-7305] Set job name in Jet Runner URL: https://github.com/apache/beam/pull/9177 Set Jet job name from pipeline options. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=284039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284039 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 29/Jul/19 06:27 Start Date: 29/Jul/19 06:27 Worklog Time Spent: 10m Work Description: jbartok commented on issue #9177: [BEAM-7305] Set job name in Jet Runner URL: https://github.com/apache/beam/pull/9177#issuecomment-515862140 @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 284039) Time Spent: 9.5h (was: 9h 20m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=266369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266369 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 25/Jun/19 06:51 Start Date: 25/Jun/19 06:51 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8937: [BEAM-7305] Add Jet Runner to Get Started documentation page URL: https://github.com/apache/beam/pull/8937 Just updating documentation so that "Get Started" web page also mentions the Jet Runner. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=266371=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266371 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 25/Jun/19 06:51 Start Date: 25/Jun/19 06:51 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8937: [BEAM-7305] Add Jet Runner to Get Started documentation page URL: https://github.com/apache/beam/pull/8937#issuecomment-505311474 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266371) Time Spent: 9h (was: 8h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=252064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-252064
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 31/May/19 21:13
Start Date: 31/May/19 21:13
Worklog Time Spent: 10m
Work Description: viliam-durina commented on pull request #8699:
[BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r289552490
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
##
@@ -19,28 +19,89 @@
import com.hazelcast.jet.core.AbstractProcessor;
import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.function.SupplierEx;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.util.WindowedValue;
/**
* /** * Jet {@link com.hazelcast.jet.core.Processor} implementation for
Beam's Impulse primitive.
*/
public class ImpulseP extends AbstractProcessor {
+ private final boolean active;
+ private final Coder outputCoder;
private final String ownerId; // do not remove it, very useful for debugging
- private ImpulseP(String ownerId) {
+ private ImpulseP(boolean active, Coder outputCoder, String ownerId) {
+this.active = active;
+this.outputCoder = outputCoder;
this.ownerId = ownerId;
}
- public static SupplierEx supplier(String ownerId) {
-return () -> new ImpulseP(ownerId);
- }
-
@Override
public boolean complete() {
-return tryEmit(
-WindowedValue.valueInGlobalWindow(
-new byte[0])); // todo: should EACH processor emit this byte[] or
just a SINGLE one?
+if (active) {
+ return tryEmit(Utils.encode(WindowedValue.valueInGlobalWindow(new
byte[0]), outputCoder));
+} else {
+ return true;
+}
+ }
+
+ public static ProcessorMetaSupplier supplier(Coder outputCoder, String
ownerId) {
+return new ImpulseMetaProcessorSupplier(outputCoder, ownerId);
+ }
+
+ private static class ImpulseMetaProcessorSupplier implements
ProcessorMetaSupplier {
+
+private final Coder outputCoder;
+private final String ownerId;
+
+private ImpulseMetaProcessorSupplier(Coder outputCoder, String ownerId) {
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+}
+
+@SuppressWarnings("unchecked")
+@Nonnull
+@Override
+public Function get(
+@Nonnull List addresses) {
+ return address -> new ImpulseProcessorSupplier(outputCoder, ownerId);
+}
+ }
+
+ private static class ImpulseProcessorSupplier implements
ProcessorSupplier {
+private final Coder outputCoder;
+private final String ownerId;
+private transient ProcessorSupplier.Context context;
+
+private ImpulseProcessorSupplier(Coder outputCoder, String ownerId) {
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+}
+
+@Override
+public void init(@Nonnull Context context) {
+ this.context = context;
+}
+
+@Nonnull
+@Override
+public Collection get(int count) {
+ int indexBase = context.memberIndex() * context.localParallelism();
+ List res = new ArrayList<>(count);
+ for (int i = 0; i < count; i++, indexBase++) {
+res.add(new ImpulseP(indexBase == 0, outputCoder, ownerId));
Review comment:
You don't need PMS/PS for this, `Processor.Context` has
`globalProcessorIndex()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 252064)
Time Spent: 8.5h (was: 8h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=252065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-252065
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 31/May/19 21:13
Start Date: 31/May/19 21:13
Worklog Time Spent: 10m
Work Description: viliam-durina commented on pull request #8699:
[BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r289551291
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
##
@@ -247,7 +255,12 @@ private void processBufferedRegularItems(Inbox inbox) {
@Override
public boolean tryProcess() {
-return outputManager.tryFlush();
+boolean successful = outputManager.tryFlush();
+if (successful && System.currentTimeMillis() > lastMetricsFlushTime +
metricsFlushPeriod) {
Review comment:
Should use `System.nanoTime`, real-time clock can move back and forth. Also
you can use `nextMetricsFlushTime` and save one add operation for most calls.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 252065)
Time Spent: 8h 40m (was: 8.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=251809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251809 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 31/May/19 14:30 Start Date: 31/May/19 14:30 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251809) Time Spent: 8h 20m (was: 8h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=251706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251706 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 31/May/19 11:14 Start Date: 31/May/19 11:14 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699#issuecomment-497672574 Sure, I just reasoned that multiple smaller changes would be simpler to review. Coming up! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251706) Time Spent: 8h 10m (was: 8h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=251704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251704 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 31/May/19 11:12 Start Date: 31/May/19 11:12 Worklog Time Spent: 10m Work Description: mxm commented on issue #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699#issuecomment-497672150 Could you squash at least the spotless fixup commits? I'll merge then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251704) Time Spent: 8h (was: 7h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=250898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250898
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 30/May/19 11:12
Start Date: 30/May/19 11:12
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8699: [BEAM-7305]
Fix issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r288950444
##
File path: runners/jet-experimental/build.gradle
##
@@ -42,38 +44,64 @@ dependencies {
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-construction-java", configuration:
"shadowTest")
shadowTest library.java.hamcrest_core
shadowTest library.java.junit
shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+needsRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+needsRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
+needsRunner project(path: project.path, configuration: "shadowTest")
+
validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadowTest")
}
task validatesRunnerBatch(type: Test) {
group = "Verification"
-systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
-"--runner=TestJetRunner",
-"--jetGroupName=jet",
-"--jetLocalParallelism=2"
-])
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+exclude '**/ParDoLifecycleTest.class' //some of these tests aren't
playing well with our test runner's parallelism...
}
maxHeapSize = '4g'
}
task validatesRunner {
group = "Verification"
-description "Validates Jet runner"
+description "Validates Jet Runner"
dependsOn validatesRunnerBatch
}
+
+task needsRunnerTests(type: Test) {
+group = "Verification"
+description = "Runs tests that require a runner to validate that
piplines/transforms work correctly"
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
+
+classpath = configurations.needsRunner
+testClassesDirs +=
files(project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+testClassesDirs +=
files(project(":runners:core-java").sourceSets.test.output.classesDirs)
+testClassesDirs +=
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories "org.apache.beam.sdk.testing.NeedsRunner"
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
Review comment:
Well, so-so: `697 tests completed, 69 failed, 6 skipped`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 250898)
Time Spent: 7h 50m (was: 7h 40m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 7h 50m
> Remaining Estimate: 0h
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=250873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250873
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 30/May/19 10:00
Start Date: 30/May/19 10:00
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8699: [BEAM-7305]
Fix issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r288930784
##
File path: runners/jet-experimental/build.gradle
##
@@ -42,38 +44,64 @@ dependencies {
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-construction-java", configuration:
"shadowTest")
shadowTest library.java.hamcrest_core
shadowTest library.java.junit
shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+needsRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+needsRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
+needsRunner project(path: project.path, configuration: "shadowTest")
+
validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadowTest")
}
task validatesRunnerBatch(type: Test) {
group = "Verification"
-systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
-"--runner=TestJetRunner",
-"--jetGroupName=jet",
-"--jetLocalParallelism=2"
-])
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+exclude '**/ParDoLifecycleTest.class' //some of these tests aren't
playing well with our test runner's parallelism...
Review comment:
I honestly think that in this particular case the tests are the incorrect
thing. I think the problem is that the tests make an assumption which doesn't
hold in the case of our Runner. They assume that by the time the FIRST
`@Teardown` method of a DoFn instance get called, all other instances of the
same DoFn have also finished their processing, hence an Exception should
already have been thrown. I think this assumption is unnecessarily restrictive,
it should be enough to check that by the time the LAST `@Teardown` is being
called the Exception has been thrown. I've made changes towards this idea, now
the test pass, but pls. let me know if this is not ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 250873)
Time Spent: 7h 40m (was: 7.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=250037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250037
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 29/May/19 11:10
Start Date: 29/May/19 11:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8699: [BEAM-7305] Fix
issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r288512260
##
File path: runners/jet-experimental/build.gradle
##
@@ -42,38 +44,64 @@ dependencies {
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-construction-java", configuration:
"shadowTest")
shadowTest library.java.hamcrest_core
shadowTest library.java.junit
shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+needsRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+needsRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
+needsRunner project(path: project.path, configuration: "shadowTest")
+
validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadowTest")
}
task validatesRunnerBatch(type: Test) {
group = "Verification"
-systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
-"--runner=TestJetRunner",
-"--jetGroupName=jet",
-"--jetLocalParallelism=2"
-])
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+exclude '**/ParDoLifecycleTest.class' //some of these tests aren't
playing well with our test runner's parallelism...
}
maxHeapSize = '4g'
}
task validatesRunner {
group = "Verification"
-description "Validates Jet runner"
+description "Validates Jet Runner"
dependsOn validatesRunnerBatch
}
+
+task needsRunnerTests(type: Test) {
+group = "Verification"
+description = "Runs tests that require a runner to validate that
piplines/transforms work correctly"
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
+
+classpath = configurations.needsRunner
+testClassesDirs +=
files(project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+testClassesDirs +=
files(project(":runners:core-java").sourceSets.test.output.classesDirs)
+testClassesDirs +=
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories "org.apache.beam.sdk.testing.NeedsRunner"
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
Review comment:
Are all of these passing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 250037)
Time Spent: 7.5h (was: 7h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
--
This message was sent by
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=250036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250036
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 29/May/19 11:10
Start Date: 29/May/19 11:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8699: [BEAM-7305] Fix
issues in and extend documentation for Hazelcast Jet Runner
URL: https://github.com/apache/beam/pull/8699#discussion_r288512243
##
File path: runners/jet-experimental/build.gradle
##
@@ -42,38 +44,64 @@ dependencies {
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-construction-java", configuration:
"shadowTest")
shadowTest library.java.hamcrest_core
shadowTest library.java.junit
shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+needsRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+needsRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
+needsRunner project(path: project.path, configuration: "shadowTest")
+
validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-construction-java",
configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadowTest")
}
task validatesRunnerBatch(type: Test) {
group = "Verification"
-systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
-"--runner=TestJetRunner",
-"--jetGroupName=jet",
-"--jetLocalParallelism=2"
-])
+systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(["--runner=TestJetRunner"])
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+exclude '**/ParDoLifecycleTest.class' //some of these tests aren't
playing well with our test runner's parallelism...
Review comment:
This looks like an important test to fix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 250036)
Time Spent: 7h 20m (was: 7h 10m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=249301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249301 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 28/May/19 12:28 Start Date: 28/May/19 12:28 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699#issuecomment-496495309 Ahh, I finally understand why the above check is failing, will fix it a bit later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249301) Time Spent: 7h 10m (was: 7h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=249295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249295 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 28/May/19 12:21 Start Date: 28/May/19 12:21 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699#issuecomment-496493018 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249295) Time Spent: 7h (was: 6h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=249272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249272 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 28/May/19 11:32 Start Date: 28/May/19 11:32 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699 Some new features: * flushing of metrics periodically instead of at job end * option to make ParDo processors cooperative Plus some bug fixes and documentation improvements. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=249273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249273 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 28/May/19 11:32 Start Date: 28/May/19 11:32 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner URL: https://github.com/apache/beam/pull/8699#issuecomment-496478284 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249273) Time Spent: 6h 50m (was: 6h 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247366 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 23/May/19 10:19 Start Date: 23/May/19 10:19 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247366) Time Spent: 6.5h (was: 6h 20m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247337
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 23/May/19 08:30
Start Date: 23/May/19 08:30
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8637: [BEAM-7305] Make
Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286831527
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
##
@@ -31,6 +31,19 @@
void setJetGroupName(String jetGroupName);
+ @Description("Specifies the addresses of the Jet cluster; needed only with
external clusters")
+ @Validation.Required
+ String getJetClusterAddresses();
+
+ void setJetClusterAddresses(String jetClusterAddresses);
+
+ @Description(
+ "Specifies where the fat-jar containing all the code is located; needed
only with external clusters")
+ @Validation.Required
+ String getCodeJarLocation();
+
+ void setCodeJarLocation(String codeJarLocation);
Review comment:
My bad, I thought this was inherited from the base `PipelineOptions`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247337)
Time Spent: 6h 10m (was: 6h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247321
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 23/May/19 07:20
Start Date: 23/May/19 07:20
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8637: [BEAM-7305]
Make Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286807170
##
File path: runners/jet-experimental/build.gradle
##
@@ -19,6 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
+plugins { id 'maven-publish' }
applyJavaNature()
Review comment:
I will remove the whole option of generating examples from sources.
Generating them from snapshots is more than enough (and more convenient).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247321)
Time Spent: 5h 50m (was: 5h 40m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247320
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 23/May/19 07:20
Start Date: 23/May/19 07:20
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8637: [BEAM-7305]
Make Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286807019
##
File path: sdks/java/maven-archetypes/examples/build.gradle
##
@@ -17,6 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
+plugins { id 'maven-publish' }
Review comment:
I will remove the whole option of generating examples from sources.
Generating them from snapshots is more than enough (and more convenient).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247320)
Time Spent: 5h 40m (was: 5.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247319 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 23/May/19 07:19 Start Date: 23/May/19 07:19 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#discussion_r286806832 ## File path: sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml ## @@ -334,6 +336,7 @@ + Review comment: Sure, will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247319) Time Spent: 5.5h (was: 5h 20m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247295 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 23/May/19 06:12 Start Date: 23/May/19 06:12 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#discussion_r286788789 ## File path: website/src/documentation/runners/jet.md ## @@ -0,0 +1,185 @@ +--- +layout: section +title: "Hazelcast Jet Runner" +section_menu: section-menu/runners.html +permalink: /documentation/runners/jet/ +redirect_from: /learn/runners/jet/ +--- + + +# Overview + +The Hazelcast Jet Runner can be used to execute Beam pipelines using [Hazelcat +Jet](https://jet.hazelcast.org/). + +The Jet Runner and Jet are suitable for large scale continuous jobs and provide: +* Support for both batch (bounded) and streaming (unbounded) data sets +* A runtime that supports very high throughput and low event latency at the same time +* Natural back-pressure in streaming programs +* Distributed massively parallel data processing engine with in memory storage + +It's important to note that the Jet Runner is currently in an *EXPERIMENTAL* state and can not make use of many of +the capabilities present in Jet: +* Jet has full Fault Tolerance support, the Jet Runner does not; if a job fails it must be restarted +* Internal performance of Jet is extremely high (see [benchmarks](https://jet.hazelcast.org/performance/)). Review comment: Will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247295) Time Spent: 5h 20m (was: 5h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=247288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247288
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 23/May/19 05:58
Start Date: 23/May/19 05:58
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8637: [BEAM-7305]
Make Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286786062
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
##
@@ -31,6 +31,19 @@
void setJetGroupName(String jetGroupName);
+ @Description("Specifies the addresses of the Jet cluster; needed only with
external clusters")
+ @Validation.Required
+ String getJetClusterAddresses();
+
+ void setJetClusterAddresses(String jetClusterAddresses);
+
+ @Description(
+ "Specifies where the fat-jar containing all the code is located; needed
only with external clusters")
+ @Validation.Required
+ String getCodeJarLocation();
+
+ void setCodeJarLocation(String codeJarLocation);
Review comment:
It can be an `URL` or a `path string` to a file (on the client machine).
I'll try to give it a better name, but I don't see a `resourcesToStage` in
`PipelineOptions`. Even the Flink Runner declares his own `filesToStage` in
`FilePipelineOptions`. Am I missing something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247288)
Time Spent: 5h 10m (was: 5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246745
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 22/May/19 14:21
Start Date: 22/May/19 14:21
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8637: [BEAM-7305]
Make Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286516132
##
File path:
runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
##
@@ -60,6 +60,8 @@
private TestJetRunner(PipelineOptions options) {
JetPipelineOptions jetPipelineOptions =
options.as(JetPipelineOptions.class);
jetPipelineOptions.setJetStartOwnCluster(false);
+jetPipelineOptions.setJetClusterAddresses("127.0.0.1:5701");
Review comment:
This is actually not used, will improve. I was planning a review of the
configuration options anyways.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246745)
Time Spent: 5h (was: 4h 50m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246744 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 22/May/19 14:19 Start Date: 22/May/19 14:19 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#discussion_r286515508 ## File path: website/src/documentation/runners/jet.md ## @@ -0,0 +1,185 @@ +--- +layout: section +title: "Hazelcast Jet Runner" +section_menu: section-menu/runners.html +permalink: /documentation/runners/jet/ +redirect_from: /learn/runners/jet/ +--- + + +# Overview + +The Hazelcast Jet Runner can be used to execute Beam pipelines using [Hazelcat +Jet](https://jet.hazelcast.org/). + +The Jet Runner and Jet are suitable for large scale continuous jobs and provide: +* Support for both batch (bounded) and streaming (unbounded) data sets +* A runtime that supports very high throughput and low event latency at the same time +* Natural back-pressure in streaming programs +* Distributed massively parallel data processing engine with in memory storage + +It's important to note that the Jet Runner is currently in an *EXPERIMENTAL* state and can not make use of many of +the capabilities present in Jet: +* Jet has full Fault Tolerance support, the Jet Runner does not; if a job fails it must be restarted +* Internal performance of Jet is extremely high (see [benchmarks](https://jet.hazelcast.org/performance/)). Review comment: Meaning you would like me to remove this benchmark link? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246744) Time Spent: 4h 50m (was: 4h 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246716
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 22/May/19 13:09
Start Date: 22/May/19 13:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8637: [BEAM-7305] Make
Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286471504
##
File path:
runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
##
@@ -60,6 +60,8 @@
private TestJetRunner(PipelineOptions options) {
JetPipelineOptions jetPipelineOptions =
options.as(JetPipelineOptions.class);
jetPipelineOptions.setJetStartOwnCluster(false);
+jetPipelineOptions.setJetClusterAddresses("127.0.0.1:5701");
Review comment:
Is this using a fixed port for the tests? This can cause issues with
multiple test instances running on the same Jenkins host.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246716)
Time Spent: 4.5h (was: 4h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246714 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 22/May/19 13:09 Start Date: 22/May/19 13:09 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#discussion_r286470316 ## File path: website/src/documentation/runners/jet.md ## @@ -0,0 +1,185 @@ +--- +layout: section +title: "Hazelcast Jet Runner" +section_menu: section-menu/runners.html +permalink: /documentation/runners/jet/ +redirect_from: /learn/runners/jet/ +--- + + +# Overview + +The Hazelcast Jet Runner can be used to execute Beam pipelines using [Hazelcat +Jet](https://jet.hazelcast.org/). + +The Jet Runner and Jet are suitable for large scale continuous jobs and provide: +* Support for both batch (bounded) and streaming (unbounded) data sets +* A runtime that supports very high throughput and low event latency at the same time +* Natural back-pressure in streaming programs +* Distributed massively parallel data processing engine with in memory storage + +It's important to note that the Jet Runner is currently in an *EXPERIMENTAL* state and can not make use of many of +the capabilities present in Jet: +* Jet has full Fault Tolerance support, the Jet Runner does not; if a job fails it must be restarted +* Internal performance of Jet is extremely high (see [benchmarks](https://jet.hazelcast.org/performance/)). Review comment: In the context of a Beam Runner page, I have mixed feelings about a synthetic benchmark which compares Runner backends. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246714) Time Spent: 4h 10m (was: 4h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246717
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 22/May/19 13:09
Start Date: 22/May/19 13:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8637: [BEAM-7305] Make
Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286477573
##
File path: runners/jet-experimental/build.gradle
##
@@ -19,6 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
+plugins { id 'maven-publish' }
applyJavaNature()
Review comment:
You should be able to load the plugin via the Java nature. See the comment
further below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246717)
Time Spent: 4h 40m (was: 4.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246715
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 22/May/19 13:09
Start Date: 22/May/19 13:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8637: [BEAM-7305] Make
Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286453905
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
##
@@ -31,6 +31,19 @@
void setJetGroupName(String jetGroupName);
+ @Description("Specifies the addresses of the Jet cluster; needed only with
external clusters")
+ @Validation.Required
+ String getJetClusterAddresses();
+
+ void setJetClusterAddresses(String jetClusterAddresses);
+
+ @Description(
+ "Specifies where the fat-jar containing all the code is located; needed
only with external clusters")
+ @Validation.Required
+ String getCodeJarLocation();
+
+ void setCodeJarLocation(String codeJarLocation);
Review comment:
Should this be a `URI`? Also there is already `resourcesToStage` in
`PipelineOptions`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246715)
Time Spent: 4h 20m (was: 4h 10m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246713 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 22/May/19 13:09 Start Date: 22/May/19 13:09 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#discussion_r286455080 ## File path: sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml ## @@ -334,6 +336,7 @@ + Review comment: Can you revert these changes here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246713) Time Spent: 4h (was: 3h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246718
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 22/May/19 13:09
Start Date: 22/May/19 13:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8637: [BEAM-7305] Make
Jet Runner usable and documented
URL: https://github.com/apache/beam/pull/8637#discussion_r286476164
##
File path: sdks/java/maven-archetypes/examples/build.gradle
##
@@ -17,6 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
+plugins { id 'maven-publish' }
Review comment:
Not sure if we should include this here. Have you seen that in
`BeamModulePlugin` there is an option to load this plugin?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246718)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246067 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 21/May/19 14:08 Start Date: 21/May/19 14:08 Worklog Time Spent: 10m Work Description: mxm commented on issue #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#issuecomment-494407194 @jbartok You can use `./gradlew :website:serveWebsite` to preview the website. You do not need to be concerned with the actual deployment which is automated after merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246067) Time Spent: 3h 50m (was: 3h 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246049 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 21/May/19 13:46 Start Date: 21/May/19 13:46 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637#issuecomment-494398335 R: @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246049) Time Spent: 3h 40m (was: 3.5h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=246048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246048 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 21/May/19 13:46 Start Date: 21/May/19 13:46 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8637: [BEAM-7305] Make Jet Runner usable and documented URL: https://github.com/apache/beam/pull/8637 Intended changes: * add Jet Runner to the capability matrix * provide some documentation for Jet Runner, including a way to run example pipelines with it * provide a way to generate the Maven Archetype for Examples from any branch/version of the code and add it to local Maven repo DISCLAIMER: Even though I've attempted to update website content in this PR I have no clear understanding how this content gets turned into the actual web-page so I might have missed making some changes that are complementary and needed. If that's the case pls. let me know and I'll fix it immediately. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [![Build
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=245306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245306 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 20/May/19 16:42 Start Date: 20/May/19 16:42 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#issuecomment-494064082 Ok, we'll try to do further development here instead of our own runner repo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245306) Time Spent: 3h 20m (was: 3h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=245305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245305 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 20/May/19 16:41 Start Date: 20/May/19 16:41 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245305) Time Spent: 3h 10m (was: 3h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=245302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245302 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 20/May/19 16:40 Start Date: 20/May/19 16:40 Worklog Time Spent: 10m Work Description: mxm commented on issue #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#issuecomment-494063377 In the long run it would be nice to have other contributors to the Jet Runner. Of course this can take a bit. I think the first step would be to have the review process with your colleagues here instead of in the fork. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245302) Time Spent: 3h (was: 2h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=244331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244331 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 17/May/19 21:42 Start Date: 17/May/19 21:42 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#discussion_r285297693 ## File path: runners/jet-experimental/build.gradle ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file Review comment: OK, I see. I did not quite understand that the archive base name was so well automated. That is great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244331) Time Spent: 2h 50m (was: 2h 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243839 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 17/May/19 05:37 Start Date: 17/May/19 05:37 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#issuecomment-493326374 Hi @mxm. Yes, I was pondering it yesterday if I should make this pull request out of multiple change-sets or squash them down to a single one... I might not have made the best choice... The thing is that I'm dumping months of my work into these two change-sets, that's why it looks so non-incremental. The actual development has been done in https://github.com/hazelcast/hazelcast-jet-beam-runner, there are 100+ commits there (debugging has proven simpler if working like this, worth the effort of migrating later). Anyways, from now on pace of development should be slower and I will make it more incremental by issuing more frequent PRs. As far as reviews are concerned, they have been done to some degree on our module by my Hazelcast colleagues. Here we would need somebody both impartial to Hazelcast and with knowledge of Jet, might be not that simple to find. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243839) Time Spent: 2h 40m (was: 2.5h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243594 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 19:20 Start Date: 16/May/19 19:20 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#discussion_r284856263 ## File path: runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java ## @@ -79,7 +76,6 @@ TRANSLATORS.put(PTransformTranslation.FLATTEN_TRANSFORM_URN, new FlattenTranslator()); TRANSLATORS.put(PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN, new WindowTranslator()); TRANSLATORS.put(PTransformTranslation.IMPULSE_TRANSFORM_URN, new ImpulseTranslator()); -TRANSLATORS.put(PTransformTranslation.TEST_STREAM_TRANSFORM_URN, new TestStreamTranslator()); Review comment: Curious, why did you remove this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243594) Time Spent: 2h 20m (was: 2h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243596
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 16/May/19 19:20
Start Date: 16/May/19 19:20
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8592: [BEAM-7305]
Improve and extend Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8592#discussion_r284856811
##
File path:
runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
##
@@ -246,4 +259,34 @@ static boolean usesStateOrTimers(AppliedPTransform appliedTransform) {
return WindowedValue.FullWindowedValueCoder.of(
ListCoder.of(elementCoder.getValueCoder()),
elementCoder.getWindowCoder());
}
+
+ /** A wrapper of {@code byte[]} that can be used as a hash-map key. */
+ public static class ByteArrayKey {
+private final byte[] value;
+private int hash;
+
+public ByteArrayKey(@Nonnull byte[] value) {
+ this.value = value;
+}
+
+@Override
+public boolean equals(Object o) {
+ if (this == o) {
+return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+return false;
+ }
+ ByteArrayKey that = (ByteArrayKey) o;
+ return Arrays.equals(value, that.value);
+}
+
+@Override
+public int hashCode() {
+ if (hash == 0) {
Review comment:
Make `hash` an `Integer` and check for null here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243596)
Time Spent: 2.5h (was: 2h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243595 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 19:20 Start Date: 16/May/19 19:20 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#discussion_r284858529 ## File path: runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java ## @@ -79,7 +76,6 @@ TRANSLATORS.put(PTransformTranslation.FLATTEN_TRANSFORM_URN, new FlattenTranslator()); TRANSLATORS.put(PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN, new WindowTranslator()); TRANSLATORS.put(PTransformTranslation.IMPULSE_TRANSFORM_URN, new ImpulseTranslator()); -TRANSLATORS.put(PTransformTranslation.TEST_STREAM_TRANSFORM_URN, new TestStreamTranslator()); Review comment: Ah, see that you moved it to the test runner. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243595) Time Spent: 2.5h (was: 2h 20m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243282 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 12:16 Start Date: 16/May/19 12:16 Worklog Time Spent: 10m Work Description: adude3141 commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#discussion_r284675684 ## File path: runners/jet-experimental/build.gradle ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file Review comment: Not quite sure, if I understand you correctly, @kennknowles I think, you are referring to archiveBaseName here. It is usually not necessary to setup on project level. Only if the default seems improper, we are able (and should) to overwrite. E.g. in this case, the folder path is `runners:jet-experimental` which will result in artefact named something about `beam-runners-jet-experimemtal`. Now, we could decide that this is not wanted as after switching from experimental mode we would like to keep artifact named consistent. So we could add an ``` archivesBaseName = 'beam-runners-jet' ``` More likely in this case though, would be that we like to drop that `experimental`on the folder name, i.e. move `runners/jet-experimental` to `runners/jet` and set ``` archivesBaseName = 'beam-runners-jet-experimental' ``` for the time being. (Hope I got it right) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243282) Time Spent: 2h 10m (was: 2h) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243280 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 12:15 Start Date: 16/May/19 12:15 Worklog Time Spent: 10m Work Description: adude3141 commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#discussion_r284675684 ## File path: runners/jet-experimental/build.gradle ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file Review comment: Not quite sure, if I understand you correctly, @kennknowles I think, you are referring to archiveBaseName here. It is usually not necessary to setup on project level. Only if the default seems improper, we are ab le to overwrite. E.g. in this case, the folder path is `runners:jet-experimental` which will result in artefact named something about `beam-runners-jet-experimemtal`. Now, we could decide that this is not wanted as after switching from experimental mode we would like to keep artifact named consistent. So we could add an ``` archivesBaseName = 'beam-runners-jet' ``` More likely in this though, would be thet we would like to drop that `experimental`on the folder name, i.e. move `runners/jet-experimental` to `runners/jet` and set ``` archivesBaseName = 'beam-runners-jet-experimental' ``` for the time being. (Hope I got it right) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243280) Time Spent: 1h 50m (was: 1h 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243281 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 12:15 Start Date: 16/May/19 12:15 Worklog Time Spent: 10m Work Description: adude3141 commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#discussion_r284675684 ## File path: runners/jet-experimental/build.gradle ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file Review comment: Not quite sure, if I understand you correctly, @kennknowles I think, you are referring to archiveBaseName here. It is usually not necessary to setup on project level. Only if the default seems improper, we are able (and should) to overwrite. E.g. in this case, the folder path is `runners:jet-experimental` which will result in artefact named something about `beam-runners-jet-experimemtal`. Now, we could decide that this is not wanted as after switching from experimental mode we would like to keep artifact named consistent. So we could add an ``` archivesBaseName = 'beam-runners-jet' ``` More likely in this though, would be thet we would like to drop that `experimental`on the folder name, i.e. move `runners/jet-experimental` to `runners/jet` and set ``` archivesBaseName = 'beam-runners-jet-experimental' ``` for the time being. (Hope I got it right) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243281) Time Spent: 2h (was: 1h 50m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243279
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 16/May/19 12:07
Start Date: 16/May/19 12:07
Worklog Time Spent: 10m
Work Description: adude3141 commented on pull request #8410: [BEAM-7305]
Add first version of Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8410#discussion_r284673258
##
File path: runners/jet-experimental/build.gradle
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+
+project.ext {
+jet_version = '3.0'
+hazelcast_version = '3.12'
+}
+
+configurations {
+validatesRunner
+}
+
+dependencies {
+shadow project(path: ":sdks:java:core", configuration: "shadow")
+shadow project(path: ":runners:core-java", configuration: "shadow")
+shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest library.java.hamcrest_core
+shadowTest library.java.junit
+shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
+shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: project.path, configuration: "shadowTest")
+}
+
+task validatesRunnerBatch(type: Test) {
+group = "Verification"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestJetRunner",
+"--jetGroupName=jet",
+"--jetLocalParallelism=2"
+])
+
+classpath = configurations.validatesRunner
+testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
+exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+}
+
+maxHeapSize = '4g'
+}
+
+task validatesRunner {
+group = "Verification"
+description "Validates Jet runner"
+dependsOn validatesRunnerBatch
+}
+
+spotless {
+java {
+paddedCell()
Review comment:
@jbartok Nice, Thx!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243279)
Time Spent: 1h 40m (was: 1.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243183
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 16/May/19 08:35
Start Date: 16/May/19 08:35
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8410: [BEAM-7305]
Add first version of Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8410#discussion_r284595827
##
File path: runners/jet-experimental/build.gradle
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+
+project.ext {
+jet_version = '3.0'
+hazelcast_version = '3.12'
+}
+
+configurations {
+validatesRunner
+}
+
+dependencies {
+shadow project(path: ":sdks:java:core", configuration: "shadow")
+shadow project(path: ":runners:core-java", configuration: "shadow")
+shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest library.java.hamcrest_core
+shadowTest library.java.junit
+shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
+shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: project.path, configuration: "shadowTest")
+}
+
+task validatesRunnerBatch(type: Test) {
+group = "Verification"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestJetRunner",
+"--jetGroupName=jet",
+"--jetLocalParallelism=2"
+])
+
+classpath = configurations.validatesRunner
+testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
+exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+}
+
+maxHeapSize = '4g'
+}
+
+task validatesRunner {
+group = "Verification"
+description "Validates Jet runner"
+dependsOn validatesRunnerBatch
+}
+
+spotless {
+java {
+paddedCell()
Review comment:
https://github.com/apache/beam/pull/8592
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243183)
Time Spent: 1.5h (was: 1h 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243177 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 08:20 Start Date: 16/May/19 08:20 Worklog Time Spent: 10m Work Description: jbartok commented on issue #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592#issuecomment-492968527 R: @mxm R: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243177) Time Spent: 1h 20m (was: 1h 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243176 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 16/May/19 08:19 Start Date: 16/May/19 08:19 Worklog Time Spent: 10m Work Description: jbartok commented on pull request #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8592 * added support for Beam UnboundedSources * optimised how Jet DAG is wired together * sorted out various metrics related issues * improved on how JetPipelineResult works * removed side input related hack * made cluster member count configurable * done a bit of cleanup Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=243122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243122
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 16/May/19 05:32
Start Date: 16/May/19 05:32
Worklog Time Spent: 10m
Work Description: jbartok commented on pull request #8410: [BEAM-7305]
Add first version of Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8410#discussion_r284543981
##
File path: runners/jet-experimental/build.gradle
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+
+project.ext {
+jet_version = '3.0'
+hazelcast_version = '3.12'
+}
+
+configurations {
+validatesRunner
+}
+
+dependencies {
+shadow project(path: ":sdks:java:core", configuration: "shadow")
+shadow project(path: ":runners:core-java", configuration: "shadow")
+shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest library.java.hamcrest_core
+shadowTest library.java.junit
+shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
+shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: project.path, configuration: "shadowTest")
+}
+
+task validatesRunnerBatch(type: Test) {
+group = "Verification"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestJetRunner",
+"--jetGroupName=jet",
+"--jetLocalParallelism=2"
+])
+
+classpath = configurations.validatesRunner
+testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
+exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+}
+
+maxHeapSize = '4g'
+}
+
+task validatesRunner {
+group = "Verification"
+description "Validates Jet runner"
+dependsOn validatesRunnerBatch
+}
+
+spotless {
+java {
+paddedCell()
Review comment:
Hi Guys. I'm submitting a second pull request today with more changes
related to jet runner and in that I'm removing this whole spotless block, it's
not really needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243122)
Time Spent: 1h (was: 50m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=242711=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242711 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 15/May/19 17:57 Start Date: 15/May/19 17:57 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#discussion_r284379886 ## File path: runners/jet-experimental/build.gradle ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file Review comment: @adude3141 should there be some publishing config here to get the artifactId set up? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 242711) Time Spent: 50m (was: 40m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Assignee: Jozsef Bartok >Priority: Major > Fix For: 2.14.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=242709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242709
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 15/May/19 17:56
Start Date: 15/May/19 17:56
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #8410:
[BEAM-7305] Add first version of Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8410#discussion_r284379323
##
File path: runners/jet-experimental/build.gradle
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+
+project.ext {
+jet_version = '3.0'
+hazelcast_version = '3.12'
+}
+
+configurations {
+validatesRunner
+}
+
+dependencies {
+shadow project(path: ":sdks:java:core", configuration: "shadow")
+shadow project(path: ":runners:core-java", configuration: "shadow")
+shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest library.java.hamcrest_core
+shadowTest library.java.junit
+shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
+shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: project.path, configuration: "shadowTest")
+}
+
+task validatesRunnerBatch(type: Test) {
+group = "Verification"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestJetRunner",
+"--jetGroupName=jet",
+"--jetLocalParallelism=2"
+])
+
+classpath = configurations.validatesRunner
+testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
+exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+}
+
+maxHeapSize = '4g'
+}
+
+task validatesRunner {
+group = "Verification"
+description "Validates Jet runner"
+dependsOn validatesRunnerBatch
+}
+
+spotless {
+java {
+paddedCell()
Review comment:
Ah I just noticed this. Yes, we have hit bugs that slowed the whole
community down because of paddedCell.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242709)
Time Spent: 40m (was: 0.5h)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=242631=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242631
]
ASF GitHub Bot logged work on BEAM-7305:
Author: ASF GitHub Bot
Created on: 15/May/19 16:25
Start Date: 15/May/19 16:25
Worklog Time Spent: 10m
Work Description: adude3141 commented on pull request #8410: [BEAM-7305]
Add first version of Hazelcast Jet based Java Runner
URL: https://github.com/apache/beam/pull/8410#discussion_r284342494
##
File path: runners/jet-experimental/build.gradle
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature()
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+
+project.ext {
+jet_version = '3.0'
+hazelcast_version = '3.12'
+}
+
+configurations {
+validatesRunner
+}
+
+dependencies {
+shadow project(path: ":sdks:java:core", configuration: "shadow")
+shadow project(path: ":runners:core-java", configuration: "shadow")
+shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+shadowTest library.java.hamcrest_core
+shadowTest library.java.junit
+shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
+shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+validatesRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
+validatesRunner project(path: ":runners:core-java", configuration:
"shadowTest")
+validatesRunner project(path: project.path, configuration: "shadowTest")
+}
+
+task validatesRunnerBatch(type: Test) {
+group = "Verification"
+systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+"--runner=TestJetRunner",
+"--jetGroupName=jet",
+"--jetLocalParallelism=2"
+])
+
+classpath = configurations.validatesRunner
+testClassesDirs =
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse
doesn't cooperate properly with Jet when multiple cluster members are used
+exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality
not yet in the runner
+}
+
+maxHeapSize = '4g'
+}
+
+task validatesRunner {
+group = "Verification"
+description "Validates Jet runner"
+dependsOn validatesRunnerBatch
+}
+
+spotless {
+java {
+paddedCell()
Review comment:
Could you elaborate on why this is needed?
Currently, `spotlessJavaCheck`passes, but `spotlessApply`resolve some issues
in codebase. Removing `paddedCell` will make `spotlessJavaCheck` fail.
So I am wondering, whether we could just remove this configuration?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 242631)
Time Spent: 0.5h (was: 20m)
> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
> Issue Type: New Feature
> Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=241964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241964 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 14/May/19 19:10 Start Date: 14/May/19 19:10 Worklog Time Spent: 10m Work Description: mxm commented on issue #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410#issuecomment-492370158 Thanks @jbartok! Could you assign this JIRA issue to yourself? https://jira.apache.org/jira/browse/BEAM-7305 If you do not have permissions, please ping me and provide your JIRA username. You mentioned that you wanted to include the Runner in the release. Please open a PR against the `release-2.13.0` branch for that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241964) Time Spent: 20m (was: 10m) > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Priority: Major > Fix For: 2.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner
[ https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=241962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241962 ] ASF GitHub Bot logged work on BEAM-7305: Author: ASF GitHub Bot Created on: 14/May/19 19:08 Start Date: 14/May/19 19:08 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner URL: https://github.com/apache/beam/pull/8410 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241962) Time Spent: 10m Remaining Estimate: 0h > Add first version of Hazelcast Jet Runner > - > > Key: BEAM-7305 > URL: https://issues.apache.org/jira/browse/BEAM-7305 > Project: Beam > Issue Type: New Feature > Components: runner-jet >Reporter: Maximilian Michels >Priority: Major > Fix For: 2.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
