[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 32712a7a3ff7842575a78a9f7d2f551784d481b6 Author: Mergebot AuthorDate: Thu Sep 6 05:43:58 2018 + Prepare repository for deployment. --- content/contribute/portability/index.html | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/contribute/portability/index.html b/content/contribute/portability/index.html index cb90344..a39d239 100644 --- a/content/contribute/portability/index.html +++ b/content/contribute/portability/index.html @@ -426,12 +426,13 @@ for details. To run a basic Python wordcount (in batch mode) with embedded Flink: - Run once to build the SDK harness container: ./gradlew -p sdks/python/container -docker + Run once to build the SDK harness container: ./gradlew :beam-sdks-python-container:docker Start the Flink portable JobService endpoint: ./gradlew :beam-runners-flink_2.11-job-server:runShadow - Submit the wordcount pipeline to above endpoint: ./gradlew :beam-sdks-python:portableWordCount + Submit the wordcount pipeline to above endpoint: ./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099 +To run the pipeline in streaming mode: ./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming + To run on a separate https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html;>Flink cluster: @@ -442,7 +443,6 @@ docker Submit the pipeline. -Note: A subset of the functionality is also supported in streaming mode; use --streaming in the command line to enable it.
[beam-site] branch asf-site updated (3adb0c6 -> 32712a7)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 3adb0c6 Prepare repository for deployment. add 510b769 Fix portable Flink runner instructions. add c54d1bb This closes #548 new 32712a7 Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/portability/index.html | 8 src/contribute/portability.md | 8 2 files changed, 8 insertions(+), 8 deletions(-)
[beam-site] 02/02: This closes #548
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit c54d1bb64e53f155ac8aa5b9d3d8d67042a81db0 Merge: 3adb0c6 510b769 Author: Mergebot AuthorDate: Thu Sep 6 05:41:07 2018 + This closes #548 src/contribute/portability.md | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam-site] branch mergebot updated (aa437b2 -> c54d1bb)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git. from aa437b2 This closes #543 add 3adb0c6 Prepare repository for deployment. new 510b769 Fix portable Flink runner instructions. new c54d1bb This closes #548 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/design-documents/index.html | 2 +- src/contribute/portability.md | 8 2 files changed, 5 insertions(+), 5 deletions(-)
[beam-site] 01/02: Fix portable Flink runner instructions.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 510b7690525db609b3ff2aab4794286defc63507 Author: Thomas Weise AuthorDate: Wed Sep 5 22:58:27 2018 +0200 Fix portable Flink runner instructions. --- src/contribute/portability.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/contribute/portability.md b/src/contribute/portability.md index c3abaad..4a4d9bb 100644 --- a/src/contribute/portability.md +++ b/src/contribute/portability.md @@ -183,10 +183,11 @@ for details. The Flink runner is currently the only runner to support portable pipeline execution. To run a basic Python wordcount (in batch mode) with embedded Flink: -1. Run once to build the SDK harness container: `./gradlew -p sdks/python/container -docker` +1. Run once to build the SDK harness container: `./gradlew :beam-sdks-python-container:docker` 2. Start the Flink portable JobService endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow` -3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount` +3. Submit the wordcount pipeline to above endpoint: `./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099` + +To run the pipeline in streaming mode: `./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming` To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html): @@ -196,4 +197,3 @@ To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink- 4. Start JobService with Flink web service endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081` 5. Submit the pipeline. -Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.
[jira] [Commented] (BEAM-5328) Java starter archetype does not contain dependency versions
[
https://issues.apache.org/jira/browse/BEAM-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605231#comment-16605231
]
Alan Myrvold commented on BEAM-5328:
[https://github.com/apache/beam/blob/519e91e6de516a2800b23bec77a78acf8b94f075/release/src/main/groovy/QuickstartArchetype.groovy#L25]
Should be easy to add a test for the starter.
[https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/build.gradle]
has the replace tokens calls
[https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/starter/build.gradle]
does not
> Java starter archetype does not contain dependency versions
> ---
>
> Key: BEAM-5328
> URL: https://issues.apache.org/jira/browse/BEAM-5328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Luke Cwik
>Priority: Major
> Labels: newbie, starter
>
> The starter archetype contains resource annotation markers instead of
> versions:
> {code:java}
> @maven-compiler-plugin.version@
> @maven-exec-plugin.version@
> @slf4j.version@
> {code}
> in the properties block at the top.
>
> This means that the starter project is broken without the user manually
> editing the pom.xml that is generated and populating the versions at the top.
>
> We also lack testing that validates that the starter archetype works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[beam] 01/01: Merge pull request #6332 from qinyeli/master
This is an automated email from the ASF dual-hosted git repository. ccy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 519e91e6de516a2800b23bec77a78acf8b94f075 Merge: 0749dc5 c6290c4 Author: Charles Chen AuthorDate: Wed Sep 5 20:32:43 2018 -0700 Merge pull request #6332 from qinyeli/master Interactive Beam -- getting started guide .../apache_beam/runners/interactive/README.md | 266 + .../examples/Interactive Beam Example.ipynb| 432 + .../Interactive Beam Running on Flink.ipynb| 419 3 files changed, 1117 insertions(+)
[beam] branch master updated (0749dc5 -> 519e91e)
This is an automated email from the ASF dual-hosted git repository. ccy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0749dc5 Merge pull request #6335: [BEAM-5308] Correct/cleanup DockerOnMac code in DockerJobBundleFactory add c6290c4 Interactive Beam -- getting started guide new 519e91e Merge pull request #6332 from qinyeli/master The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache_beam/runners/interactive/README.md | 266 + .../examples/Interactive Beam Example.ipynb| 432 + .../Interactive Beam Running on Flink.ipynb| 419 3 files changed, 1117 insertions(+) create mode 100644 sdks/python/apache_beam/runners/interactive/README.md create mode 100644 sdks/python/apache_beam/runners/interactive/examples/Interactive Beam Example.ipynb create mode 100644 sdks/python/apache_beam/runners/interactive/examples/Interactive Beam Running on Flink.ipynb
[jira] [Comment Edited] (BEAM-5296) python beam_integration_benchmark(0) continuously failed.
[ https://issues.apache.org/jira/browse/BEAM-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605115#comment-16605115 ] Mark Liu edited comment on BEAM-5296 at 9/6/18 12:55 AM: - Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 If no further progress on the fix, we probably should consider upgrade fastavro on Jenkins nodes or rollback relevant [PR|https://github.com/apache/beam/pull/6321] since it affects many performance benchmarks. was (Author: markflyhigh): Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 If no further progress on the fix, we probably should consider upgrade fastavro on Jenkins nodes or rollback relevant PR https://github.com/apache/beam/pull/6321 > python beam_integration_benchmark(0) continuously failed. > - > > Key: BEAM-5296 > URL: https://issues.apache.org/jira/browse/BEAM-5296 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Mark Liu >Priority: Major > > beam_integration_benchmark0) failed > https://builds.apache.org/job/beam_PerformanceTests_Python/1377/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5296) python beam_integration_benchmark(0) continuously failed.
[ https://issues.apache.org/jira/browse/BEAM-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605115#comment-16605115 ] Mark Liu edited comment on BEAM-5296 at 9/6/18 12:54 AM: - Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 If no further progress on the fix, we probably should consider upgrade fastavro on Jenkins nodes or rollback relevant PR https://github.com/apache/beam/pull/6321 was (Author: markflyhigh): Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 We probably should consider upgrade fastavro on Jenkins nodes or rollback relevant PR https://github.com/apache/beam/pull/6321 > python beam_integration_benchmark(0) continuously failed. > - > > Key: BEAM-5296 > URL: https://issues.apache.org/jira/browse/BEAM-5296 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Mark Liu >Priority: Major > > beam_integration_benchmark0) failed > https://builds.apache.org/job/beam_PerformanceTests_Python/1377/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5296) python beam_integration_benchmark(0) continuously failed.
[ https://issues.apache.org/jira/browse/BEAM-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605115#comment-16605115 ] Mark Liu edited comment on BEAM-5296 at 9/6/18 12:52 AM: - Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 We probably should consider upgrade fastavro on Jenkins nodes or rollback relevant PR https://github.com/apache/beam/pull/6321 was (Author: markflyhigh): Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 > python beam_integration_benchmark(0) continuously failed. > - > > Key: BEAM-5296 > URL: https://issues.apache.org/jira/browse/BEAM-5296 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Mark Liu >Priority: Major > > beam_integration_benchmark0) failed > https://builds.apache.org/job/beam_PerformanceTests_Python/1377/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5296) python beam_integration_benchmark(0) continuously failed.
[ https://issues.apache.org/jira/browse/BEAM-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605115#comment-16605115 ] Mark Liu commented on BEAM-5296: Investigations are made by Lukasz and summarized in https://github.com/apache/beam/pull/6321#issuecomment-418339689 > python beam_integration_benchmark(0) continuously failed. > - > > Key: BEAM-5296 > URL: https://issues.apache.org/jira/browse/BEAM-5296 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Mark Liu >Priority: Major > > beam_integration_benchmark0) failed > https://builds.apache.org/job/beam_PerformanceTests_Python/1377/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141585 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 06/Sep/18 00:14 Start Date: 06/Sep/18 00:14 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418921593 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141585) Time Spent: 3h 20m (was: 3h 10m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141584 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 06/Sep/18 00:13 Start Date: 06/Sep/18 00:13 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418921460 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141584) Time Spent: 3h 10m (was: 3h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141583 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 06/Sep/18 00:12 Start Date: 06/Sep/18 00:12 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418921172 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141583) Time Spent: 3h (was: 2h 50m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141582 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 06/Sep/18 00:08 Start Date: 06/Sep/18 00:08 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418920474 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141582) Time Spent: 2h 50m (was: 2h 40m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141580 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 06/Sep/18 00:03 Start Date: 06/Sep/18 00:03 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418919687 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141580) Time Spent: 2h 40m (was: 2.5h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141578 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:56 Start Date: 05/Sep/18 23:56 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418918357 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141578) Time Spent: 2.5h (was: 2h 20m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141571 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:40 Start Date: 05/Sep/18 23:40 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418915687 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141571) Time Spent: 2h 20m (was: 2h 10m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141570 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:37 Start Date: 05/Sep/18 23:37 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418915321 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141570) Time Spent: 2h 10m (was: 2h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141569 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:37 Start Date: 05/Sep/18 23:37 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418915227 Python SDK PostCommit Tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141569) Time Spent: 2h (was: 1h 50m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141566 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:33 Start Date: 05/Sep/18 23:33 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418914603 Python SDK PostCommit Tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141566) Time Spent: 1h 50m (was: 1h 40m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141565 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:30 Start Date: 05/Sep/18 23:30 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418914068 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141565) Time Spent: 1h 40m (was: 1.5h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141563=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141563 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:27 Start Date: 05/Sep/18 23:27 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418913503 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141563) Time Spent: 1.5h (was: 1h 20m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141554 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 23:02 Start Date: 05/Sep/18 23:02 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418909126 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141554) Time Spent: 1h 20m (was: 1h 10m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141551 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 22:57 Start Date: 05/Sep/18 22:57 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418908108 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141551) Time Spent: 1h 10m (was: 1h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141550 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 22:52 Start Date: 05/Sep/18 22:52 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418907034 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141550) Time Spent: 1h (was: 50m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5329) Increase test coverage of Perfkit benchmark for Python case
Mark Liu created BEAM-5329: -- Summary: Increase test coverage of Perfkit benchmark for Python case Key: BEAM-5329 URL: https://issues.apache.org/jira/browse/BEAM-5329 Project: Beam Issue Type: Test Components: testing Reporter: Mark Liu Assignee: Jason Kuster Perfkit benchmark is defined in PerfKitBenchmarker repo (mostly [here|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py] and [here|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py]) but not enough test coverage to verify changes to the benchmark or it's utils. Breakage can be surfaced only when scheduled Beam Postcommit job runs which will directly affect performance measurement. We want to add more tests starting from unit tests to cover missing branches and utils like Python benchmark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5328) Java starter archetype does not contain dependency versions
[
https://issues.apache.org/jira/browse/BEAM-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605008#comment-16605008
]
Luke Cwik commented on BEAM-5328:
-
[~alanmyrvold], can you link to where we test our examples archetypes so that
they work. Would it be difficult to test the starter archetype as well?
> Java starter archetype does not contain dependency versions
> ---
>
> Key: BEAM-5328
> URL: https://issues.apache.org/jira/browse/BEAM-5328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Luke Cwik
>Priority: Major
> Labels: newbie, starter
>
> The starter archetype contains resource annotation markers instead of
> versions:
> {code:java}
> @maven-compiler-plugin.version@
> @maven-exec-plugin.version@
> @slf4j.version@
> {code}
> in the properties block at the top.
>
> This means that the starter project is broken without the user manually
> editing the pom.xml that is generated and populating the versions at the top.
>
> We also lack testing that validates that the starter archetype works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5308) JobBundleFactory BindException with FlinkRunner and remote cluster
[ https://issues.apache.org/jira/browse/BEAM-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605007#comment-16605007 ] Thomas Weise commented on BEAM-5308: After the port range fix multiple jobs can run on the cluster. However, the second issue of containers not terminating still exists. The docker containers remain active after the job has finished and are only removed when the Flink cluster is stopped. That's different from behavior in embedded mode, where the containers exit after 30s. [~angoenka] any ideas? > JobBundleFactory BindException with FlinkRunner and remote cluster > -- > > Key: BEAM-5308 > URL: https://issues.apache.org/jira/browse/BEAM-5308 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability > Time Spent: 0.5h > Remaining Estimate: 0h > > Repeated execution of the same job on remote Flink cluster (not embedded in > job server) fails with bind exception. There seem to be 2 issues: > * Multiple instances of job bundle factory cannot be created (port conflict) > * Job bundle factory is not released after job completes (and Docker > container keeps on running). That's not the case in embedded mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5328) Java starter archetype does not contain dependency versions
[
https://issues.apache.org/jira/browse/BEAM-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik updated BEAM-5328:
Labels: newbie starter (was: )
> Java starter archetype does not contain dependency versions
> ---
>
> Key: BEAM-5328
> URL: https://issues.apache.org/jira/browse/BEAM-5328
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Luke Cwik
>Priority: Major
> Labels: newbie, starter
>
> The starter archetype contains resource annotation markers instead of
> versions:
> {code:java}
> @maven-compiler-plugin.version@
> @maven-exec-plugin.version@
> @slf4j.version@
> {code}
> in the properties block at the top.
>
> This means that the starter project is broken without the user manually
> editing the pom.xml that is generated and populating the versions at the top.
>
> We also lack testing that validates that the starter archetype works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5328) Java starter archetype does not contain dependency versions
Luke Cwik created BEAM-5328:
---
Summary: Java starter archetype does not contain dependency
versions
Key: BEAM-5328
URL: https://issues.apache.org/jira/browse/BEAM-5328
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Affects Versions: 2.6.0, 2.7.0
Reporter: Luke Cwik
The starter archetype contains resource annotation markers instead of versions:
{code:java}
@maven-compiler-plugin.version@
@maven-exec-plugin.version@
@slf4j.version@
{code}
in the properties block at the top.
This means that the starter project is broken without the user manually editing
the pom.xml that is generated and populating the versions at the top.
We also lack testing that validates that the starter archetype works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5308) JobBundleFactory BindException with FlinkRunner and remote cluster
[
https://issues.apache.org/jira/browse/BEAM-5308?focusedWorklogId=141542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141542
]
ASF GitHub Bot logged work on BEAM-5308:
Author: ASF GitHub Bot
Created on: 05/Sep/18 22:14
Start Date: 05/Sep/18 22:14
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6335: [BEAM-5308]
Correct/cleanup DockerOnMac code in DockerJobBundleFactory
URL: https://github.com/apache/beam/pull/6335
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
index 3178a2e2b3b..fec29231a97 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
@@ -43,11 +43,6 @@
public class DockerJobBundleFactory extends JobBundleFactoryBase {
private static final Logger LOG =
LoggerFactory.getLogger(DockerJobBundleFactory.class);
- // Port offset for MacOS since we don't have host networking and need to use
published ports
- private static final int MAC_PORT_START = 8100;
- private static final int MAC_PORT_END = 8200;
- private static final AtomicInteger MAC_PORT = new
AtomicInteger(MAC_PORT_START);
-
/** Factory that creates {@link JobBundleFactory} for the given {@link
JobInfo}. */
public interface JobBundleFactoryFactory {
JobBundleFactory create(JobInfo jobInfo) throws Exception;
@@ -63,10 +58,6 @@ public JobBundleFactory create(JobInfo jobInfo) throws
Exception {
}
});
- // TODO: This host name seems to change with every other Docker release. Do
we attempt to keep up
- // or attempt to document the supported Docker version(s)?
- private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
-
public static JobBundleFactory create(JobInfo jobInfo) throws Exception {
return FACTORY.get().create(jobInfo);
}
@@ -113,21 +104,7 @@ protected ServerFactory getServerFactory() {
case LINUX:
return ServerFactory.createDefault();
case MAC:
-// NOTE: Deployment on Macs is intended for local development. As of
18.03, Docker-for-Mac
-// does not implement host networking (--networking=host is
effectively a no-op). Instead,
-// we use a special DNS entry that points to the host:
-//
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds
-// The special hostname has historically changed between versions, so
this is subject to
-// breakages and will likely only support the latest version at any
time.
-
-// We need to use a fixed port range due to non-existing host
networking in Docker-for-Mac.
-// The port range needs to be published when bringing up the Docker
container, see
-// DockerEnvironmentFactory.
-
-return ServerFactory.createWithUrlFactoryAndPortSupplier(
-(host, port) -> HostAndPort.fromParts(DOCKER_FOR_MAC_HOST,
port).toString(),
-// We only use the published Docker ports 8100-8200 in a
round-robin fashion
-() -> MAC_PORT.getAndUpdate(val -> val == MAC_PORT_END ?
MAC_PORT_START : val + 1));
+return DockerOnMac.getServerFactory();
default:
LOG.warn("Unknown Docker platform. Falling back to default server
factory");
return ServerFactory.createDefault();
@@ -140,7 +117,7 @@ private static Platform getPlatform() {
// The DOCKER_MAC_CONTAINER environment variable is necessary to detect
whether we run on
// a container on MacOs. MacOs internally uses a Linux VM which makes it
indistinguishable from Linux.
// We still need to apply port mapping due to missing host networking.
-if (osName.startsWith("mac") ||
"1".equals(System.getenv("DOCKER_MAC_CONTAINER"))) {
+if (osName.startsWith("mac") || DockerOnMac.RUNNING_INSIDE_DOCKER_ON_MAC) {
return Platform.MAC;
} else if (osName.startsWith("linux")) {
return Platform.LINUX;
@@ -154,6 +131,44 @@ private static Platform getPlatform() {
OTHER,
}
+ /**
+ * NOTE: Deployment on Macs is intended for local development. As of 18.03,
Docker-for-Mac does
+ * not implement host networking (--networking=host is effectively a no-op).
Instead, we use a
+ * special DNS entry that points to the host:
+ *
[beam] 01/01: Merge pull request #6335: [BEAM-5308] Correct/cleanup DockerOnMac code in DockerJobBundleFactory
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0749dc56348880b30d7d6824adf125c88ab895ce Merge: 96df192 d701a91 Author: Thomas Weise AuthorDate: Thu Sep 6 00:14:16 2018 +0200 Merge pull request #6335: [BEAM-5308] Correct/cleanup DockerOnMac code in DockerJobBundleFactory .../control/DockerJobBundleFactory.java| 65 +- 1 file changed, 40 insertions(+), 25 deletions(-)
[beam] branch master updated (96df192 -> 0749dc5)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 96df192 Merge pull request #6290 from amaliujia/rui_wang-correct_test_case_of_complex_type add d701a91 [BEAM-5308] Correct/cleanup DockerOnMac code in DockerJobBundleFactory new 0749dc5 Merge pull request #6335: [BEAM-5308] Correct/cleanup DockerOnMac code in DockerJobBundleFactory The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../control/DockerJobBundleFactory.java| 65 +- 1 file changed, 40 insertions(+), 25 deletions(-)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141541 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 22:12 Start Date: 05/Sep/18 22:12 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340#issuecomment-418898557 Run Python Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141541) Time Spent: 50m (was: 40m) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?focusedWorklogId=141540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141540 ] ASF GitHub Bot logged work on BEAM-5283: Author: ASF GitHub Bot Created on: 05/Sep/18 22:12 Start Date: 05/Sep/18 22:12 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6340: [BEAM-5283] Fixing Comment in Flink Post commit jenkins task URL: https://github.com/apache/beam/pull/6340 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141540) Time Spent: 40m (was: 0.5h) > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA
[jira] [Work logged] (BEAM-5262) JobState support for Reference Runner
[ https://issues.apache.org/jira/browse/BEAM-5262?focusedWorklogId=141539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141539 ] ASF GitHub Bot logged work on BEAM-5262: Author: ASF GitHub Bot Created on: 05/Sep/18 21:38 Start Date: 05/Sep/18 21:38 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6301: [BEAM-5262] Add Reference runner support for add state stream URL: https://github.com/apache/beam/pull/6301#issuecomment-418890184 Ping! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141539) Time Spent: 0.5h (was: 20m) > JobState support for Reference Runner > - > > Key: BEAM-5262 > URL: https://issues.apache.org/jira/browse/BEAM-5262 > Project: Beam > Issue Type: Bug > Components: runner-direct >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Reference runner does not support getStateStream which is needed by portable > SDK -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5301) Migrate integration tests for datastore_wordcount
[ https://issues.apache.org/jira/browse/BEAM-5301?focusedWorklogId=141531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141531 ] ASF GitHub Bot logged work on BEAM-5301: Author: ASF GitHub Bot Created on: 05/Sep/18 20:59 Start Date: 05/Sep/18 20:59 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #6333: [BEAM-5301] migrate datastore_worcount integration test to Beam URL: https://github.com/apache/beam/pull/6333#issuecomment-418879614 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141531) Time Spent: 20m (was: 10m) > Migrate integration tests for datastore_wordcount > - > > Key: BEAM-5301 > URL: https://issues.apache.org/jira/browse/BEAM-5301 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 3adb0c6f504df0fc04477e8dddb36077eeebfc3d Author: Mergebot AuthorDate: Wed Sep 5 20:59:24 2018 + Prepare repository for deployment. --- content/contribute/design-documents/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/contribute/design-documents/index.html b/content/contribute/design-documents/index.html index 505efbb..fb2559e 100644 --- a/content/contribute/design-documents/index.html +++ b/content/contribute/design-documents/index.html @@ -275,7 +275,7 @@ limitations under the License. Error Reporting Callback (WIP) [https://docs.google.com/document/d/1o2VXwCL97k3G-1BR9RSKNc6XtJTIA6SEKPMne91S67Y;>doc] Snapshotting and Updating Beam Pipelines [https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY;>doc] Requiring PTransform to set a coder on its resulting collections [https://lists.apache.org/thread.html/1dde0b5a93c2983cbab5f68ce7c74580102f5bb2baaa816585d7eabb@%3Cdev.beam.apache.org%3E;>mail] - [PROPOSAL] Support of @RequiresStableInput annotation [https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM;>doc], [https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E;>mail] + Support of @RequiresStableInput annotation [https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM;>doc], [https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E;>mail] [PROPOSAL] @onwindowexpiration [https://lists.apache.org/thread.html/1dab7f17c97378e665928b6cbd887dc7be93390ab26c593ee49a@%3Cdev.beam.apache.org%3E;>mail]
[beam-site] branch asf-site updated (5a00fd9 -> 3adb0c6)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 5a00fd9 Prepare repository for deployment. add 288f55d Remove the [PROPOSAL] label of the @RequiresStableInput design doc add aa437b2 This closes #543 new 3adb0c6 Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/design-documents/index.html | 2 +- src/contribute/design-documents.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam-site] branch mergebot updated (49563e7 -> aa437b2)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 49563e7 This closes #545 add 5a00fd9 Prepare repository for deployment. new 288f55d Remove the [PROPOSAL] label of the @RequiresStableInput design doc new aa437b2 This closes #543 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/index.html | 36 src/contribute/design-documents.md | 2 +- 2 files changed, 37 insertions(+), 1 deletion(-)
[beam-site] 02/02: This closes #543
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit aa437b26935f27c97e0ee3a855bf8b21776c477b Merge: 5a00fd9 288f55d Author: Mergebot AuthorDate: Wed Sep 5 20:56:26 2018 + This closes #543 src/contribute/design-documents.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam-site] 01/02: Remove the [PROPOSAL] label of the @RequiresStableInput design doc
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 288f55dc1594ed5be14a26f1d8a31732a8279f18 Author: Yueyang Qiu AuthorDate: Mon Aug 27 16:03:12 2018 -0700 Remove the [PROPOSAL] label of the @RequiresStableInput design doc --- src/contribute/design-documents.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/contribute/design-documents.md b/src/contribute/design-documents.md index a37beca..deed450 100644 --- a/src/contribute/design-documents.md +++ b/src/contribute/design-documents.md @@ -53,7 +53,7 @@ This is a collection of documents that may or may not be up to date. - Error Reporting Callback (WIP) [[doc](https://docs.google.com/document/d/1o2VXwCL97k3G-1BR9RSKNc6XtJTIA6SEKPMne91S67Y)] - Snapshotting and Updating Beam Pipelines [[doc](https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY)] - Requiring PTransform to set a coder on its resulting collections [[mail](https://lists.apache.org/thread.html/1dde0b5a93c2983cbab5f68ce7c74580102f5bb2baaa816585d7eabb@%3Cdev.beam.apache.org%3E)] -- [PROPOSAL] Support of @RequiresStableInput annotation [[doc](https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM)], [[mail](https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E)] +- Support of @RequiresStableInput annotation [[doc](https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM)], [[mail](https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E)] - [PROPOSAL] @onwindowexpiration [[mail](https://lists.apache.org/thread.html/1dab7f17c97378e665928b6cbd887dc7be93390ab26c593ee49a@%3Cdev.beam.apache.org%3E)] ### IO / Filesystem
[beam-site] branch asf-site updated (2aca6fa -> 5a00fd9)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 2aca6fa Prepare repository for deployment. add f1dcace Contribute: document how to find and add reviewers add 49563e7 This closes #545 new 5a00fd9 Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/index.html | 36 src/contribute/index.md | 30 ++ 2 files changed, 66 insertions(+)
[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 5a00fd9a0d3de646f5a4d4cea554d3930e0231e8 Author: Mergebot AuthorDate: Wed Sep 5 20:55:27 2018 + Prepare repository for deployment. --- content/contribute/index.html | 36 1 file changed, 36 insertions(+) diff --git a/content/contribute/index.html b/content/contribute/index.html index c6f1426..b2767bf 100644 --- a/content/contribute/index.html +++ b/content/contribute/index.html @@ -207,6 +207,12 @@ Developing with the Python SDK + Reviews + + Finding reviewers + Adding yourself as a reviewer + + Contributing to the website Works in progress @@ -479,6 +485,36 @@ $ python setup.py sdist +Reviews + +Reviewers for https://github.com/apache/beam;>apache/beam are listed in +Prow-style OWNERS files. A description of these files can be found +https://go.k8s.io/owners;>here. + +Finding reviewers + +Currently this is a manual process. Tracking bug for automating this: +https://issues.apache.org/jira/browse/BEAM-4790;>BEAM-4790. + +For each file to be reviewed, look for an OWNERS file in its directory. Pick a +single reviewer from that file. If the directory doesn’t contain an OWNERS file, +go up a directory. Keep going until you find one. Try to limit the number of +reviewers to 2 per PR if possible, to minimize reviewer load. + +Adding yourself as a reviewer + +Find the deepest sub-directory that contains the files you want to be a reviewer +for and add your Github username under reviewers in the OWNERS file (create a +new OWNERS file if necessary). + +The Beam project currently only uses the reviewers key in OWNERS and no other +features, as reviewer selection is still a manual process. + + + Contributing to the website The Beam website is in the https://github.com/apache/beam-site;>Beam Site GitHub
[jira] [Work logged] (BEAM-4790) Make it easier to select code reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=141530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141530 ] ASF GitHub Bot logged work on BEAM-4790: Author: ASF GitHub Bot Created on: 05/Sep/18 20:55 Start Date: 05/Sep/18 20:55 Worklog Time Spent: 10m Work Description: asfgit closed pull request #545: [BEAM-4790] Contribute: document how to find and add reviewers URL: https://github.com/apache/beam-site/pull/545 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/src/contribute/index.md b/src/contribute/index.md index b202214218..a26ca8939a 100644 --- a/src/contribute/index.md +++ b/src/contribute/index.md @@ -228,6 +228,36 @@ $ python setup.py sdist > /dev/null && \ --sdk_location dist/apache-beam-2.5.0.dev0.tar.gz ``` +## Reviews + +Reviewers for [apache/beam](https://github.com/apache/beam) are listed in +Prow-style OWNERS files. A description of these files can be found +[here](https://go.k8s.io/owners). + +### Finding reviewers + +Currently this is a manual process. Tracking bug for automating this: +[BEAM-4790](https://issues.apache.org/jira/browse/BEAM-4790). + +For each file to be reviewed, look for an OWNERS file in its directory. Pick a +single reviewer from that file. If the directory doesn't contain an OWNERS file, +go up a directory. Keep going until you find one. Try to limit the number of +reviewers to 2 per PR if possible, to minimize reviewer load. + +### Adding yourself as a reviewer + +Find the deepest sub-directory that contains the files you want to be a reviewer +for and add your Github username under `reviewers` in the OWNERS file (create a +new OWNERS file if necessary). + +The Beam project currently only uses the `reviewers` key in OWNERS and no other +features, as reviewer selection is still a manual process. + + + ## Contributing to the website The Beam website is in the [Beam Site GitHub This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141530) Time Spent: 3h 40m (was: 3.5h) > Make it easier to select code reviewers for PRs > --- > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[beam-site] 01/02: Contribute: document how to find and add reviewers
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit f1dcace7db47779c5a9589202c0522687d15a815 Author: Udi Meiri AuthorDate: Fri Aug 31 11:42:37 2018 -0700 Contribute: document how to find and add reviewers --- src/contribute/index.md | 30 ++ 1 file changed, 30 insertions(+) diff --git a/src/contribute/index.md b/src/contribute/index.md index b202214..a26ca89 100644 --- a/src/contribute/index.md +++ b/src/contribute/index.md @@ -228,6 +228,36 @@ $ python setup.py sdist > /dev/null && \ --sdk_location dist/apache-beam-2.5.0.dev0.tar.gz ``` +## Reviews + +Reviewers for [apache/beam](https://github.com/apache/beam) are listed in +Prow-style OWNERS files. A description of these files can be found +[here](https://go.k8s.io/owners). + +### Finding reviewers + +Currently this is a manual process. Tracking bug for automating this: +[BEAM-4790](https://issues.apache.org/jira/browse/BEAM-4790). + +For each file to be reviewed, look for an OWNERS file in its directory. Pick a +single reviewer from that file. If the directory doesn't contain an OWNERS file, +go up a directory. Keep going until you find one. Try to limit the number of +reviewers to 2 per PR if possible, to minimize reviewer load. + +### Adding yourself as a reviewer + +Find the deepest sub-directory that contains the files you want to be a reviewer +for and add your Github username under `reviewers` in the OWNERS file (create a +new OWNERS file if necessary). + +The Beam project currently only uses the `reviewers` key in OWNERS and no other +features, as reviewer selection is still a manual process. + + + ## Contributing to the website The Beam website is in the [Beam Site GitHub
[beam-site] branch mergebot updated (d065f1e -> 49563e7)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git. from d065f1e This closes #547 add 2aca6fa Prepare repository for deployment. new f1dcace Contribute: document how to find and add reviewers new 49563e7 This closes #545 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/eclipse/index.html | 10 +- src/contribute/index.md | 30 ++ 2 files changed, 31 insertions(+), 9 deletions(-)
[beam-site] 02/02: This closes #545
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 49563e7d97d2b37a2f02869c45976830c7f33fec Merge: 2aca6fa f1dcace Author: Mergebot AuthorDate: Wed Sep 5 20:52:44 2018 + This closes #545 src/contribute/index.md | 30 ++ 1 file changed, 30 insertions(+)
[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 2aca6fa7a6d7be380f1fb5fc941a2edbc29b805c Author: Mergebot AuthorDate: Wed Sep 5 20:52:02 2018 + Prepare repository for deployment. --- content/contribute/eclipse/index.html | 10 +- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/content/contribute/eclipse/index.html b/content/contribute/eclipse/index.html index 6ee2b6d..480e7f2 100644 --- a/content/contribute/eclipse/index.html +++ b/content/contribute/eclipse/index.html @@ -227,15 +227,7 @@ limitations under the License. Eclipse Tips - These are best-effort community-contributed tips, and are not… - - -… guaranteed to work with any particular Eclipse setup. -… the only or best way to work. -… an endorsement of Eclipse over alternatives. - - - Mastering Eclipse is, ultimately, your responsibility. + These are best-effort community-contributed tips, and are not guaranteed to work with any particular Eclipse setup. Eclipse version
[beam-site] branch asf-site updated (7a75616 -> 2aca6fa)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 7a75616 Prepare repository for deployment. add 8a1a418 Update eclipse.md add d065f1e This closes #547 new 2aca6fa Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/eclipse/index.html | 10 +- src/contribute/eclipse.md | 8 +--- 2 files changed, 2 insertions(+), 16 deletions(-)
[jira] [Work logged] (BEAM-5124) Write Euphoria in Beam documentation
[ https://issues.apache.org/jira/browse/BEAM-5124?focusedWorklogId=141529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141529 ] ASF GitHub Bot logged work on BEAM-5124: Author: ASF GitHub Bot Created on: 05/Sep/18 20:50 Start Date: 05/Sep/18 20:50 Worklog Time Spent: 10m Work Description: melap commented on issue #540: [BEAM-5124] DSL Euphoria documentation update URL: https://github.com/apache/beam-site/pull/540#issuecomment-418876911 R: @jbonofre This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141529) Time Spent: 2h 10m (was: 2h) > Write Euphoria in Beam documentation > > > Key: BEAM-5124 > URL: https://issues.apache.org/jira/browse/BEAM-5124 > Project: Beam > Issue Type: Sub-task > Components: dsl-euphoria >Reporter: Vaclav Plajt >Assignee: Vaclav Plajt >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[beam-site] 01/02: Update eclipse.md
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 8a1a418c54a669080519cf0a54795f8a7d3ebed8 Author: Rafael Fernandez AuthorDate: Fri Aug 31 19:12:29 2018 -0700 Update eclipse.md Friendlier language... --- src/contribute/eclipse.md | 8 +--- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/src/contribute/eclipse.md b/src/contribute/eclipse.md index bedf2bd..4eff1f0 100644 --- a/src/contribute/eclipse.md +++ b/src/contribute/eclipse.md @@ -20,13 +20,7 @@ limitations under the License. # Eclipse Tips -> These are best-effort community-contributed tips, and are not... -> -> - ... guaranteed to work with any particular Eclipse setup. -> - ... the only or best way to work. -> - ... an endorsement of Eclipse over alternatives. -> -> Mastering Eclipse is, ultimately, your responsibility. +> These are best-effort community-contributed tips, and are not guaranteed to work with any particular Eclipse setup. ## Eclipse version
[beam-site] 02/02: This closes #547
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit d065f1e64f095dd2f4f98a1fa0463663d3a08851 Merge: 7a75616 8a1a418 Author: Mergebot AuthorDate: Wed Sep 5 20:48:59 2018 + This closes #547 src/contribute/eclipse.md | 8 +--- 1 file changed, 1 insertion(+), 7 deletions(-)
[beam-site] branch mergebot updated (91d5227 -> d065f1e)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 91d5227 This closes #544 add 7a75616 Prepare repository for deployment. new 8a1a418 Update eclipse.md new d065f1e This closes #547 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/portability/index.html | 5 +++-- src/contribute/eclipse.md | 8 +--- 2 files changed, 4 insertions(+), 9 deletions(-)
[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 7a7561660612a9561be7107fd3fc92a9e2eb4a5f Author: Mergebot AuthorDate: Wed Sep 5 20:48:19 2018 + Prepare repository for deployment. --- content/contribute/portability/index.html | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/contribute/portability/index.html b/content/contribute/portability/index.html index bae564b..cb90344 100644 --- a/content/contribute/portability/index.html +++ b/content/contribute/portability/index.html @@ -417,7 +417,7 @@ their respective components. Status MVP in progress (near completion for Flink runner). See the -https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing;>Portability support table +https://s.apache.org/apache-beam-portability-support-table;>Portability support table for details. Running Python wordcount on Flink @@ -437,7 +437,8 @@ docker Start local Flink cluster Create shaded JobService jar: ./gradlew :beam-runners-flink_2.11-job-server:installShadowDist - Start JobService with Flink web service endpoint: java -jar ./runners/flink/job-server/build/install/beam-runners-flink_2.11-job-server-shadow/lib/beam-runners-flink_2.11-job-server-*.jar "--job-host=localhost:8099" "--artifacts-dir=/tmp/flink-artifacts" "--flink-master-url=localhost:8081" + Optional optimization step: Place the generated JobServer Jar beam/runners/flink/job-server/build/libs/beam-runners-flink_2.11-job-server-2.7.0-SNAPSHOT.jar in flink/lib and change class loading order for Flink by adding classloader.resolve-order: parent-first to conf/flink-conf.yaml. + Start JobService with Flink web service endpoint: ./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081 Submit the pipeline.
[beam-site] branch asf-site updated (27eac2e -> 7a75616)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 27eac2e Prepare repository for deployment. add 0c32c28 Update Portable Flink cluster run instructions add 91d5227 This closes #544 new 7a75616 Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/portability/index.html | 5 +++-- src/contribute/portability.md | 7 --- 2 files changed, 7 insertions(+), 5 deletions(-)
[beam-site] 02/02: This closes #544
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 91d522768cd811cc7549b135961e819d130c7c98 Merge: 27eac2e 0c32c28 Author: Mergebot AuthorDate: Wed Sep 5 20:45:00 2018 + This closes #544 src/contribute/portability.md | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-)
[beam-site] branch mergebot updated (6932df9 -> 91d5227)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 6932df9 This closes #546 add 27eac2e Prepare repository for deployment. new 0c32c28 Update Portable Flink cluster run instructions new 91d5227 This closes #544 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/intellij/index.html | 10 +- src/contribute/portability.md | 7 --- 2 files changed, 5 insertions(+), 12 deletions(-)
[beam-site] 01/02: Update Portable Flink cluster run instructions
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 0c32c2823673bcfa5f146be9705b2d469b13d3ab Author: Ankur AuthorDate: Wed Aug 29 14:42:47 2018 -0700 Update Portable Flink cluster run instructions --- src/contribute/portability.md | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/contribute/portability.md b/src/contribute/portability.md index 4e3be37..c3abaad 100644 --- a/src/contribute/portability.md +++ b/src/contribute/portability.md @@ -175,7 +175,7 @@ their respective components. ### Status MVP in progress (near completion for Flink runner). See the -[Portability support table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing) +[Portability support table](https://s.apache.org/apache-beam-portability-support-table) for details. Running Python wordcount on Flink @@ -192,7 +192,8 @@ To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink- 1. Start local Flink cluster 2. Create shaded JobService jar: `./gradlew :beam-runners-flink_2.11-job-server:installShadowDist` -3. Start JobService with Flink web service endpoint: `java -jar ./runners/flink/job-server/build/install/beam-runners-flink_2.11-job-server-shadow/lib/beam-runners-flink_2.11-job-server-*.jar "--job-host=localhost:8099" "--artifacts-dir=/tmp/flink-artifacts" "--flink-master-url=localhost:8081"` -4. Submit the pipeline. +3. Optional optimization step: Place the generated JobServer Jar `beam/runners/flink/job-server/build/libs/beam-runners-flink_2.11-job-server-2.7.0-SNAPSHOT.jar` in `flink/lib` and change class loading order for Flink by adding `classloader.resolve-order: parent-first` to `conf/flink-conf.yaml`. +4. Start JobService with Flink web service endpoint: `./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081` +5. Submit the pipeline. Note: A subset of the functionality is also supported in streaming mode; use `--streaming` in the command line to enable it.
[jira] [Work logged] (BEAM-4790) Make it easier to select code reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=141528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141528 ] ASF GitHub Bot logged work on BEAM-4790: Author: ASF GitHub Bot Created on: 05/Sep/18 20:45 Start Date: 05/Sep/18 20:45 Worklog Time Spent: 10m Work Description: melap commented on issue #545: [BEAM-4790] Contribute: document how to find and add reviewers URL: https://github.com/apache/beam-site/pull/545#issuecomment-418875200 @asfgit merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141528) Time Spent: 3.5h (was: 3h 20m) > Make it easier to select code reviewers for PRs > --- > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5258) Investigate if we can disable Row type flattening in Calcite
[ https://issues.apache.org/jira/browse/BEAM-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604918#comment-16604918 ] Rui Wang commented on BEAM-5258: The feedback from Calcite community saying that disabling flatterer makes sense. It might not be a minor change though. > Investigate if we can disable Row type flattening in Calcite > > > Key: BEAM-5258 > URL: https://issues.apache.org/jira/browse/BEAM-5258 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > > Either disable the flattening in PlannerImpl or Flattener could be a good > start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[beam-site] 01/01: Prepare repository for deployment.
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 27eac2e570dac5df1a1767f9e6aa19851d8a2dd8 Author: Mergebot AuthorDate: Wed Sep 5 20:44:19 2018 + Prepare repository for deployment. --- content/contribute/intellij/index.html | 10 +- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/content/contribute/intellij/index.html b/content/contribute/intellij/index.html index 5da99be..c4de113 100644 --- a/content/contribute/intellij/index.html +++ b/content/contribute/intellij/index.html @@ -225,15 +225,7 @@ limitations under the License. IntelliJ Tips - These are best-effort community-contributed tips, and are not… - - -… guaranteed to work with any particular IntelliJ setup. -… the only or best way to work. -… an endorsement of IntelliJ over alternatives. - - - Mastering IntelliJ is, ultimately, your responsibility. + These are best-effort community-contributed tips, and are not guaranteed to work with any particular IntelliJ setup. Create a working Gradle module
[beam-site] branch asf-site updated (423f9e8 -> 27eac2e)
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git. from 423f9e8 Prepare repository for deployment. add 1627e8b Update intellij.md add 6932df9 This closes #546 new 27eac2e Prepare repository for deployment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/contribute/intellij/index.html | 10 +- src/contribute/intellij.md | 8 +--- 2 files changed, 2 insertions(+), 16 deletions(-)
[beam-site] 02/02: This closes #546
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 6932df9bde9c39892210b0128f6412828620a0f6 Merge: 423f9e8 1627e8b Author: Mergebot AuthorDate: Wed Sep 5 20:41:16 2018 + This closes #546 src/contribute/intellij.md | 8 +--- 1 file changed, 1 insertion(+), 7 deletions(-)
[beam-site] branch mergebot updated (d8070cb -> 6932df9)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from d8070cb This closes #542
add 423f9e8 Prepare repository for deployment.
new 1627e8b Update intellij.md
new 6932df9 This closes #546
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../2018/08/21/beam-summit-europe.html}| 61 +-
content/blog/index.html| 19 +
content/feed.xml | 639 ++---
{src => content}/images/blog/Facebook-AD.png | Bin
content/index.html | 10 +-
src/contribute/intellij.md | 8 +-
6 files changed, 98 insertions(+), 639 deletions(-)
copy content/{beam/capability/2016/04/03/presentation-materials.html =>
blog/2018/08/21/beam-summit-europe.html} (74%)
copy {src => content}/images/blog/Facebook-AD.png (100%)
[beam-site] 01/02: Update intellij.md
This is an automated email from the ASF dual-hosted git repository. mergebot-role pushed a commit to branch mergebot in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 1627e8bd84eb04265f38ffe89e53e6bbc58e Author: Rafael Fernandez AuthorDate: Fri Aug 31 19:11:31 2018 -0700 Update intellij.md Soften the language a bit. --- src/contribute/intellij.md | 8 +--- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/src/contribute/intellij.md b/src/contribute/intellij.md index 2b4f5bd..e30911e 100644 --- a/src/contribute/intellij.md +++ b/src/contribute/intellij.md @@ -20,13 +20,7 @@ limitations under the License. # IntelliJ Tips -> These are best-effort community-contributed tips, and are not... -> -> - ... guaranteed to work with any particular IntelliJ setup. -> - ... the only or best way to work. -> - ... an endorsement of IntelliJ over alternatives. -> -> Mastering IntelliJ is, ultimately, your responsibility. +> These are best-effort community-contributed tips, and are not guaranteed to work with any particular IntelliJ setup. ## Create a working Gradle module
[jira] [Work logged] (BEAM-4790) Make it easier to select code reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=141523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141523 ] ASF GitHub Bot logged work on BEAM-4790: Author: ASF GitHub Bot Created on: 05/Sep/18 20:33 Start Date: 05/Sep/18 20:33 Worklog Time Spent: 10m Work Description: udim commented on issue #545: [BEAM-4790] Contribute: document how to find and add reviewers URL: https://github.com/apache/beam-site/pull/545#issuecomment-418871778 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141523) Time Spent: 3h 20m (was: 3h 10m) > Make it easier to select code reviewers for PRs > --- > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4790) Make it easier to select code reviewers for PRs
[ https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=141522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141522 ] ASF GitHub Bot logged work on BEAM-4790: Author: ASF GitHub Bot Created on: 05/Sep/18 20:26 Start Date: 05/Sep/18 20:26 Worklog Time Spent: 10m Work Description: melap commented on issue #545: [BEAM-4790] Contribute: document how to find and add reviewers URL: https://github.com/apache/beam-site/pull/545#issuecomment-418869774 @asfgit merge This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141522) Time Spent: 3h 10m (was: 3h) > Make it easier to select code reviewers for PRs > --- > > Key: BEAM-4790 > URL: https://issues.apache.org/jira/browse/BEAM-4790 > Project: Beam > Issue Type: Improvement > Components: project-management >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > The original idea was to use Github's CODEOWNERS feature, but that turned out > to not work well for Apache Beam. > Thread: > https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E > Document discussing this idea: > https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8 > Mailing list thread: > https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141521
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 20:25
Start Date: 05/Sep/18 20:25
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #6222: [BEAM-5149] Add
support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#issuecomment-418869259
@youngoli PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141521)
Time Spent: 1h 40m (was: 1.5h)
> Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-4696) Execute Jenkins website tests in a Docker container
[ https://issues.apache.org/jira/browse/BEAM-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4696. --- Resolution: Fixed Fix Version/s: Not applicable > Execute Jenkins website tests in a Docker container > --- > > Key: BEAM-4696 > URL: https://issues.apache.org/jira/browse/BEAM-4696 > Project: Beam > Issue Type: Improvement > Components: testing, website >Reporter: Scott Wegner >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > > Currently, the website tests run in a vanilla Linux environment, which > require a prerequisite step to install Ruby. The install script is flaky and > adds extra time to the job. > Instead, we should run the website pre-commits inside the pre-built ruby/2.5 > docker image so that we don't need to worry about installing extra > dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-4606) Upper bound for pytz dependency
[ https://issues.apache.org/jira/browse/BEAM-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-4606. --- Resolution: Fixed Fix Version/s: Not applicable > Upper bound for pytz dependency > --- > > Key: BEAM-4606 > URL: https://issues.apache.org/jira/browse/BEAM-4606 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Do we need an upper bound for the pytz dependency? > ([https://github.com/apache/beam/blob/release-2.5.0/sdks/python/setup.py#L108)] > We typically have upper bounds, in order to avoid future breakages due to a > possibility of breaking/backward incompatible change of that depepdency. > Good practice is to upper bound either at known version, or next major > version. Do we need an exception for pytz because it does not seem to be > following semantic versioning? > cc: [~yifanzou] Is this something dependency notifier can warn on? Dependency > without upper version bounds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema
[ https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=141520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141520 ] ASF GitHub Bot logged work on BEAM-4374: Author: ASF GitHub Bot Created on: 05/Sep/18 20:20 Start Date: 05/Sep/18 20:20 Worklog Time Spent: 10m Work Description: pabloem commented on issue #6205: [BEAM-4374] Implementing a subset of the new metrics framework in python. URL: https://github.com/apache/beam/pull/6205#issuecomment-418867827 Lint issue is breaking python precommits ^^ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141520) Time Spent: 6h (was: 5h 50m) > Update existing metrics in the FN API to use new Metric Schema > -- > > Key: BEAM-4374 > URL: https://issues.apache.org/jira/browse/BEAM-4374 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Alex Amato >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Update existing metrics to use the new proto and cataloging schema defined in: > [_https://s.apache.org/beam-fn-api-metrics_] > * Check in new protos > * Define catalog file for metrics > * Port existing metrics to use this new format, based on catalog > names+metadata -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141518
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 20:16
Start Date: 05/Sep/18 20:16
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r215411282
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
+import org.apache.beam.runners.core.construction.BeamUrns;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowFn.MergeContext;
+import org.apache.beam.sdk.values.KV;
+
+/**
+ * Merges windows using a {@link
org.apache.beam.sdk.transforms.windowing.WindowFn}.
+ *
+ * Window merging function:
+ *
+ *
+ * Input: {@code KV>}
+ * Output: {@code KV,
iterable Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141517
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 20:15
Start Date: 05/Sep/18 20:15
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r215410924
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
##
@@ -32,9 +32,17 @@
import org.apache.beam.sdk.values.KV;
/**
- * Maps windows using a window mapping fn. The input is {@link KV} with the
key being a nonce and
- * the value being a window, the output must be a {@link KV} with the key
being the same nonce as
- * the input and the value being the mapped window.
+ * Represents mapping of main input window onto side input window.
+ *
+ * Side input window mapping function:
+ *
+ *
+ * Input: {@code KV}
+ * Output: {@code KV}
+ *
+ *
+ * For each main input window, the side input window is returned. The nonce
is used by a runner
Review comment:
Ditto on copy paste typo
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141517)
Time Spent: 1h 10m (was: 1h)
> Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5107) Support ES 6.x for ElasticsearchIO
[ https://issues.apache.org/jira/browse/BEAM-5107?focusedWorklogId=141513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141513 ] ASF GitHub Bot logged work on BEAM-5107: Author: ASF GitHub Bot Created on: 05/Sep/18 20:06 Start Date: 05/Sep/18 20:06 Worklog Time Spent: 10m Work Description: timrobertson100 commented on issue #6211: [BEAM-5107] Support ES-6.x for ElasticsearchIO URL: https://github.com/apache/beam/pull/6211#issuecomment-418863949 Thank you @dattran-vn01 and congratulations. This is a really great addition. Sorry for my silence, but I have been / am very busy on other things this week and next. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141513) Time Spent: 11.5h (was: 11h 20m) > Support ES 6.x for ElasticsearchIO > -- > > Key: BEAM-5107 > URL: https://issues.apache.org/jira/browse/BEAM-5107 > Project: Beam > Issue Type: Improvement > Components: io-java-elasticsearch >Reporter: Dat Tran >Assignee: Etienne Chauchot >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > > Elasticsearch has released 6.3.2 but ElasticsearchIO only supports 2x-5.x. > We should support ES 6.x for ElasticsearchIO. > https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html > https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5308) JobBundleFactory BindException with FlinkRunner and remote cluster
[ https://issues.apache.org/jira/browse/BEAM-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604840#comment-16604840 ] Ankur Goenka commented on BEAM-5308: I agree, the bug is setting the port range. > JobBundleFactory BindException with FlinkRunner and remote cluster > -- > > Key: BEAM-5308 > URL: https://issues.apache.org/jira/browse/BEAM-5308 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability > Time Spent: 20m > Remaining Estimate: 0h > > Repeated execution of the same job on remote Flink cluster (not embedded in > job server) fails with bind exception. There seem to be 2 issues: > * Multiple instances of job bundle factory cannot be created (port conflict) > * Job bundle factory is not released after job completes (and Docker > container keeps on running). That's not the case in embedded mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5308) JobBundleFactory BindException with FlinkRunner and remote cluster
[ https://issues.apache.org/jira/browse/BEAM-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604836#comment-16604836 ] Thomas Weise edited comment on BEAM-5308 at 9/5/18 7:40 PM: That is an optimization and a different issue. Here we have a bug that multiple job bundle factories cannot coexist in a single JVM (which must be possible). Max has already identified the bug, see linked PR. was (Author: thw): That is an optimization and a different issue. Here we have a bug that multiple harnesses cannot run in a single JVM (which must be possible). Max has already identified the bug, see linked PR. > JobBundleFactory BindException with FlinkRunner and remote cluster > -- > > Key: BEAM-5308 > URL: https://issues.apache.org/jira/browse/BEAM-5308 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability > Time Spent: 20m > Remaining Estimate: 0h > > Repeated execution of the same job on remote Flink cluster (not embedded in > job server) fails with bind exception. There seem to be 2 issues: > * Multiple instances of job bundle factory cannot be created (port conflict) > * Job bundle factory is not released after job completes (and Docker > container keeps on running). That's not the case in embedded mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5308) JobBundleFactory BindException with FlinkRunner and remote cluster
[ https://issues.apache.org/jira/browse/BEAM-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604836#comment-16604836 ] Thomas Weise commented on BEAM-5308: That is an optimization and a different issue. Here we have a bug that multiple harnesses cannot run in a single JVM (which must be possible). Max has already identified the bug, see linked PR. > JobBundleFactory BindException with FlinkRunner and remote cluster > -- > > Key: BEAM-5308 > URL: https://issues.apache.org/jira/browse/BEAM-5308 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability > Time Spent: 20m > Remaining Estimate: 0h > > Repeated execution of the same job on remote Flink cluster (not embedded in > job server) fails with bind exception. There seem to be 2 issues: > * Multiple instances of job bundle factory cannot be created (port conflict) > * Job bundle factory is not released after job completes (and Docker > container keeps on running). That's not the case in embedded mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604833#comment-16604833 ] Ankur Goenka commented on BEAM-5283: The root cause seems to be related to permission. :beam-sdks-python:setupVirtualenv FAILED New python executable in /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle/src/sdks/python/build/gradleenv/bin/python2 Also creating executable in /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle/src/sdks/python/build/gradleenv/bin/python Installing setuptools, pkg_resources, pip, wheel...done. Running virtualenv with interpreter /usr/bin/python2 Collecting tox==3.0.0 Using cached https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl Collecting grpcio-tools==1.3.5 Using cached https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0) Using cached https://files.pythonhosted.org/packages/f5/f1/5a93c118663896d83f7bcbfb7f657ce1d0c0d617e6b4a443a53abcc658ca/pluggy-0.7.1-py2.py3-none-any.whl Requirement not upgraded as not directly required: six in /usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0) Requirement not upgraded as not directly required: virtualenv>=1.11.2 in /usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1) Collecting py>=1.4.17 (from tox==3.0.0) Using cached https://files.pythonhosted.org/packages/c8/47/d179b80ab1dc1bfd46a0c87e391be47e6c7ef5831a9c138c5c49d1756288/py-1.6.0-py2.py3-none-any.whl Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/bd/a6/4bad0d1a49071363dc6547a5178656fe375c80535128c12bb65c59d1a329/grpcio-1.14.2-cp27-cp27mu-manylinux1_x86_64.whl Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/b8/c2/b7f587c0aaf8bf2201405e8162323037fe8d17aa21d3c7dda811b8d01469/protobuf-3.6.1-cp27-cp27mu-manylinux1_x86_64.whl Requirement not upgraded as not directly required: enum34>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6) Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5) Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl Requirement not upgraded as not directly required: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1) Installing collected packages: pluggy, py, tox, futures, grpcio, protobuf, grpcio-tools Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.7.1.dist-info' Consider using the `--user` option or check the permissions. You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. > Enable Python Portable Flink PostCommit Tests to Jenkins > > > Key: BEAM-5283 > URL: https://issues.apache.org/jira/browse/BEAM-5283 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Ankur Goenka >Assignee: Jason Kuster >Priority: Major > Labels: CI > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141502
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 19:05
Start Date: 05/Sep/18 19:05
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r212776929
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -202,9 +202,35 @@ message StandardPTransforms {
// Payload: TestStreamPayload
TEST_STREAM = 5 [(beam_urn) = "urn:beam:transform:teststream:v1"];
-// Represents mapping of main input window into side input window.
-// Payload: serialized WindowMappingFn.
+// Represents mapping of main input window onto side input window.
+//
+// Side input window mapping function:
+// Input: KV
+// Output: KV
+//
+// For each main input window, the side input window is returned. The
Review comment:
Doesn't the Input and Output above imply the opposite of this? This line
makes it sound like the input is a MainInputWindow and the output a
SideInputWindow,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141502)
Time Spent: 50m (was: 40m)
> Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141499
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 19:05
Start Date: 05/Sep/18 19:05
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r212777355
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
##
@@ -32,9 +32,17 @@
import org.apache.beam.sdk.values.KV;
/**
- * Maps windows using a window mapping fn. The input is {@link KV} with the
key being a nonce and
- * the value being a window, the output must be a {@link KV} with the key
being the same nonce as
- * the input and the value being the mapped window.
+ * Represents mapping of main input window onto side input window.
+ *
+ * Side input window mapping function:
+ *
+ *
+ * Input: {@code KV}
+ * Output: {@code KV}
+ *
+ *
+ * For each main input window, the side input window is returned. The nonce
is used by a runner
Review comment:
Same question as earlier comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141499)
Time Spent: 40m (was: 0.5h)
> Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141500=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141500
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 19:05
Start Date: 05/Sep/18 19:05
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r215102198
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
+import org.apache.beam.runners.core.construction.BeamUrns;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowFn.MergeContext;
+import org.apache.beam.sdk.values.KV;
+
+/**
+ * Merges windows using a {@link
org.apache.beam.sdk.transforms.windowing.WindowFn}.
+ *
+ * Window merging function:
+ *
+ *
+ * Input: {@code KV>}
+ * Output: {@code KV,
iterable, ...>` here being used simply
because there's no coder that handles non-keyed pairs? Because it doesn't seem
that the set of unmerged windows is actually used as a key associated with the
map of `MergedWindow` to `ConsumedOriginalWindow`, but instead as just data
paired with the map.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141500)
> Add support to the Java SDK harness to merge windows
>
>
> Key: BEAM-5149
> URL: https://issues.apache.org/jira/browse/BEAM-5149
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Window merging function:
>
> {code:java}
> Input: KV>
> Output: KV,
> iterable>>>
> {code}
> For each set of original windows, a list of all unmerged windows is output
> alongside a map of merged window to set of consumed windows. All original
> windows must be contained in either the unmerged original window set or one
> of the consumed original window sets. Each original window can only be part
> of one output set. The nonce is used by a runner to associate each input with
> its output. The nonce is represented as an opaque set of bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5149) Add support to the Java SDK harness to merge windows
[
https://issues.apache.org/jira/browse/BEAM-5149?focusedWorklogId=141501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141501
]
ASF GitHub Bot logged work on BEAM-5149:
Author: ASF GitHub Bot
Created on: 05/Sep/18 19:05
Start Date: 05/Sep/18 19:05
Worklog Time Spent: 10m
Work Description: youngoli commented on a change in pull request #6222:
[BEAM-5149] Add support for the Java SDK harness to merge windows.
URL: https://github.com/apache/beam/pull/6222#discussion_r215382967
##
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
+import org.apache.beam.runners.core.construction.BeamUrns;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowFn.MergeContext;
+import org.apache.beam.sdk.values.KV;
+
+/**
+ * Merges windows using a {@link
org.apache.beam.sdk.transforms.windowing.WindowFn}.
+ *
+ * Window merging function:
+ *
+ *
+ * Input: {@code KV>}
+ * Output: {@code KV,
iterable>>>}
+ *
+ *
+ * For each set of original windows, a list of all unmerged windows is
output alongside a map of
+ * merged window to set of consumed windows. All original windows must be
contained in either the
+ * unmerged original window set or one of the consumed original window sets.
Each original window
+ * can only be part of one output set. The nonce is used by a runner to
associate each input with
+ * its output. The nonce is represented as an opaque set of bytes.
+ */
+public abstract class WindowMergingFnRunner {
+ static final String URN =
BeamUrns.getUrn(StandardPTransforms.Primitives.MERGE_WINDOWS);
+
+ /**
+ * A registrar which provides a factory to handle merging windows based upon
the {@link WindowFn}.
+ */
+ @AutoService(PTransformRunnerFactory.Registrar.class)
+ public static class Registrar implements PTransformRunnerFactory.Registrar {
+
+@Override
+public Map getPTransformRunnerFactories()
{
+ return ImmutableMap.of(
+ URN,
+
MapFnRunners.forValueMapFnFactory(WindowMergingFnRunner::createMapFunctionForPTransform));
+}
+ }
+
+ static
+ ThrowingFunction>, KV,
Iterable>
+ createMapFunctionForPTransform(String ptransformId, PTransform
ptransform)
+ throws IOException {
+RunnerApi.SdkFunctionSpec payload =
+RunnerApi.SdkFunctionSpec.parseFrom(ptransform.getSpec().getPayload());
+
+WindowFn windowFn =
+(WindowFn)
WindowingStrategyTranslation.windowFnFromProto(payload);
+return WindowMergingFnRunner.create(windowFn)::mergeWindows;
+ }
+
+ static WindowMergingFnRunner
create(WindowFn windowFn) {
+if (windowFn.isNonMerging()) {
+ return new NonMergingWindowFnRunner();
+} else {
+ return new MergingWindowFnRunner(windowFn);
Review comment:
It seems confusing that there's a class named MergingWindowFnRunner that
extends WindowMergingFnRunner. Maybe name it something like
WindowMergingFnRunnerImpl? Or if you want to be descriptive, maybe
MergingViaWindowFnRunner, or WindowMergingViaWindowFnRunner, or some other
variation like that.
This is an automated message from the Apache Git Service.
To respond to
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141488
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 18:35
Start Date: 05/Sep/18 18:35
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6318:
[BEAM-4461] Some fixes to Combiners needed for Schema support.
URL: https://github.com/apache/beam/pull/6318#discussion_r215360597
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
##
@@ -112,6 +113,16 @@ public static ComposeCombineFnBuilder compose() {
return new ComposedCombineFn().with(extractInputFn, combineFn,
outputTag);
}
+/** Like {@link #with(SimpleFunction, CombineFn, TupleTag)} bit with an
explicit input coder. */
Review comment:
typo *but
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141488)
Time Spent: 5h 40m (was: 5.5h)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141489
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 18:35
Start Date: 05/Sep/18 18:35
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6318:
[BEAM-4461] Some fixes to Combiners needed for Schema support.
URL: https://github.com/apache/beam/pull/6318#discussion_r215381299
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Top.java
##
@@ -168,7 +168,40 @@ private Top() {
* {@code PCollection} of {@code KV}s and return the top values associated
with each key.
*/
public static > Combine.Globally>
largest(int count) {
-return Combine.globally(new TopCombineFn<>(count, new Natural()));
+return Combine.globally(largestFn(count));
+ }
+
+ /** Returns a {@link TopCombineFn} that aggregates the largest count values.
*/
+ public static > TopCombineFn>
largestFn(int count) {
+return new TopCombineFn>(count, new Natural()) {};
+ }
+ /** Returns a {@link TopCombineFn} that aggregates the largest count long
values. */
+ public static TopCombineFn> largestLongsFn(int count) {
+return new TopCombineFn>(count, new Natural())
{};
Review comment:
can java infer more of `<>`? E.g. `new TopCombineFn<>(count, new
Natural()) {};`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141489)
Time Spent: 5h 50m (was: 5h 40m)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141490
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 18:35
Start Date: 05/Sep/18 18:35
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6318:
[BEAM-4461] Some fixes to Combiners needed for Schema support.
URL: https://github.com/apache/beam/pull/6318#discussion_r215361562
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
##
@@ -470,7 +589,10 @@ public CoCombineResult extractOutput(Object[]
accumulator, Context c) {
throws CannotProvideCoderException {
List> coders = Lists.newArrayList();
for (int i = 0; i < combineFnCount; ++i) {
-Coder inputCoder =
registry.getOutputCoder(extractInputFns.get(i), dataCoder);
+Coder inputCoder =
+combineInputCoders.get(i).isPresent()
Review comment:
I think something like
`combineInputCoders.get(i).or(registry.getOutputCoder(extractInputFns.get(i),
dataCoder))` should work.
Also, I believe this logic should be changed, it would be clearer if the
coders was not a list of optionals, but a list of coders, and the registry was
consulted before this instead of adding `Optional.absent()`. The end goal is
not to have optionals but use the values from registry instead when no coder is
passed. Similar to how `CombineFnUtil.toFnWithContext(globalCombineFn)` is
called.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141490)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141487=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141487
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 18:35
Start Date: 05/Sep/18 18:35
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6318:
[BEAM-4461] Some fixes to Combiners needed for Schema support.
URL: https://github.com/apache/beam/pull/6318#discussion_r215378279
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
##
@@ -272,6 +301,60 @@ private ComposedCombineFn(
.addAll(extractInputFns)
.add(extractInputFn)
.build(),
+ ImmutableList.>builder()
+ .addAll(combineInputCoders)
+ .add(Optional.absent())
+ .build(),
+ ImmutableList.>builder()
+ .addAll(fnsWithContext)
+ .add(combineFn)
+ .build(),
+
ImmutableList.>builder().addAll(outputTags).add(outputTag).build());
+}
+
+/** Returns a {@link ComposedCombineFn} with an additional {@link
CombineFn}. */
+public ComposedCombineFn with(
+SimpleFunction extractInputFn,
+Coder combineInputCoder,
+CombineFn combineFn,
+TupleTag outputTag) {
+ checkUniqueness(outputTags, outputTag);
+ return new ComposedCombineFn<>(
+ ImmutableList.>builder()
+ .addAll(extractInputFns)
+ .add(extractInputFn)
+ .build(),
+ ImmutableList.>builder()
+ .addAll(combineInputCoders)
+ .add(Optional.of(combineInputCoder))
+ .build(),
+ ImmutableList.>builder().addAll(combineFns).add(combineFn).build(),
+
ImmutableList.>builder().addAll(outputTags).add(outputTag).build());
+}
+
+/**
+ * Returns a {@link ComposedCombineFnWithContext} with an additional {@link
+ * CombineFnWithContext}.
+ */
+public ComposedCombineFnWithContext with(
+SimpleFunction extractInputFn,
+Coder combineInputCoder,
+CombineFnWithContext combineFn,
+TupleTag outputTag) {
+ checkUniqueness(outputTags, outputTag);
+ List> fnsWithContext =
Lists.newArrayList();
+ for (CombineFn fn : combineFns) {
+fnsWithContext.add(CombineFnUtil.toFnWithContext(fn));
Review comment:
nit: can this be rewritten as
`combineFns.stream().map(CombilneFnUtil::toFnWithContext).collect(toList())`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141487)
Time Spent: 5h 40m (was: 5.5h)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5283) Enable Python Portable Flink PostCommit Tests to Jenkins
[
https://issues.apache.org/jira/browse/BEAM-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604761#comment-16604761
]
Thomas Weise commented on BEAM-5283:
The Jenkins job currently fails because the build it is trying to access a
non-existing directory:
{code:java}
20:02:31 file or directory
'/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle/src/runners/flink/job-server/src/main/java',
not found
20:02:31 Skipping task ':beam-runners-flink_2.11-job-server:compileJava' as it
has no source files and no previous output files.
{code}
[https://builds.apache.org/job/beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle/29/console]
[~angoenka] can you please take a look?
> Enable Python Portable Flink PostCommit Tests to Jenkins
>
>
> Key: BEAM-5283
> URL: https://issues.apache.org/jira/browse/BEAM-5283
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Ankur Goenka
>Assignee: Jason Kuster
>Priority: Major
> Labels: CI
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5297) Support for provided configuration in Intellij Idea
[ https://issues.apache.org/jira/browse/BEAM-5297?focusedWorklogId=141464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141464 ] ASF GitHub Bot logged work on BEAM-5297: Author: ASF GitHub Bot Created on: 05/Sep/18 18:03 Start Date: 05/Sep/18 18:03 Worklog Time Spent: 10m Work Description: dmvk edited a comment on issue #6329: [BEAM-5297] Add propdeps-idea plugin. URL: https://github.com/apache/beam/pull/6329#issuecomment-418824832 Hello Ryan, I also see deps listed as `compile` scoped, but they don't get picked up as in your setup. https://user-images.githubusercontent.com/299781/45111783-1b617900-b146-11e8-8ede-4a1558ad9c50.png;> I don't really understand idea internals. The plugin adds deps to PROVIDED scope. https://github.com/spring-gradle-plugins/propdeps-plugin/blob/master/src/main/groovy/io/spring/gradle/propdeps/PropDepsIdeaPlugin.groovy Also, the plugin authors list few relevant issues in the javadoc: http://youtrack.jetbrains.com/issue/IDEA-107046 http://youtrack.jetbrains.com/issue/IDEA-117668 Applying the plugin solves the problem. I hope this will helps. D. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141464) Time Spent: 50m (was: 40m) > Support for provided configuration in Intellij Idea > --- > > Key: BEAM-5297 > URL: https://issues.apache.org/jira/browse/BEAM-5297 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: David Moravek >Assignee: David Moravek >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Intellij Idea (2018.2.1) does not pick up provided dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5297) Support for provided configuration in Intellij Idea
[ https://issues.apache.org/jira/browse/BEAM-5297?focusedWorklogId=141465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141465 ] ASF GitHub Bot logged work on BEAM-5297: Author: ASF GitHub Bot Created on: 05/Sep/18 18:03 Start Date: 05/Sep/18 18:03 Worklog Time Spent: 10m Work Description: dmvk edited a comment on issue #6329: [BEAM-5297] Add propdeps-idea plugin. URL: https://github.com/apache/beam/pull/6329#issuecomment-418824832 Hello Ryan, I also see deps listed as `compile` scoped, but they don't get picked up as in your setup. https://user-images.githubusercontent.com/299781/45111783-1b617900-b146-11e8-8ede-4a1558ad9c50.png;> I don't really understand idea internals. The plugin adds deps to PROVIDED scope. https://github.com/spring-gradle-plugins/propdeps-plugin/blob/master/src/main/groovy/io/spring/gradle/propdeps/PropDepsIdeaPlugin.groovy Also, the plugin authors list few relevant issues in the javadoc: http://youtrack.jetbrains.com/issue/IDEA-107046 http://youtrack.jetbrains.com/issue/IDEA-117668 Applying the plugin solves the problem. I hope this helps. D. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141465) Time Spent: 1h (was: 50m) > Support for provided configuration in Intellij Idea > --- > > Key: BEAM-5297 > URL: https://issues.apache.org/jira/browse/BEAM-5297 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: David Moravek >Assignee: David Moravek >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Intellij Idea (2018.2.1) does not pick up provided dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5297) Support for provided configuration in Intellij Idea
[ https://issues.apache.org/jira/browse/BEAM-5297?focusedWorklogId=141463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141463 ] ASF GitHub Bot logged work on BEAM-5297: Author: ASF GitHub Bot Created on: 05/Sep/18 18:03 Start Date: 05/Sep/18 18:03 Worklog Time Spent: 10m Work Description: dmvk edited a comment on issue #6329: [BEAM-5297] Add propdeps-idea plugin. URL: https://github.com/apache/beam/pull/6329#issuecomment-418824832 Hello Ryan, I also see deps listed as `compile` scoped, but they don't get picked up as in your setup. https://user-images.githubusercontent.com/299781/45111783-1b617900-b146-11e8-8ede-4a1558ad9c50.png;> I don't really understand idea internals. The plugin adds deps to PROVIDED scope. https://github.com/spring-gradle-plugins/propdeps-plugin/blob/master/src/main/groovy/io/spring/gradle/propdeps/PropDepsIdeaPlugin.groovy Also they list some relevant issues in the comment: http://youtrack.jetbrains.com/issue/IDEA-107046 http://youtrack.jetbrains.com/issue/IDEA-117668 Applying the plugin solves the problem. I hope this will helps. D. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141463) Time Spent: 40m (was: 0.5h) > Support for provided configuration in Intellij Idea > --- > > Key: BEAM-5297 > URL: https://issues.apache.org/jira/browse/BEAM-5297 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: David Moravek >Assignee: David Moravek >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Intellij Idea (2018.2.1) does not pick up provided dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5297) Support for provided configuration in Intellij Idea
[ https://issues.apache.org/jira/browse/BEAM-5297?focusedWorklogId=141460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141460 ] ASF GitHub Bot logged work on BEAM-5297: Author: ASF GitHub Bot Created on: 05/Sep/18 18:02 Start Date: 05/Sep/18 18:02 Worklog Time Spent: 10m Work Description: dmvk commented on issue #6329: [BEAM-5297] Add propdeps-idea plugin. URL: https://github.com/apache/beam/pull/6329#issuecomment-418824832 Hello Ryan, I also see files listed as `compile` scoped, but they don't get picked up as in your setup. https://user-images.githubusercontent.com/299781/45111783-1b617900-b146-11e8-8ede-4a1558ad9c50.png;> I don't really understand idea internals, but the plugin seems to be adding them PROVIDED scope. https://github.com/spring-gradle-plugins/propdeps-plugin/blob/master/src/main/groovy/io/spring/gradle/propdeps/PropDepsIdeaPlugin.groovy Also they list some relevant issues in the comment: http://youtrack.jetbrains.com/issue/IDEA-107046 http://youtrack.jetbrains.com/issue/IDEA-117668 Applying the plugin solves the problem. I hope this will helps. D. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141460) Time Spent: 0.5h (was: 20m) > Support for provided configuration in Intellij Idea > --- > > Key: BEAM-5297 > URL: https://issues.apache.org/jira/browse/BEAM-5297 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: David Moravek >Assignee: David Moravek >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Intellij Idea (2018.2.1) does not pick up provided dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4863) Implement consistentWithEquals/structuralValue on FullWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-4863?focusedWorklogId=141450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141450
]
ASF GitHub Bot logged work on BEAM-4863:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:39
Start Date: 05/Sep/18 17:39
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6057:
[BEAM-4863] Implement consistentWithEquals/structuralValue on
FullWindowedValueCoder
URL: https://github.com/apache/beam/pull/6057#discussion_r215363155
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -497,7 +497,7 @@ private void ensureWindowsAreASet() {
// right, and cast the window type away here.
@SuppressWarnings({"unchecked", "rawtypes"})
Coder> collectionCoder =
- (Coder) CollectionCoder.of(this.windowCoder);
+ (Coder) ListCoder.of(this.windowCoder);
Review comment:
Collections aren't required to be ordered (e.g. HashSet). It would be valid
to return any collection which maintains duplicates such as a multiset that
doesn't maintain order. Swapping to a ListCoder ensures that we maintain the
ordering of windows.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141450)
Time Spent: 1h 10m (was: 1h)
> Implement consistentWithEquals/structuralValue on FullWindowedValueCoder
>
>
> Key: BEAM-4863
> URL: https://issues.apache.org/jira/browse/BEAM-4863
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Implementing *consistentWithEquals*/*structuralValue* boosts significantly
> the performance of using these values in comparison operations since it
> doesn't require encoding the values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4863) Implement consistentWithEquals/structuralValue on FullWindowedValueCoder
[
https://issues.apache.org/jira/browse/BEAM-4863?focusedWorklogId=141449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141449
]
ASF GitHub Bot logged work on BEAM-4863:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:33
Start Date: 05/Sep/18 17:33
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6057:
[BEAM-4863] Implement consistentWithEquals/structuralValue on
FullWindowedValueCoder
URL: https://github.com/apache/beam/pull/6057#discussion_r215361516
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
##
@@ -497,7 +497,7 @@ private void ensureWindowsAreASet() {
// right, and cast the window type away here.
@SuppressWarnings({"unchecked", "rawtypes"})
Coder> collectionCoder =
- (Coder) CollectionCoder.of(this.windowCoder);
Review comment:
The encoded/decoded bytes will still be the same since both ListCoder and
CollectionCoder rely on the IterableLikeCoder for encoding/decoding records.
Does Dataflows update care about the names of the coders used or only about
the encoding?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141449)
Time Spent: 1h (was: 50m)
> Implement consistentWithEquals/structuralValue on FullWindowedValueCoder
>
>
> Key: BEAM-4863
> URL: https://issues.apache.org/jira/browse/BEAM-4863
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Implementing *consistentWithEquals*/*structuralValue* boosts significantly
> the performance of using these values in comparison operations since it
> doesn't require encoding the values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141441
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:25
Start Date: 05/Sep/18 17:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6316:
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r215347139
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude:
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name,
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+ public static Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+ }
+ /**
+ * This is the default naming policy for naming fields. Every field name in
the path to a given
+ * field is concated with . characters.
+ */
+ public static final SerializableFunction, String>
CONCAT_FIELD_NAMES =
+ l -> {
+return String.join(".", l);
+ };
+ /**
+ * This policy keeps the raw nested field name. If two differently-nested
fields have the same
+ * name, unnesting will fail with this policy.
+ */
+ public static final SerializableFunction, String>
KEEP_NESTED_NAME =
+ l -> {
+return l.get(l.size() - 1);
+ };
+ /** Returns the result of unnesting the given schema. The default naming
policy is used. */
+ static Schema getUnnestedSchema(Schema schema, int maxLevels) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, CONCAT_FIELD_NAMES,
maxLevels, 0);
+ }
+ /** Returns the result of unnesting the given schema with the given naming
policy. */
+ static Schema getUnnestedSchema(
+ Schema schema, int maxLevels, SerializableFunction, String>
fn) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, fn, maxLevels, 0);
+ }
+
+ private static Schema getUnnestedSchema(
+ Schema schema,
+ List nameComponents,
+ SerializableFunction, String> fn,
+ int maxLevel,
+ int currentLevel) {
+Schema.Builder builder = Schema.builder();
+for (Field field : schema.getFields()) {
+ nameComponents.add(field.getName());
+ if (field.getType().getTypeName().isCompositeType() && currentLevel <
maxLevel) {
Review comment:
throw for arrays/maps?
This is an automated message from the Apache Git Service.
To
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141442=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141442
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:25
Start Date: 05/Sep/18 17:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6316:
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r215354112
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude:
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name,
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+ public static Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+ }
+ /**
+ * This is the default naming policy for naming fields. Every field name in
the path to a given
+ * field is concated with . characters.
+ */
+ public static final SerializableFunction, String>
CONCAT_FIELD_NAMES =
+ l -> {
+return String.join(".", l);
Review comment:
I would probably use some symbol other than `"."` (maybe use `"_"` instead)
to avoid accidental conflicts with field access operator in weird places
downstream (e.g. accessing fields in SQL, or parsing json with these dotted
field names). E.g. you cannot do `JSON.parse("{ \"asd.asd\" : \"123123\"
}").asd.asd`, which probably a some people would expect, and `eval("{ asd.asd :
\"123123\" }")` doesn't work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141442)
Time Spent: 5h 20m (was: 5h 10m)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141443=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141443
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:25
Start Date: 05/Sep/18 17:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6316:
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r215352115
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude:
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name,
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+ public static Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+ }
+ /**
+ * This is the default naming policy for naming fields. Every field name in
the path to a given
+ * field is concated with . characters.
+ */
+ public static final SerializableFunction, String>
CONCAT_FIELD_NAMES =
+ l -> {
+return String.join(".", l);
+ };
+ /**
+ * This policy keeps the raw nested field name. If two differently-nested
fields have the same
+ * name, unnesting will fail with this policy.
+ */
+ public static final SerializableFunction, String>
KEEP_NESTED_NAME =
+ l -> {
+return l.get(l.size() - 1);
+ };
+ /** Returns the result of unnesting the given schema. The default naming
policy is used. */
+ static Schema getUnnestedSchema(Schema schema, int maxLevels) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, CONCAT_FIELD_NAMES,
maxLevels, 0);
+ }
+ /** Returns the result of unnesting the given schema with the given naming
policy. */
+ static Schema getUnnestedSchema(
+ Schema schema, int maxLevels, SerializableFunction, String>
fn) {
+List nameComponents = Lists.newArrayList();
+return getUnnestedSchema(schema, nameComponents, fn, maxLevels, 0);
+ }
+
+ private static Schema getUnnestedSchema(
+ Schema schema,
+ List nameComponents,
+ SerializableFunction, String> fn,
+ int maxLevel,
+ int currentLevel) {
+Schema.Builder builder = Schema.builder();
+for (Field field : schema.getFields()) {
+ nameComponents.add(field.getName());
+ if (field.getType().getTypeName().isCompositeType() && currentLevel <
maxLevel) {
+Schema nestedSchema =
+getUnnestedSchema(
+field.getType().getRowSchema(), nameComponents, fn, maxLevel,
currentLevel + 1);
+for
[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas
[
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=141439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141439
]
ASF GitHub Bot logged work on BEAM-4461:
Author: ASF GitHub Bot
Created on: 05/Sep/18 17:25
Start Date: 05/Sep/18 17:25
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6316:
[BEAM-4461] Add Unnest transform.
URL: https://github.com/apache/beam/pull/6316#discussion_r215348371
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude:
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name,
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */
+@Experimental(Kind.SCHEMAS)
+public class Unnest {
+ public static Inner create() {
+return new AutoValue_Unnest_Inner.Builder()
+.setMaxLevels(Integer.MAX_VALUE)
+.setFieldNameFunction(CONCAT_FIELD_NAMES)
+.build();
+ }
+ /**
+ * This is the default naming policy for naming fields. Every field name in
the path to a given
+ * field is concated with . characters.
+ */
+ public static final SerializableFunction, String>
CONCAT_FIELD_NAMES =
+ l -> {
+return String.join(".", l);
+ };
+ /**
+ * This policy keeps the raw nested field name. If two differently-nested
fields have the same
+ * name, unnesting will fail with this policy.
+ */
+ public static final SerializableFunction, String>
KEEP_NESTED_NAME =
Review comment:
Do we really need to customize naming in the first iteration? I suggest
removing `CONCAT_FIELD_NAMES` and `KEEP_NESTED_NAME`, they have a
[YAGNI](https://martinfowler.com/bliki/Yagni.html) smell. My opinion is that
documentation of how naming behaves should be enough to start with, and then we
could add policies later if needed. Or we could make the two behaviors a flag
for now, e.g. `concatFieldNames()` vs `useLeafFieldName()`, I think it would be
clearer to the users
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 141439)
Time Spent: 5h (was: 4h 50m)
> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>
