[jira] [Comment Edited] (BEAM-9183) apache_beam.options.pipeline_options_test.PipelineOptionsTest.test_value_provider_options is flaky in precommits
[ https://issues.apache.org/jira/browse/BEAM-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023355#comment-17023355 ] Valentyn Tymofieiev edited comment on BEAM-9183 at 1/28/20 5:50 AM: Actually pytest (when executed with xdist) parallelizes the execution within multiple processes (workers), but within one worker tests are running sequentially. The flake may be caused by the order of the test execution. Similar issue: BEAM-9026. was (Author: tvalentyn): Actually pytest (when executed with xdist) parallelizes the execution within multiple processes (workers), but within one worker tests are running sequentially. The may be caused by the order of the test execution. Similar issue: BEAM-9026. > apache_beam.options.pipeline_options_test.PipelineOptionsTest.test_value_provider_options > is flaky in precommits > > > Key: BEAM-9183 > URL: https://issues.apache.org/jira/browse/BEAM-9183 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, test-failures >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > The test appears to fail in -pytest environments on precommits on several PRs: > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1402 > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1397/ > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1405 > I was not able to reproduce it locally via: > - python ./setup.py test -s apache_beam.options.pipeline_options_test > - tox -e py27-cython-pytest > - pytest apache_beam/options/pipeline_options_test.py > Not sure what's happening. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9183) apache_beam.options.pipeline_options_test.PipelineOptionsTest.test_value_provider_options is flaky in precommits
[ https://issues.apache.org/jira/browse/BEAM-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev resolved BEAM-9183. --- Fix Version/s: Not applicable Resolution: Fixed > apache_beam.options.pipeline_options_test.PipelineOptionsTest.test_value_provider_options > is flaky in precommits > > > Key: BEAM-9183 > URL: https://issues.apache.org/jira/browse/BEAM-9183 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, test-failures >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > The test appears to fail in -pytest environments on precommits on several PRs: > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1402 > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1397/ > https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1405 > I was not able to reproduce it locally via: > - python ./setup.py test -s apache_beam.options.pipeline_options_test > - tox -e py27-cython-pytest > - pytest apache_beam/options/pipeline_options_test.py > Not sure what's happening. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8550) @RequiresTimeSortedInput DoFn annotation
[
https://issues.apache.org/jira/browse/BEAM-8550?focusedWorklogId=378046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378046
]
ASF GitHub Bot logged work on BEAM-8550:
Author: ASF GitHub Bot
Created on: 28/Jan/20 05:40
Start Date: 28/Jan/20 05:40
Worklog Time Spent: 10m
Work Description: JozoVilcek commented on issue #8774: [BEAM-8550]
Requires time sorted input
URL: https://github.com/apache/beam/pull/8774#issuecomment-579089156
lgtm
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 378046)
Time Spent: 9h 10m (was: 9h)
> @RequiresTimeSortedInput DoFn annotation
>
>
> Key: BEAM-8550
> URL: https://issues.apache.org/jira/browse/BEAM-8550
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, sdk-java-core
>Reporter: Jan Lukavský
>Assignee: Jan Lukavský
>Priority: Major
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Implement new annotation {{@RequiresTimeSortedInput}} for stateful DoFn as
> described in [design
> document|https://docs.google.com/document/d/1ObLVUFsf1NcG8ZuIZE4aVy2RYKx2FfyMhkZYWPnI9-c/edit?usp=sharing].
> First implementation will assume that:
> - time is defined by timestamp in associated WindowedValue
> - allowed lateness is explicitly zero and all late elements are dropped
> (due to being out of order)
> The above properties are considered temporary and will be resolved by
> subsequent extensions (backwards compatible).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9184) Add ToSet() combiner, similar to ToList() and ToDict()
[ https://issues.apache.org/jira/browse/BEAM-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev reassigned BEAM-9184: - Assignee: Jeffrey Sorensen > Add ToSet() combiner, similar to ToList() and ToDict() > -- > > Key: BEAM-9184 > URL: https://issues.apache.org/jira/browse/BEAM-9184 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Jeffrey Sorensen >Assignee: Jeffrey Sorensen >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > ToList() doesn't do deduplication, and ToDict() requires key/value tuples. > Sets are a different type than dicts in Python, so ToSet() is required to > combine very large PCollections while deduplicating. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9184) Add ToSet() combiner, similar to ToList() and ToDict()
[ https://issues.apache.org/jira/browse/BEAM-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev closed BEAM-9184. - Fix Version/s: 2.20.0 Resolution: Fixed > Add ToSet() combiner, similar to ToList() and ToDict() > -- > > Key: BEAM-9184 > URL: https://issues.apache.org/jira/browse/BEAM-9184 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Jeffrey Sorensen >Assignee: Jeffrey Sorensen >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > ToList() doesn't do deduplication, and ToDict() requires key/value tuples. > Sets are a different type than dicts in Python, so ToSet() is required to > combine very large PCollections while deduplicating. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev reassigned BEAM-9186: - Assignee: Jeffrey Sorensen > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Assignee: Jeffrey Sorensen >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev closed BEAM-9186. - Fix Version/s: Not applicable Resolution: Fixed > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Assignee: Jeffrey Sorensen >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-2970) Add comparator function to equal_to
[ https://issues.apache.org/jira/browse/BEAM-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentyn Tymofieiev resolved BEAM-2970. --- Fix Version/s: Not applicable Resolution: Fixed > Add comparator function to equal_to > --- > > Key: BEAM-2970 > URL: https://issues.apache.org/jira/browse/BEAM-2970 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Sarah Walters >Priority: Minor > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > The equal_to function provided by testing/util.py > (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/util.py#L54) > assumes that the actual and expected lists can be sorted using Python's > sorted method (which relies on the < operator) and compared using the == > operator. > If this isn't the case, equal_to sometimes reports False incorrectly, when > the expected and actual lists are in different orders. > Add a comparator function to equal_to in order to allow callers to define a > total order. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378043=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378043 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 05:24 Start Date: 28/Jan/20 05:24 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637#issuecomment-579085878 Thanks, @sorensenjs ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378043) Time Spent: 1h 10m (was: 1h) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378042 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 05:24 Start Date: 28/Jan/20 05:24 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378042) Time Spent: 1h (was: 50m) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378021 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 02:59 Start Date: 28/Jan/20 02:59 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637#issuecomment-579056939 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378021) Time Spent: 40m (was: 0.5h) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378022 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 02:59 Start Date: 28/Jan/20 02:59 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637#issuecomment-579056983 Run PythonLint Precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378022) Time Spent: 50m (was: 40m) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378020 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 02:51 Start Date: 28/Jan/20 02:51 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637#issuecomment-579055320 Run PythonLint Precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378020) Time Spent: 0.5h (was: 20m) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9186) Allow passing a comparison function for assert_that(..., equal_to()) tests
[ https://issues.apache.org/jira/browse/BEAM-9186?focusedWorklogId=378019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378019 ] ASF GitHub Bot logged work on BEAM-9186: Author: ASF GitHub Bot Created on: 28/Jan/20 02:51 Start Date: 28/Jan/20 02:51 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10637: [BEAM-9186] Allow injection of custom equality function. URL: https://github.com/apache/beam/pull/10637#issuecomment-579055282 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378019) Time Spent: 20m (was: 10m) > Allow passing a comparison function for assert_that(..., equal_to()) tests > -- > > Key: BEAM-9186 > URL: https://issues.apache.org/jira/browse/BEAM-9186 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Currently testing beam pipelines that produce objects that cannot be sorted > or compared is currently problematical, and developers are introducing > Map(str) and similar transforms in tests to transform PCollections into a > testable form. Adding by injection a comparison function will allow the > creation of more comprehensible tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug
[ https://issues.apache.org/jira/browse/BEAM-9203?focusedWorklogId=378012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378012 ] ASF GitHub Bot logged work on BEAM-9203: Author: ASF GitHub Bot Created on: 28/Jan/20 02:36 Start Date: 28/Jan/20 02:36 Worklog Time Spent: 10m Work Description: kennknowles commented on pull request #10699: [BEAM-9203] Clarify exceptions in SQL modules URL: https://github.com/apache/beam/pull/10699 This can make it possible to know if it is user error, bad table metadata, unsupported SQL features, or real bug. - InvalidTableException in table providers to show malformed metadata - UnsupportedOperationException when features incomplete Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Updated] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug
[ https://issues.apache.org/jira/browse/BEAM-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9203: -- Issue Type: Improvement (was: Bug) > Programmatically determine if SQL exception is user error, unsupported, or bug > -- > > Key: BEAM-9203 > URL: https://issues.apache.org/jira/browse/BEAM-9203 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, dsl-sql-zetasql >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Right now there are a lot exceptions thrown by the Calcite SQL dialect and > ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are > user errors, or just the errors that are unsupported operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug
Kenneth Knowles created BEAM-9203: - Summary: Programmatically determine if SQL exception is user error, unsupported, or bug Key: BEAM-9203 URL: https://issues.apache.org/jira/browse/BEAM-9203 Project: Beam Issue Type: Bug Components: dsl-sql, dsl-sql-zetasql Reporter: Kenneth Knowles Assignee: Kenneth Knowles Right now there are a lot exceptions thrown by the Calcite SQL dialect and ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are user errors, or just the errors that are unsupported operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=378009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378009 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 28/Jan/20 02:22 Start Date: 28/Jan/20 02:22 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 378009) Time Spent: 7.5h (was: 7h 20m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7810) Allow ValueProvider arguments to ReadFromDatastore
[ https://issues.apache.org/jira/browse/BEAM-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024819#comment-17024819 ] Udi Meiri commented on BEAM-7810: - Thanks Elias! > Allow ValueProvider arguments to ReadFromDatastore > -- > > Key: BEAM-7810 > URL: https://issues.apache.org/jira/browse/BEAM-7810 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Elias Djurfeldt >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 1h > Remaining Estimate: 0h > > From: > https://stackoverflow.com/questions/56748893/trying-to-achieve-runtime-value-of-namespace-of-datastore-in-dataflow-template -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8970) Spark portable runner supports Yarn
[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024818#comment-17024818
]
Ankur Goenka commented on BEAM-8970:
{code:java}
['--runner=SparkRunner',
'--output_executable_path=~/path/to/output.jar']
{code}
Would be the best way forward.
Once you have the jar, you can use regular jar submission mode as described in
spark documentation here
https://spark.apache.org/docs/latest/running-on-yarn.html
The entry class would be
{code:java}
--class org.apache.beam.runners.spark.SparkPipelineRunner
{code}
> Spark portable runner supports Yarn
> ---
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=378005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378005
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:57
Start Date: 28/Jan/20 01:57
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579043016
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 378005)
Time Spent: 43h (was: 42h 50m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 43h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=378004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378004
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:56
Start Date: 28/Jan/20 01:56
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579042894
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 378004)
Time Spent: 42h 50m (was: 42h 40m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42h 50m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=378001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378001
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:56
Start Date: 28/Jan/20 01:56
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579042827
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 378001)
Time Spent: 42h 40m (was: 42.5h)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42h 40m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377999
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:55
Start Date: 28/Jan/20 01:55
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579042606
Rebased to upstream head to pick up a lint change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377999)
Time Spent: 42h 20m (was: 42h 10m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42h 20m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=378000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378000
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:55
Start Date: 28/Jan/20 01:55
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579042711
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 378000)
Time Spent: 42.5h (was: 42h 20m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42.5h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4735) Make HBaseIO.read() based on SDF
[ https://issues.apache.org/jira/browse/BEAM-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024812#comment-17024812 ] Luke Cwik commented on BEAM-4735: - I noticed there was a bug in the `@SplitRestriction`. The range input parameter is not being used to restrict the splitRanges that are being returned. If multiple rounds of splitting happened, it could be that `@SplitRestriction` is invoked multiple times, once for each split leading to duplication of work. https://github.com/apache/beam/blob/0a37f19e274b9d766f9eee2228460226c81b6b7c/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseReadSplittableDoFn.java#L87 > Make HBaseIO.read() based on SDF > > > Key: BEAM-4735 > URL: https://issues.apache.org/jira/browse/BEAM-4735 > Project: Beam > Issue Type: Improvement > Components: io-java-hbase >Reporter: Ismaël Mejía >Priority: Minor > > BEAM-4020 introduces HBaseIO reads based on SDF. So far the read() method > still uses the Source based API for two reasons: > 1. Most distributed runners don't supports Bounded SDF today. > 2. SDF does not support Dynamic Work Rebalancing but the Source API of HBase > already supports it so changing it means losing some functionality. > Once there is improvements in both (1) and (2) we should consider moving the > main read() function to use the SDF API and remove the Source based > implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377997 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:50 Start Date: 28/Jan/20 01:50 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579041460 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377997) Time Spent: 58h 10m (was: 58h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 58h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377998 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:50 Start Date: 28/Jan/20 01:50 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579041576 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377998) Time Spent: 58h 20m (was: 58h 10m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 58h 20m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9202) lintPy37 precommit broken
[
https://issues.apache.org/jira/browse/BEAM-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay resolved BEAM-9202.
---
Fix Version/s: Not applicable
Resolution: Fixed
> lintPy37 precommit broken
> -
>
> Key: BEAM-9202
> URL: https://issues.apache.org/jira/browse/BEAM-9202
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
> Fix For: Not applicable
>
>
> Culprit: https://github.com/apache/beam/pull/10683
> Jenkins tests are not started automatically.
> {code}
> 09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
> 09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
> 09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too
> long (87/80) (line-too-long)
> {code}
> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9202) lintPy37 precommit broken
[
https://issues.apache.org/jira/browse/BEAM-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024807#comment-17024807
]
Ahmet Altay commented on BEAM-9202:
---
Yes, it is fixed here: [https://github.com/apache/beam/pull/10697]
> lintPy37 precommit broken
> -
>
> Key: BEAM-9202
> URL: https://issues.apache.org/jira/browse/BEAM-9202
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> Culprit: https://github.com/apache/beam/pull/10683
> Jenkins tests are not started automatically.
> {code}
> 09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
> 09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
> 09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too
> long (87/80) (line-too-long)
> {code}
> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=377994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377994 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 28/Jan/20 01:42 Start Date: 28/Jan/20 01:42 Worklog Time Spent: 10m Work Description: angoenka commented on issue #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#issuecomment-579039629 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377994) Time Spent: 7h 20m (was: 7h 10m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9202) lintPy37 precommit broken
[
https://issues.apache.org/jira/browse/BEAM-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024800#comment-17024800
]
Chad Dombrova commented on BEAM-9202:
-
Looks like this issue was already fixed by Ankur
> lintPy37 precommit broken
> -
>
> Key: BEAM-9202
> URL: https://issues.apache.org/jira/browse/BEAM-9202
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> Culprit: https://github.com/apache/beam/pull/10683
> Jenkins tests are not started automatically.
> {code}
> 09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
> 09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
> 09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too
> long (87/80) (line-too-long)
> {code}
> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9202) lintPy37 precommit broken
[
https://issues.apache.org/jira/browse/BEAM-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9202:
---
Assignee: Chad Dombrova
> lintPy37 precommit broken
> -
>
> Key: BEAM-9202
> URL: https://issues.apache.org/jira/browse/BEAM-9202
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> Culprit: https://github.com/apache/beam/pull/10683
> Jenkins tests are not started automatically.
> {code}
> 09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
> 09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
> 09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too
> long (87/80) (line-too-long)
> {code}
> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377993 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:33 Start Date: 28/Jan/20 01:33 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579037876 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377993) Time Spent: 58h (was: 57h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 58h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377992 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:32 Start Date: 28/Jan/20 01:32 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579036819 > lint failure bug: https://issues.apache.org/jira/browse/BEAM-9202 @udim Do you want me to solve that here? If you're okay with that, then please do This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377992) Time Spent: 57h 50m (was: 57h 40m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57h 50m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024797#comment-17024797
]
Udi Meiri commented on BEAM-9119:
-
Another:
https://builds.apache.org/job/beam_PreCommit_Python_Commit/10897/
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value =
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377990 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:29 Start Date: 28/Jan/20 01:29 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579036967 > Also, do you have any insight into the python test failures. I'm having a hard time connecting the dots between my changes and those errors. nevermind, just saw your message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377990) Time Spent: 57h 40m (was: 57.5h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57h 40m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=377989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377989 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 28/Jan/20 01:29 Start Date: 28/Jan/20 01:29 Worklog Time Spent: 10m Work Description: angoenka commented on issue #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#issuecomment-579036933 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377989) Time Spent: 7h 10m (was: 7h) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377980 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:29 Start Date: 28/Jan/20 01:29 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579036819 > lint failure bug: https://issues.apache.org/jira/browse/BEAM-9202 @udim Do you want me to solve that here? Also, do you have any insight into the python test failures. I'm having a hard time connecting the dots between my changes and those errors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377980) Time Spent: 57.5h (was: 57h 20m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57.5h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=377976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377976 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 28/Jan/20 01:28 Start Date: 28/Jan/20 01:28 Worklog Time Spent: 10m Work Description: milantracy commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-579036611 > Thanks for this @milantracy! Right now my only concern with this is that it's duplicating a lot of code from the PubsubJsonTableProvider for supporting both flat and nested schemas. What do you think about adding an abstract `PubsubTableProvider` that includes the logic for the different types of schemas, and then each implementation can be responsible for translating the payload to/from avro or JSON? > > That way when we get rid of support for nested schemas we only have to do it in one place. @TheNeuralBit Thanks, this is a good idea, I will created a abstract class `PubsubTableProvider` and two final class `JsonPubsubTableProvider` and `AvroPubsubTableProvider` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377976) Time Spent: 2h 50m (was: 2h 40m) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377979 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:28 Start Date: 28/Jan/20 01:28 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579036780 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377979) Time Spent: 57h 20m (was: 57h 10m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57h 20m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377978 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:28 Start Date: 28/Jan/20 01:28 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579036756 test_large_elements seems to be flaky, I'll retry precommits https://issues.apache.org/jira/browse/BEAM-9119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377978) Time Spent: 57h 10m (was: 57h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=377974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377974 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 28/Jan/20 01:27 Start Date: 28/Jan/20 01:27 Worklog Time Spent: 10m Work Description: rohdesamuel commented on pull request #10368: [BEAM-8335] Modify PipelineInstrument to add TestStream for unbounded PCollections URL: https://github.com/apache/beam/pull/10368#discussion_r371571266 ## File path: sdks/python/apache_beam/runners/interactive/pipeline_instrument.py ## @@ -418,6 +423,45 @@ def _replace_with_cached_inputs(self, pipeline): cache, noop. """ +# Find all cached unbounded PCollections. +class CacheableUnboundedPCollectionVisitor(PipelineVisitor): + def __init__(self, pin): +self._pin = pin +self.unbounded_pcolls = set() + + def enter_composite_transform(self, transform_node): +self.visit_transform(transform_node) + + def visit_transform(self, transform_node): +if transform_node.inputs: + for input_pcoll in transform_node.inputs: +key = self._pin.cache_key(input_pcoll) +if (key in self._pin._cached_pcoll_read and +not input_pcoll.is_bounded): + self.unbounded_pcolls.add(key) + +v = CacheableUnboundedPCollectionVisitor(self) +pipeline.visit(v) + +# The set of keys from the cached unbounded PCollections will be used as the +# output tags for the TestStream. This is to remember what cache-key is +# associated with which PCollection. +unbounded_cacheables = v.unbounded_pcolls +output_tags = unbounded_cacheables + +# Take the PCollections that will be read from the TestStream and insert +# them back into the dictionary of cached PCollections. The next step will +# replace the downstream consumer of the non-cached PCollections with these +# PCollections. +if output_tags: + output_pcolls = pipeline | test_stream.TestStream(output_tags=output_tags) + if len(output_tags) == 1: +self._cached_pcoll_read[None] = output_pcolls + else: +for tag, pcoll in output_pcolls.items(): + self._cached_pcoll_read[tag] = pcoll Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377974) Time Spent: 54h 10m (was: 54h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 54h 10m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=377975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377975 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 28/Jan/20 01:27 Start Date: 28/Jan/20 01:27 Worklog Time Spent: 10m Work Description: y1chi commented on issue #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#issuecomment-579036476 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377975) Time Spent: 7h (was: 6h 50m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[
https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=377973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377973
]
ASF GitHub Bot logged work on BEAM-8335:
Author: ASF GitHub Bot
Created on: 28/Jan/20 01:26
Start Date: 28/Jan/20 01:26
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #10368:
[BEAM-8335] Modify PipelineInstrument to add TestStream for unbounded
PCollections
URL: https://github.com/apache/beam/pull/10368#discussion_r371571135
##
File path: sdks/python/apache_beam/testing/test_stream.py
##
@@ -172,13 +172,14 @@ class TestStream(PTransform):
output.
"""
- def __init__(self, coder=coders.FastPrimitivesCoder(), events=None):
+ def __init__(self, coder=coders.FastPrimitivesCoder(), events=None,
+ output_tags=None):
super(TestStream, self).__init__()
assert coder is not None
self.coder = coder
self.watermarks = {None: timestamp.MIN_TIMESTAMP}
-self._events = [] if events is None else list(events)
-self.output_tags = set()
+self._events = list(events) if events is not None else []
+self.output_tags = set(output_tags) if output_tags is not None else set()
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377973)
Time Spent: 54h (was: 53h 50m)
> Add streaming support to Interactive Beam
> -
>
> Key: BEAM-8335
> URL: https://issues.apache.org/jira/browse/BEAM-8335
> Project: Beam
> Issue Type: Improvement
> Components: runner-py-interactive
>Reporter: Sam Rohde
>Assignee: Sam Rohde
>Priority: Major
> Time Spent: 54h
> Remaining Estimate: 0h
>
> This issue tracks the work items to introduce streaming support to the
> Interactive Beam experience. This will allow users to:
> * Write and run a streaming job in IPython
> * Automatically cache records from unbounded sources
> * Add a replay experience that replays all cached records to simulate the
> original pipeline execution
> * Add controls to play/pause/stop/step individual elements from the cached
> records
> * Add ability to inspect/visualize unbounded PCollections
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=377971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377971 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 28/Jan/20 01:25 Start Date: 28/Jan/20 01:25 Worklog Time Spent: 10m Work Description: udim commented on issue #10592: [BEAM-7746] Introduce a protocol to handle various types of partitioning buffers URL: https://github.com/apache/beam/pull/10592#issuecomment-579035972 lint failure bug: https://issues.apache.org/jira/browse/BEAM-9202 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377971) Time Spent: 57h (was: 56h 50m) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 57h > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7810) Allow ValueProvider arguments to ReadFromDatastore
[ https://issues.apache.org/jira/browse/BEAM-7810?focusedWorklogId=377970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377970 ] ASF GitHub Bot logged work on BEAM-7810: Author: ASF GitHub Bot Created on: 28/Jan/20 01:25 Start Date: 28/Jan/20 01:25 Worklog Time Spent: 10m Work Description: udim commented on issue #10683: [BEAM-7810] Added ValueProvider support for Datastore query namespaces URL: https://github.com/apache/beam/pull/10683#issuecomment-579035939 Opened https://issues.apache.org/jira/browse/BEAM-9202 for linter failures This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377970) Time Spent: 1h (was: 50m) > Allow ValueProvider arguments to ReadFromDatastore > -- > > Key: BEAM-7810 > URL: https://issues.apache.org/jira/browse/BEAM-7810 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Elias Djurfeldt >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 1h > Remaining Estimate: 0h > > From: > https://stackoverflow.com/questions/56748893/trying-to-achieve-runtime-value-of-namespace-of-datastore-in-dataflow-template -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9202) lintPy37 precommit broken
Udi Meiri created BEAM-9202:
---
Summary: lintPy37 precommit broken
Key: BEAM-9202
URL: https://issues.apache.org/jira/browse/BEAM-9202
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Udi Meiri
Culprit: https://github.com/apache/beam/pull/10683
Jenkins tests are not started automatically.
{code}
09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too long
(87/80) (line-too-long)
{code}
https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9202) lintPy37 precommit broken
[
https://issues.apache.org/jira/browse/BEAM-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-9202:
Status: Open (was: Triage Needed)
> lintPy37 precommit broken
> -
>
> Key: BEAM-9202
> URL: https://issues.apache.org/jira/browse/BEAM-9202
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Priority: Major
>
> Culprit: https://github.com/apache/beam/pull/10683
> Jenkins tests are not started automatically.
> {code}
> 09:47:37 > Task :sdks:python:test-suites:tox:py37:lintPy37
> 09:47:37 * Module apache_beam.io.gcp.datastore.v1new.types
> 09:47:37 apache_beam/io/gcp/datastore/v1new/types.py:47:0: C0301: Line too
> long (87/80) (line-too-long)
> {code}
> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2033/console
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=377968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377968 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 28/Jan/20 01:22 Start Date: 28/Jan/20 01:22 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10612: [DO NOT MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-579035289 :upside_down_face: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377968) Time Spent: 5h 10m (was: 5h) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 5h 10m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=377969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377969 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 28/Jan/20 01:22 Start Date: 28/Jan/20 01:22 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10612: [DO NOT MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-579035289 :upside_down_face: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377969) Time Spent: 5h 20m (was: 5h 10m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 5h 20m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9175) Introduce an autoformatting tool to Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9175?focusedWorklogId=377967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377967 ] ASF GitHub Bot logged work on BEAM-9175: Author: ASF GitHub Bot Created on: 28/Jan/20 01:19 Start Date: 28/Jan/20 01:19 Worklog Time Spent: 10m Work Description: udim commented on issue #10684: [BEAM-9175] Introduce an autoformatting tool to Python SDK URL: https://github.com/apache/beam/pull/10684#issuecomment-579034661 > Why was SPLIT_BEFORE_LOGICAL_OPERATOR disabled? Also, +1 to a precommit. I asked for that: https://github.com/apache/beam/pull/10684/files/1250a914122615df951619361cc7f961f51e60b9..6baa4d4d2d8cf6e14348c55a521e38e9f84b8d21#diff-e710cc10a73c533a8ef446f99dce944e This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377967) Time Spent: 2h 40m (was: 2.5h) > Introduce an autoformatting tool to Python SDK > -- > > Key: BEAM-9175 > URL: https://issues.apache.org/jira/browse/BEAM-9175 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Michał Walenia >Assignee: Kamil Wasilewski >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > It seems there are three main options: > * black - very simple, but not configurable at all (except for line length), > would drastically change code style > * yapf - more options to tweak, can omit parts of code > * autopep8 - more similar to spotless - only touches code that breaks > formatting guidelines, can use pycodestyle and flake8 as configuration > The rigidity of Black makes it unusable for Beam. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=377959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377959 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 28/Jan/20 01:12 Start Date: 28/Jan/20 01:12 Worklog Time Spent: 10m Work Description: vnorigoog commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-579032981 let me first test this change internally and fix it completely - BEFORE I re-upload the changes. please wait on reviewing this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377959) Remaining Estimate: 165h (was: 165h 10m) Time Spent: 3h (was: 2h 50m) > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 3h > Remaining Estimate: 165h > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=377958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377958 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 28/Jan/20 01:10 Start Date: 28/Jan/20 01:10 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-579032599 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377958) Remaining Estimate: 165h 10m (was: 165h 20m) Time Spent: 2h 50m (was: 2h 40m) > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 2h 50m > Remaining Estimate: 165h 10m > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377957 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 01:08 Start Date: 28/Jan/20 01:08 Worklog Time Spent: 10m Work Description: udim commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579032166 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377957) Time Spent: 20h 20m (was: 20h 10m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377956 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 01:07 Start Date: 28/Jan/20 01:07 Worklog Time Spent: 10m Work Description: udim commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579031862 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377956) Time Spent: 20h 10m (was: 20h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 10m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=377951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377951 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 28/Jan/20 01:03 Start Date: 28/Jan/20 01:03 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10612: [DO NOT MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-579030701 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377951) Time Spent: 5h (was: 4h 50m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 5h > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9201) Update release scripts and guide
[ https://issues.apache.org/jira/browse/BEAM-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-9201. - Fix Version/s: Not applicable Resolution: Fixed > Update release scripts and guide > > > Key: BEAM-9201 > URL: https://issues.apache.org/jira/browse/BEAM-9201 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: Not applicable > > Time Spent: 10m > Remaining Estimate: 0h > > New bq flag for project and updated phrases for postcommits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9201) Update release scripts and guide
[ https://issues.apache.org/jira/browse/BEAM-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-9201: Status: Open (was: Triage Needed) > Update release scripts and guide > > > Key: BEAM-9201 > URL: https://issues.apache.org/jira/browse/BEAM-9201 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > > New bq flag for project and updated phrases for postcommits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9201) Update release scripts and guide
Udi Meiri created BEAM-9201: --- Summary: Update release scripts and guide Key: BEAM-9201 URL: https://issues.apache.org/jira/browse/BEAM-9201 Project: Beam Issue Type: Bug Components: build-system Reporter: Udi Meiri Assignee: Udi Meiri New bq flag for project and updated phrases for postcommits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9201) Update release scripts and guide
[ https://issues.apache.org/jira/browse/BEAM-9201?focusedWorklogId=377950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377950 ] ASF GitHub Bot logged work on BEAM-9201: Author: ASF GitHub Bot Created on: 28/Jan/20 01:02 Start Date: 28/Jan/20 01:02 Worklog Time Spent: 10m Work Description: udim commented on pull request #10609: [BEAM-9201] Release scripts fixes: run_rc_validation.sh, verify_release_build.sh URL: https://github.com/apache/beam/pull/10609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377950) Remaining Estimate: 0h Time Spent: 10m > Update release scripts and guide > > > Key: BEAM-9201 > URL: https://issues.apache.org/jira/browse/BEAM-9201 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > New bq flag for project and updated phrases for postcommits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=377944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377944 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 28/Jan/20 00:52 Start Date: 28/Jan/20 00:52 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10612: [DO NOT MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-579028123 @ibzib , can we retrigger the test? I resolved conflicts from master. And about website task, I cannot reproduce the failure locally with head. Let's run it one more time to see if it is still happening. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377944) Time Spent: 4h 50m (was: 4h 40m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 50m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377931 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:25 Start Date: 28/Jan/20 00:25 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579021351 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377931) Time Spent: 20h (was: 19h 50m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[
https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=377933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377933
]
ASF GitHub Bot logged work on BEAM-5504:
Author: ASF GitHub Bot
Created on: 28/Jan/20 00:25
Start Date: 28/Jan/20 00:25
Worklog Time Spent: 10m
Work Description: milantracy commented on pull request #10487:
[BEAM-5504] Introduce PubsubAvroTable
URL: https://github.com/apache/beam/pull/10487#discussion_r371555975
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/AvroPubsubMessageToRow.java
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.joda.time.Instant;
+
+/** Read side converter for {@link PubsubMessage} with Avro payload. */
+@Internal
+@Experimental
+@AutoValue
+public abstract class AvroPubsubMessageToRow extends PubsubMessageToRow
implements Serializable {
+
+ @Override
+ public PCollectionTuple expand(PCollection input) {
+PCollectionTuple rows =
+input.apply(
+ParDo.of(
+useFlatSchema()
+? new FlatSchemaPubsubMessageToRow(messageSchema(),
useDlq())
+: new NestedSchemaPubsubMessageToRow(messageSchema(),
useDlq()))
+.withOutputTags(
+MAIN_TAG, useDlq() ? TupleTagList.of(DLQ_TAG) :
TupleTagList.empty()));
+return rows;
+ }
+
+ public static Builder builder() {
+return new AutoValue_AvroPubsubMessageToRow.Builder();
+ }
+
+ @Internal
+ private static class FlatSchemaPubsubMessageToRow extends
DoFn {
+
+private final Schema messageSchema;
+
+private final boolean useDlq;
+
+protected FlatSchemaPubsubMessageToRow(Schema messageSchema, boolean
useDlq) {
+ this.messageSchema = messageSchema;
+ this.useDlq = useDlq;
+}
+
+private GenericRecord parsePayload(PubsubMessage pubsubMessage) {
+ byte[] avroPayload = pubsubMessage.getPayload();
+
+ // Construct payload flat schema.
+ Schema payloadSchema =
+ new Schema(
+ messageSchema.getFields().stream()
+ .filter(field -> !TIMESTAMP_FIELD.equals(field.getName()))
+ .collect(Collectors.toList()));
+ org.apache.avro.Schema avroSchema =
AvroUtils.toAvroSchema(payloadSchema);
+ return AvroUtils.toGenericRecord(avroPayload, avroSchema);
+}
+
+private Object getValuedForFieldFlatSchema(Field field, Instant timestamp,
Row payload) {
+ String fieldName = field.getName();
+ if (TIMESTAMP_FIELD.equals(fieldName)) {
+return timestamp;
+ } else {
+return payload.getValue(fieldName);
+ }
+}
+
+@ProcessElement
+public void processElement(ProcessContext context) {
+ try {
+GenericRecord record = parsePayload(context.element());
+System.out.println(record);
+Row row = AvroUtils.toBeamRowStrict(record, null);
+List values =
+messageSchema.getFields().stream()
+.map(field -> getValuedForFieldFlatSchema(field,
context.timestamp(), row))
+.collect(Collectors.toList());
+
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[
https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=377932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377932
]
ASF GitHub Bot logged work on BEAM-5504:
Author: ASF GitHub Bot
Created on: 28/Jan/20 00:25
Start Date: 28/Jan/20 00:25
Worklog Time Spent: 10m
Work Description: milantracy commented on pull request #10487:
[BEAM-5504] Introduce PubsubAvroTable
URL: https://github.com/apache/beam/pull/10487#discussion_r371555975
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/AvroPubsubMessageToRow.java
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.joda.time.Instant;
+
+/** Read side converter for {@link PubsubMessage} with Avro payload. */
+@Internal
+@Experimental
+@AutoValue
+public abstract class AvroPubsubMessageToRow extends PubsubMessageToRow
implements Serializable {
+
+ @Override
+ public PCollectionTuple expand(PCollection input) {
+PCollectionTuple rows =
+input.apply(
+ParDo.of(
+useFlatSchema()
+? new FlatSchemaPubsubMessageToRow(messageSchema(),
useDlq())
+: new NestedSchemaPubsubMessageToRow(messageSchema(),
useDlq()))
+.withOutputTags(
+MAIN_TAG, useDlq() ? TupleTagList.of(DLQ_TAG) :
TupleTagList.empty()));
+return rows;
+ }
+
+ public static Builder builder() {
+return new AutoValue_AvroPubsubMessageToRow.Builder();
+ }
+
+ @Internal
+ private static class FlatSchemaPubsubMessageToRow extends
DoFn {
+
+private final Schema messageSchema;
+
+private final boolean useDlq;
+
+protected FlatSchemaPubsubMessageToRow(Schema messageSchema, boolean
useDlq) {
+ this.messageSchema = messageSchema;
+ this.useDlq = useDlq;
+}
+
+private GenericRecord parsePayload(PubsubMessage pubsubMessage) {
+ byte[] avroPayload = pubsubMessage.getPayload();
+
+ // Construct payload flat schema.
+ Schema payloadSchema =
+ new Schema(
+ messageSchema.getFields().stream()
+ .filter(field -> !TIMESTAMP_FIELD.equals(field.getName()))
+ .collect(Collectors.toList()));
+ org.apache.avro.Schema avroSchema =
AvroUtils.toAvroSchema(payloadSchema);
+ return AvroUtils.toGenericRecord(avroPayload, avroSchema);
+}
+
+private Object getValuedForFieldFlatSchema(Field field, Instant timestamp,
Row payload) {
+ String fieldName = field.getName();
+ if (TIMESTAMP_FIELD.equals(fieldName)) {
+return timestamp;
+ } else {
+return payload.getValue(fieldName);
+ }
+}
+
+@ProcessElement
+public void processElement(ProcessContext context) {
+ try {
+GenericRecord record = parsePayload(context.element());
+System.out.println(record);
+Row row = AvroUtils.toBeamRowStrict(record, null);
+List values =
+messageSchema.getFields().stream()
+.map(field -> getValuedForFieldFlatSchema(field,
context.timestamp(), row))
+.collect(Collectors.toList());
+
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377928 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:16 Start Date: 28/Jan/20 00:16 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579018961 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377928) Time Spent: 19h 50m (was: 19h 40m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377927 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:12 Start Date: 28/Jan/20 00:12 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579018096 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377927) Time Spent: 19h 40m (was: 19.5h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 40m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377926 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:12 Start Date: 28/Jan/20 00:12 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579018032 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377926) Time Spent: 19.5h (was: 19h 20m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 19.5h > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377925 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:10 Start Date: 28/Jan/20 00:10 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579017490 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377925) Time Spent: 19h 20m (was: 19h 10m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377924 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 28/Jan/20 00:10 Start Date: 28/Jan/20 00:10 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579017397 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377924) Time Spent: 19h 10m (was: 19h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 10m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377923
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 28/Jan/20 00:08
Start Date: 28/Jan/20 00:08
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-579016898
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377923)
Time Spent: 42h 10m (was: 42h)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42h 10m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9200) Portable job jar postcommits failing
[ https://issues.apache.org/jira/browse/BEAM-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9200: -- Status: Open (was: Triage Needed) > Portable job jar postcommits failing > > > Key: BEAM-9200 > URL: https://issues.apache.org/jira/browse/BEAM-9200 > Project: Beam > Issue Type: Improvement > Components: runner-flink, runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink, portability-spark > > 15:25:58 Execution failed for task > ':runners:spark:job-server:testJavaJarCreatorPy37'. > 15:25:58 > Could not get unknown property 'python_sdk_version' for project > ':runners:spark:job-server' of type org.gradle.api.Project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377921
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 28/Jan/20 00:01
Start Date: 28/Jan/20 00:01
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10051: [BEAM-7961] Add
tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371549724
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesCrossLanguageTransforms;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ConnectivityState;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test External transforms. */
+@RunWith(JUnit4.class)
+public class ValidateRunnerXlangTest implements Serializable {
+ @Rule public transient TestPipeline testPipeline = TestPipeline.create();
+ private PipelineResult pipelineResult;
+
+ private static final String TEST_PREFIX_URN =
"beam:transforms:xlang:test:prefix";
+ private static final String TEST_MULTI_URN =
"beam:transforms:xlang:test:multi";
+ private static final String TEST_GBK_URN = "beam:transforms:xlang:test:gbk";
+ private static final String TEST_CGBK_URN =
"beam:transforms:xlang:test:cgbk";
+ private static final String TEST_COMGL_URN =
"beam:transforms:xlang:test:comgl";
+ private static final String TEST_COMPK_URN =
"beam:transforms:xlang:test:compk";
+ private static final String TEST_FLATTEN_URN =
"beam:transforms:xlang:test:flatten";
+ private static final String TEST_PARTITION_URN =
"beam:transforms:xlang:test:partition";
+
+ private static String expansionAddr;
+ private static String expansionJar;
+
+ @BeforeClass
+ public static void setUpClass() {
+expansionAddr =
+String.format("localhost:%s",
Integer.valueOf(System.getProperty("expansionPort")));
+expansionJar = System.getProperty("expansionJar");
+ }
+
+ @Before
+ public void setUp() {
+testPipeline
+.getOptions()
+.as(ExperimentalOptions.class)
+.setExperiments(ImmutableList.of("jar_packages=" + expansionJar));
+waitForReady();
+ }
+
+ @After
+ public void tearDown() {
+pipelineResult.waitUntilFinish();
+assertThat(pipelineResult.getState(), equalTo(PipelineResult.State.DONE));
+ }
+
+ private void waitForReady() {
+try {
+ ManagedChannel channel =
ManagedChannelBuilder.forTarget(expansionAddr).build();
+ ConnectivityState state
[jira] [Commented] (BEAM-9200) Portable job jar postcommits failing
[ https://issues.apache.org/jira/browse/BEAM-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024776#comment-17024776 ] Kyle Weaver commented on BEAM-9200: --- This should be fixed in https://github.com/apache/beam/pull/10612. > Portable job jar postcommits failing > > > Key: BEAM-9200 > URL: https://issues.apache.org/jira/browse/BEAM-9200 > Project: Beam > Issue Type: Improvement > Components: runner-flink, runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink, portability-spark > > 15:25:58 Execution failed for task > ':runners:spark:job-server:testJavaJarCreatorPy37'. > 15:25:58 > Could not get unknown property 'python_sdk_version' for project > ':runners:spark:job-server' of type org.gradle.api.Project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8970) Spark portable runner supports Yarn
[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024769#comment-17024769
]
Kyle Weaver edited comment on BEAM-8970 at 1/27/20 11:54 PM:
-
Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark
REST API along with YARN, because normally the Spark REST API is started along
with the Spark master.
You should be able to spark-submit portable jars. To create portable jars:
[--runner=SparkRunner,
--output_executable_path=~/path/to/output.jar]
(Without using the spark_submit_uber_jar option.)
Also, note that this will require YARN nodes to have installed or otherwise be
able to access Beam worker code. [~angoenka] might know more.
was (Author: ibzib):
Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark
REST API along with YARN, because normally the Spark REST API is started along
with the Spark master.
You should be able to spark-submit portable jars. To create portable jars:
{{
['--runner=SparkRunner',
--output_executable_path "$OUTPUT_JAR"]
}}
(Without using the spark_submit_uber_jar option.)
Also, note that this will require YARN nodes to have installed or otherwise be
able to access Beam worker code. [~angoenka] might know more.
> Spark portable runner supports Yarn
> ---
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8970) Spark portable runner supports Yarn
[ https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024769#comment-17024769 ] Kyle Weaver edited comment on BEAM-8970 at 1/27/20 11:55 PM: - Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark REST API along with YARN, because normally the Spark REST API is started along with the Spark master. You should be able to spark-submit portable jars. To create portable jars: ['--runner=SparkRunner', '--output_executable_path=~/path/to/output.jar'] (Without using the spark_submit_uber_jar option.) Also, note that this will require YARN nodes to have installed or otherwise be able to access Beam worker code. [~angoenka] might know more. was (Author: ibzib): Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark REST API along with YARN, because normally the Spark REST API is started along with the Spark master. You should be able to spark-submit portable jars. To create portable jars: [--runner=SparkRunner, --output_executable_path=~/path/to/output.jar] (Without using the spark_submit_uber_jar option.) Also, note that this will require YARN nodes to have installed or otherwise be able to access Beam worker code. [~angoenka] might know more. > Spark portable runner supports Yarn > --- > > Key: BEAM-8970 > URL: https://issues.apache.org/jira/browse/BEAM-8970 > Project: Beam > Issue Type: Wish > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8970) Spark portable runner supports Yarn
[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024769#comment-17024769
]
Kyle Weaver commented on BEAM-8970:
---
Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark
REST API along with YARN, because normally the Spark REST API is started along
with the Spark master.
You should be able to spark-submit portable jars. To create portable jars:
{{
['--runner=SparkRunner',
--output_executable_path "$OUTPUT_JAR"]
}}
(Without using the spark_submit_uber_jar option.)
Also, note that this will require YARN nodes to have installed or otherwise be
able to access Beam worker code. [~angoenka] might know more.
> Spark portable runner supports Yarn
> ---
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9200) Portable job jar postcommits failing
Kyle Weaver created BEAM-9200: - Summary: Portable job jar postcommits failing Key: BEAM-9200 URL: https://issues.apache.org/jira/browse/BEAM-9200 Project: Beam Issue Type: Improvement Components: runner-flink, runner-spark Reporter: Kyle Weaver Assignee: Kyle Weaver 15:25:58 Execution failed for task ':runners:spark:job-server:testJavaJarCreatorPy37'. 15:25:58 > Could not get unknown property 'python_sdk_version' for project ':runners:spark:job-server' of type org.gradle.api.Project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377913
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 27/Jan/20 23:28
Start Date: 27/Jan/20 23:28
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10051: [BEAM-7961] Add
tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371540025
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesCrossLanguageTransforms;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ConnectivityState;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test External transforms. */
+@RunWith(JUnit4.class)
+public class ValidateRunnerXlangTest implements Serializable {
+ @Rule public transient TestPipeline testPipeline = TestPipeline.create();
+ private PipelineResult pipelineResult;
+
+ private static final String TEST_PREFIX_URN =
"beam:transforms:xlang:test:prefix";
+ private static final String TEST_MULTI_URN =
"beam:transforms:xlang:test:multi";
+ private static final String TEST_GBK_URN = "beam:transforms:xlang:test:gbk";
+ private static final String TEST_CGBK_URN =
"beam:transforms:xlang:test:cgbk";
+ private static final String TEST_COMGL_URN =
"beam:transforms:xlang:test:comgl";
+ private static final String TEST_COMPK_URN =
"beam:transforms:xlang:test:compk";
+ private static final String TEST_FLATTEN_URN =
"beam:transforms:xlang:test:flatten";
+ private static final String TEST_PARTITION_URN =
"beam:transforms:xlang:test:partition";
+
+ private static String expansionAddr;
+ private static String expansionJar;
+
+ @BeforeClass
+ public static void setUpClass() {
+expansionAddr =
+String.format("localhost:%s",
Integer.valueOf(System.getProperty("expansionPort")));
+expansionJar = System.getProperty("expansionJar");
+ }
+
+ @Before
+ public void setUp() {
+testPipeline
+.getOptions()
+.as(ExperimentalOptions.class)
+.setExperiments(ImmutableList.of("jar_packages=" + expansionJar));
+waitForReady();
+ }
+
+ @After
+ public void tearDown() {
+pipelineResult.waitUntilFinish();
+assertThat(pipelineResult.getState(), equalTo(PipelineResult.State.DONE));
+ }
+
+ private void waitForReady() {
+try {
+ ManagedChannel channel =
ManagedChannelBuilder.forTarget(expansionAddr).build();
+ ConnectivityState state
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377906 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 27/Jan/20 23:22 Start Date: 27/Jan/20 23:22 Worklog Time Spent: 10m Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579003483 @chamikaramj You can start the xlang test with `Run XVR_Flink PostCommit` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377906) Time Spent: 18h 40m (was: 18.5h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 40m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377905 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 27/Jan/20 23:21 Start Date: 27/Jan/20 23:21 Worklog Time Spent: 10m Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-579003483 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377905) Time Spent: 18.5h (was: 18h 20m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 18.5h > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9177) Include Flink container image verification instructions in the release guide
[ https://issues.apache.org/jira/browse/BEAM-9177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024736#comment-17024736 ] Kyle Weaver commented on BEAM-9177: --- Keeping this open. PR #10673 added images to the webpage, but not to the release guide. > Include Flink container image verification instructions in the release guide > > > Key: BEAM-9177 > URL: https://issues.apache.org/jira/browse/BEAM-9177 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9184) Add ToSet() combiner, similar to ToList() and ToDict()
[ https://issues.apache.org/jira/browse/BEAM-9184?focusedWorklogId=377902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377902 ] ASF GitHub Bot logged work on BEAM-9184: Author: ASF GitHub Bot Created on: 27/Jan/20 23:10 Start Date: 27/Jan/20 23:10 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10636: [BEAM-9184] Add ToSet combiner URL: https://github.com/apache/beam/pull/10636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377902) Time Spent: 1h 20m (was: 1h 10m) > Add ToSet() combiner, similar to ToList() and ToDict() > -- > > Key: BEAM-9184 > URL: https://issues.apache.org/jira/browse/BEAM-9184 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Jeffrey Sorensen >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > ToList() doesn't do deduplication, and ToDict() requires key/value tuples. > Sets are a different type than dicts in Python, so ToSet() is required to > combine very large PCollections while deduplicating. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9177) Include Flink container image verification instructions in the release guide
[ https://issues.apache.org/jira/browse/BEAM-9177?focusedWorklogId=377903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377903 ] ASF GitHub Bot logged work on BEAM-9177: Author: ASF GitHub Bot Created on: 27/Jan/20 23:10 Start Date: 27/Jan/20 23:10 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #10673: [DO NOT MERGE][BEAM-9177] Update Flink runner webpage for 2.18 URL: https://github.com/apache/beam/pull/10673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377903) Time Spent: 0.5h (was: 20m) > Include Flink container image verification instructions in the release guide > > > Key: BEAM-9177 > URL: https://issues.apache.org/jira/browse/BEAM-9177 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9199) Make --region a required flag for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9199: -- Status: Open (was: Triage Needed) > Make --region a required flag for DataflowRunner > > > Key: BEAM-9199 > URL: https://issues.apache.org/jira/browse/BEAM-9199 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > We've been warning users since Beam 2.15.0 that --region will be required. > That is sufficient time, so now we can start requiring the flag. > While this is a small change in and of itself, I'm guessing many (maybe even > hundreds of) tests and examples will need to be updated to add --region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9199) Make --region a required flag for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9199: -- Description: We've been warning users since Beam 2.15.0 that --region will be required. That is sufficient time, so now we can start requiring the flag. While this is a small change in and of itself, I'm guessing many (maybe even hundreds of) tests and examples will need to be updated to add --region. was:We've been warning users since Beam 2.15.0 that --region will be required. That is sufficient time, so now we can start requiring the flag. > Make --region a required flag for DataflowRunner > > > Key: BEAM-9199 > URL: https://issues.apache.org/jira/browse/BEAM-9199 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > We've been warning users since Beam 2.15.0 that --region will be required. > That is sufficient time, so now we can start requiring the flag. > While this is a small change in and of itself, I'm guessing many (maybe even > hundreds of) tests and examples will need to be updated to add --region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377899
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 27/Jan/20 23:01
Start Date: 27/Jan/20 23:01
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-578997042
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377899)
Time Spent: 42h (was: 41h 50m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 42h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377898
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 27/Jan/20 23:01
Start Date: 27/Jan/20 23:01
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-578996983
Run PythonLint PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377898)
Time Spent: 41h 50m (was: 41h 40m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 41h 50m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9199) Make --region a required flag for DataflowRunner
Kyle Weaver created BEAM-9199: - Summary: Make --region a required flag for DataflowRunner Key: BEAM-9199 URL: https://issues.apache.org/jira/browse/BEAM-9199 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: Kyle Weaver Assignee: Kyle Weaver We've been warning users since Beam 2.15.0 that --region will be required. That is sufficient time, so now we can start requiring the flag. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377897
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 27/Jan/20 23:01
Start Date: 27/Jan/20 23:01
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-578996904
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377897)
Time Spent: 41h 40m (was: 41.5h)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 41h 40m
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=377896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377896
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 27/Jan/20 23:00
Start Date: 27/Jan/20 23:00
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-578996626
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377896)
Time Spent: 41.5h (was: 41h 20m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 41.5h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8630) Prototype of BeamSQL Calc using ZetaSQL Expression Evaluator
[ https://issues.apache.org/jira/browse/BEAM-8630?focusedWorklogId=377895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377895 ] ASF GitHub Bot logged work on BEAM-8630: Author: ASF GitHub Bot Created on: 27/Jan/20 22:57 Start Date: 27/Jan/20 22:57 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10618: [BEAM-8630] Turn on BeamZetaSqlCalcRel URL: https://github.com/apache/beam/pull/10618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377895) Time Spent: 9h 40m (was: 9.5h) > Prototype of BeamSQL Calc using ZetaSQL Expression Evaluator > > > Key: BEAM-8630 > URL: https://issues.apache.org/jira/browse/BEAM-8630 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377892
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 27/Jan/20 22:54
Start Date: 27/Jan/20 22:54
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10051:
[BEAM-7961] Add tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371527681
##
File path:
sdks/python/apache_beam/runners/portability/expansion_service_test.py
##
@@ -75,39 +88,139 @@ def from_runner_api_parameter(payload, unused_context):
return FilterLessThanTransform(payload.decode('utf8'))
-@ptransform.PTransform.register_urn('simple', None)
-class SimpleTransform(ptransform.PTransform):
+@ptransform.PTransform.register_urn(TEST_PREFIX_URN, None)
+@beam.typehints.with_output_types(unicode)
+class PrefixTransform(ptransform.PTransform):
+ def __init__(self, payload):
+self._payload = payload
+
def expand(self, pcoll):
-return pcoll | 'TestLabel' >> beam.Map(lambda x: 'Simple(%s)' % x)
+return pcoll | 'TestLabel' >> beam.Map(
+lambda x: '{}{}'.format(self._payload, x))
def to_runner_api_parameter(self, unused_context):
-return 'simple', None
+return TEST_PREFIX_URN, None
@staticmethod
- def from_runner_api_parameter(unused_parameter, unused_context):
-return SimpleTransform()
+ def from_runner_api_parameter(payload, unused_context):
+return PrefixTransform(parse_string_payload(payload)['data'])
-@ptransform.PTransform.register_urn('multi', None)
+@ptransform.PTransform.register_urn(TEST_MULTI_URN, None)
class MutltiTransform(ptransform.PTransform):
def expand(self, pcolls):
return {
'main':
(pcolls['main1'], pcolls['main2'])
| beam.Flatten()
| beam.Map(lambda x, s: x + s,
- beam.pvalue.AsSingleton(pcolls['side'])),
-'side': pcolls['side'] | beam.Map(lambda x: x + x),
+ beam.pvalue.AsSingleton(pcolls['side'])
+ ).with_output_types(unicode),
+'side': pcolls['side'] | beam.Map(
+lambda x: x + x).with_output_types(unicode),
}
def to_runner_api_parameter(self, unused_context):
-return 'multi', None
+return TEST_MULTI_URN, None
@staticmethod
def from_runner_api_parameter(unused_parameter, unused_context):
return MutltiTransform()
+@ptransform.PTransform.register_urn(TEST_GBK_URN, None)
+class GBKTransform(ptransform.PTransform):
+ def expand(self, pcoll):
+return pcoll | 'TestLabel' >> beam.GroupByKey()
+
+ def to_runner_api_parameter(self, unused_context):
+return TEST_GBK_URN, None
+
+ @staticmethod
+ def from_runner_api_parameter(unused_parameter, unused_context):
+return GBKTransform()
+
+
+@ptransform.PTransform.register_urn(TEST_CGBK_URN, None)
+class CoGBKTransform(ptransform.PTransform):
+ class ConcatFn(beam.DoFn):
+def process(self, element):
+ (k, v) = element
+ return [(k, v['col1'] + v['col2'])]
+
+ def expand(self, pcoll):
+return pcoll \
+ | beam.CoGroupByKey() \
+ | beam.ParDo(self.ConcatFn()).with_output_types(
+ typing.Tuple[int, typing.Iterable[unicode]])
+
+ def to_runner_api_parameter(self, unused_context):
+return TEST_CGBK_URN, None
+
+ @staticmethod
+ def from_runner_api_parameter(unused_parameter, unused_context):
+return CoGBKTransform()
+
+
+@ptransform.PTransform.register_urn(TEST_COMGL_URN, None)
+class CombineGloballyTransform(ptransform.PTransform):
+ def expand(self, pcoll):
+return pcoll \
Review comment:
Nit: use parenthesis instead of \ for line wrap.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377892)
Time Spent: 18h (was: 17h 50m)
> Add tests for all runner native transforms and some widely used composite
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
> Issue Type: Improvement
> Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 18h
> Remaining Estimate: 0h
>
> Add
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377890
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 27/Jan/20 22:54
Start Date: 27/Jan/20 22:54
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10051:
[BEAM-7961] Add tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371523701
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesCrossLanguageTransforms;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ConnectivityState;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test External transforms. */
+@RunWith(JUnit4.class)
+public class ValidateRunnerXlangTest implements Serializable {
+ @Rule public transient TestPipeline testPipeline = TestPipeline.create();
+ private PipelineResult pipelineResult;
+
+ private static final String TEST_PREFIX_URN =
"beam:transforms:xlang:test:prefix";
+ private static final String TEST_MULTI_URN =
"beam:transforms:xlang:test:multi";
+ private static final String TEST_GBK_URN = "beam:transforms:xlang:test:gbk";
+ private static final String TEST_CGBK_URN =
"beam:transforms:xlang:test:cgbk";
+ private static final String TEST_COMGL_URN =
"beam:transforms:xlang:test:comgl";
+ private static final String TEST_COMPK_URN =
"beam:transforms:xlang:test:compk";
+ private static final String TEST_FLATTEN_URN =
"beam:transforms:xlang:test:flatten";
+ private static final String TEST_PARTITION_URN =
"beam:transforms:xlang:test:partition";
+
+ private static String expansionAddr;
+ private static String expansionJar;
+
+ @BeforeClass
+ public static void setUpClass() {
+expansionAddr =
+String.format("localhost:%s",
Integer.valueOf(System.getProperty("expansionPort")));
+expansionJar = System.getProperty("expansionJar");
+ }
+
+ @Before
+ public void setUp() {
+testPipeline
+.getOptions()
+.as(ExperimentalOptions.class)
+.setExperiments(ImmutableList.of("jar_packages=" + expansionJar));
+waitForReady();
+ }
+
+ @After
+ public void tearDown() {
+pipelineResult.waitUntilFinish();
+assertThat(pipelineResult.getState(), equalTo(PipelineResult.State.DONE));
+ }
+
+ private void waitForReady() {
+try {
+ ManagedChannel channel =
ManagedChannelBuilder.forTarget(expansionAddr).build();
+ ConnectivityState
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=377889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377889 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 27/Jan/20 22:53 Start Date: 27/Jan/20 22:53 Worklog Time Spent: 10m Work Description: vnorigoog commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-578994330 1-1 chat with Chat. To debug this, I am going to run internal tests like Beam team does. my current practice of throwing up a PR without running a lot of tests first is not good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377889) Remaining Estimate: 165h 20m (was: 165.5h) Time Spent: 2h 40m (was: 2.5h) > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 2h 40m > Remaining Estimate: 165h 20m > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377891
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 27/Jan/20 22:54
Start Date: 27/Jan/20 22:54
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10051:
[BEAM-7961] Add tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371523896
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesCrossLanguageTransforms;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ConnectivityState;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test External transforms. */
+@RunWith(JUnit4.class)
+public class ValidateRunnerXlangTest implements Serializable {
+ @Rule public transient TestPipeline testPipeline = TestPipeline.create();
+ private PipelineResult pipelineResult;
+
+ private static final String TEST_PREFIX_URN =
"beam:transforms:xlang:test:prefix";
+ private static final String TEST_MULTI_URN =
"beam:transforms:xlang:test:multi";
+ private static final String TEST_GBK_URN = "beam:transforms:xlang:test:gbk";
+ private static final String TEST_CGBK_URN =
"beam:transforms:xlang:test:cgbk";
+ private static final String TEST_COMGL_URN =
"beam:transforms:xlang:test:comgl";
+ private static final String TEST_COMPK_URN =
"beam:transforms:xlang:test:compk";
+ private static final String TEST_FLATTEN_URN =
"beam:transforms:xlang:test:flatten";
+ private static final String TEST_PARTITION_URN =
"beam:transforms:xlang:test:partition";
+
+ private static String expansionAddr;
+ private static String expansionJar;
+
+ @BeforeClass
+ public static void setUpClass() {
+expansionAddr =
+String.format("localhost:%s",
Integer.valueOf(System.getProperty("expansionPort")));
+expansionJar = System.getProperty("expansionJar");
+ }
+
+ @Before
+ public void setUp() {
+testPipeline
+.getOptions()
+.as(ExperimentalOptions.class)
+.setExperiments(ImmutableList.of("jar_packages=" + expansionJar));
+waitForReady();
+ }
+
+ @After
+ public void tearDown() {
+pipelineResult.waitUntilFinish();
+assertThat(pipelineResult.getState(), equalTo(PipelineResult.State.DONE));
+ }
+
+ private void waitForReady() {
+try {
+ ManagedChannel channel =
ManagedChannelBuilder.forTarget(expansionAddr).build();
+ ConnectivityState
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377893 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 27/Jan/20 22:54 Start Date: 27/Jan/20 22:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-578994684 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377893) Time Spent: 18h 10m (was: 18h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 10m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377894 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 27/Jan/20 22:54 Start Date: 27/Jan/20 22:54 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-578994771 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377894) Time Spent: 18h 20m (was: 18h 10m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=377884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377884
]
ASF GitHub Bot logged work on BEAM-7961:
Author: ASF GitHub Bot
Created on: 27/Jan/20 22:33
Start Date: 27/Jan/20 22:33
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10051:
[BEAM-7961] Add tests for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#discussion_r371520636
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -261,10 +262,12 @@ class BeamModulePlugin implements Plugin {
class CrossLanguageValidatesRunnerConfiguration {
// Task name for cross-language validate runner case.
String name = 'validatesCrossLanguageRunner'
-// Fully qualified JobServerClass name to use.
-String jobServerDriver
-// A string representing the jobServer Configuration.
-String jobServerConfig
+// Job endpoint to use.
+String jobEndpoint = 'localhost:8099'
Review comment:
Do we have to hardcode this port ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 377884)
Time Spent: 17.5h (was: 17h 20m)
> Add tests for all runner native transforms and some widely used composite
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
> Issue Type: Improvement
> Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 17.5h
> Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite
> transforms to cross-language validates runner test suite
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage
[ https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=377883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-377883 ] ASF GitHub Bot logged work on BEAM-8889: Author: ASF GitHub Bot Created on: 27/Jan/20 22:31 Start Date: 27/Jan/20 22:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10617: [BEAM-8889] adding gRPC connectivity to Beam/GCS connector URL: https://github.com/apache/beam/pull/10617#issuecomment-578987179 Failures seems to be related ? https://scans.gradle.com/s/f3f37f5nzigt2/tests/w3mdubixzm3oq-7bnlvldbct3tg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 377883) Remaining Estimate: 165.5h (was: 165h 40m) Time Spent: 2.5h (was: 2h 20m) > Make GcsUtil use GoogleCloudStorage > --- > > Key: BEAM-8889 > URL: https://issues.apache.org/jira/browse/BEAM-8889 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.16.0 >Reporter: Esun Kim >Assignee: VASU NORI >Priority: Major > Labels: gcs > Original Estimate: 168h > Time Spent: 2.5h > Remaining Estimate: 165.5h > > [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java] > is a primary class to access Google Cloud Storage on Apache Beam. Current > implementation directly creates GoogleCloudStorageReadChannel and > GoogleCloudStorageWriteChannel by itself to read and write GCS data rather > than using > [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java] > which is an abstract class providing basic IO capability which eventually > creates channel objects. This request is about updating GcsUtil to use > GoogleCloudStorage to create read and write channel, which is expected > flexible because it can easily pick up the new change; e.g. new channel > implementation using new protocol without code change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
