[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383795 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 07/Feb/20 18:58 Start Date: 07/Feb/20 18:58 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #10762: [BEAM-3453] Use project specified in pipeline_options when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383795) Time Spent: 4h 50m (was: 4h 40m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383796 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 07/Feb/20 18:59 Start Date: 07/Feb/20 18:59 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use project specified in pipeline_options when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583555284 Thank you @davidyan74. Thank you @yifanzou for running the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383796) Time Spent: 5h (was: 4h 50m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383759 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 07/Feb/20 18:04 Start Date: 07/Feb/20 18:04 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #10762: [BEAM-3453] Use project specified in pipeline_options when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583531997 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383759) Time Spent: 4h 40m (was: 4.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383313 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 07/Feb/20 01:45 Start Date: 07/Feb/20 01:45 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #10762: [BEAM-3453] Use project specified in pipeline_options when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583195865 > @yifanzou -- we cannot seem to trigger tests on this PR. Is there another way to trigger the tests? Sorry about this issue. I don't have an easy workaround for it. I tried sign into the Jenkins, find the latest Python Precommit triggered by this PR and rebuild with the latest commit hash. The job is waiting for the available worker. But, I am not sure this is the correct way to do. I checked our Jenkins agents and they all look good. The problem could be malfunction of the Jenkins plugin. @markflyhigh has a bug that tracking this problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383313) Time Spent: 4.5h (was: 4h 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383309 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 07/Feb/20 01:32 Start Date: 07/Feb/20 01:32 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #10762: [BEAM-3453] Use project specified in pipeline_options when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583192885 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383309) Time Spent: 4h 20m (was: 4h 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383067 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 19:13 Start Date: 06/Feb/20 19:13 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583063734 @yifanzou -- we cannot seem to trigger tests on this PR. Is there another way to trigger the tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383067) Time Spent: 4h 10m (was: 4h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=383066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-383066 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 19:11 Start Date: 06/Feb/20 19:11 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583063171 comment for tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 383066) Time Spent: 4h (was: 3h 50m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382997 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 17:46 Start Date: 06/Feb/20 17:46 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583025147 Merge conflicts introduced by: https://github.com/apache/beam/pull/10684 Instructions for running the autoformatter: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382997) Time Spent: 3h 50m (was: 3h 40m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382996 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 17:45 Start Date: 06/Feb/20 17:45 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-583025147 Merge conflicts introduced by: https://github.com/apache/beam/pull/10684 Instructions for running the autoformatter: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382996) Time Spent: 3h 40m (was: 3.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382932 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 14:56 Start Date: 06/Feb/20 14:56 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582943489 I looked up last Precommit builds that were requested on this PR, and the last one seems to be 15h ago: https://builds.apache.org/job/beam_PreCommit_Python_Commit/11013 , and it ran against f78dd7d507cf3e6ffb60698a40b1f2b657665a61, but did not run against the last commit on this PR. So I think sometimes Jenkins trigger commands don't trigger anything... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382932) Time Spent: 3.5h (was: 3h 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382925 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 14:46 Start Date: 06/Feb/20 14:46 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582713735 The tests are triggered but test result does not appear in the UI - I looked up the test logs on Jenkins builds for targets I ran... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382925) Time Spent: 3h 20m (was: 3h 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382856 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 12:21 Start Date: 06/Feb/20 12:21 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582881034 Just to let you know that we've just introduced Python autoformatter. Your merge conflict might be a result of this. Here you can find an instruction on how to run autoformatter: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips, section Formatting. Sorry for inconvenience. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382856) Time Spent: 3h 10m (was: 3h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382698 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 03:00 Start Date: 06/Feb/20 03:00 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582713735 The test are triggered but test result does not appear in the UI - I looked up the test logs on Jenkins builds for targets I ran... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382698) Time Spent: 3h (was: 2h 50m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382689 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:13 Start Date: 06/Feb/20 02:13 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582703495 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382689) Time Spent: 2h 50m (was: 2h 40m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382686 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:12 Start Date: 06/Feb/20 02:12 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582703132 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382686) Time Spent: 2.5h (was: 2h 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382687 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:12 Start Date: 06/Feb/20 02:12 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582703189 Run PythonLint PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382687) Time Spent: 2h 40m (was: 2.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382685=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382685 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:11 Start Date: 06/Feb/20 02:11 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582702982 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382685) Time Spent: 2h 20m (was: 2h 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382684 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:10 Start Date: 06/Feb/20 02:10 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582702732 My last comment did not trigger tests. (@tvalentyn - how did you resolved it last time?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382684) Time Spent: 2h 10m (was: 2h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382683 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:10 Start Date: 06/Feb/20 02:10 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582702541 ping to trigger tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382683) Time Spent: 2h (was: 1h 50m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382681 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:09 Start Date: 06/Feb/20 02:09 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582702417 ping to trigger tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382681) Time Spent: 1h 40m (was: 1.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382682 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 02:09 Start Date: 06/Feb/20 02:09 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582702541 ping to trigger tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382682) Time Spent: 1h 50m (was: 1h 40m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382645 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 01:02 Start Date: 06/Feb/20 01:02 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-582686497 LGTM, thank you. @angoenka do you have additional comments? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382645) Time Spent: 1.5h (was: 1h 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382616=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382616 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 06/Feb/20 00:11 Start Date: 06/Feb/20 00:11 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r375579010 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: According to our offline discussion, I've removed the code that gets the project from google.auth and we now only rely on the project specified in pipeline_options. PTAL again. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382616) Time Spent: 1h 20m (was: 1h 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382063 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 05/Feb/20 04:46 Start Date: 05/Feb/20 04:46 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r375057192 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: I _think_ changing that would be a backward incompatible change and might break users. We can keep the existing behavior in place, and add a warning along the lines of this is wrong/deprecated/would change in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 382063) Time Spent: 1h 10m (was: 1h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381985 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 05/Feb/20 01:09 Start Date: 05/Feb/20 01:09 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r375011076 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -499,13 +499,21 @@ def __init__(self, evaluation_context, applied_ptransform, if self.source.id_label: raise NotImplementedError( 'DirectRunner: id_label is not supported for PubSub reads') + +import google.auth + +_, sub_project = google.auth.default() Review comment: Done. I'm defaulting to the default project from auth if the project is not specified. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381985) Time Spent: 1h (was: 50m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381969 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 05/Feb/20 00:40 Start Date: 05/Feb/20 00:40 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r375003261 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: Falling back to topic's project may make things less predictable, and failure to create the subscription could be transient and could be because of many reasons. The user should be sure which project it will create the subscription under. My take is that using the topic's project was a mistake from the beginning and it should be fixed. But let me know if you have other thoughts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381969) Time Spent: 50m (was: 40m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381348 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 03:03 Start Date: 04/Feb/20 03:03 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374450518 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: We can fall back to topic's project if this fails. This also maintains the current functionality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381348) Time Spent: 40m (was: 0.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381345 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:50 Start Date: 04/Feb/20 02:50 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374450518 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: We can fall back to topic's project if this fails. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381345) Time Spent: 0.5h (was: 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381344 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:50 Start Date: 04/Feb/20 02:50 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374449591 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -499,13 +499,21 @@ def __init__(self, evaluation_context, applied_ptransform, if self.source.id_label: raise NotImplementedError( 'DirectRunner: id_label is not supported for PubSub reads') + +import google.auth + +_, sub_project = google.auth.default() Review comment: Beam has this pipeline option, project which is generally used to specify project explicitly. Asking user explicitly for resource avoid surprises. Shall we use it. https://github.com/apache/beam/blob/40c2be03303d7c37cabfae4d4da6258a2ce33579/sdks/python/apache_beam/options/pipeline_options.py#L473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381344) Time Spent: 0.5h (was: 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381336 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:28 Start Date: 04/Feb/20 02:28 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-581712749 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381336) Time Spent: 20m (was: 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381334 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:24 Start Date: 04/Feb/20 02:24 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762 When creating a pubsub subscription, use the default project in GCP if available, rather than the project specified in the pubsub topic. R: @aaltay Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [