[
https://issues.apache.org/jira/browse/BEAM-10962?focusedWorklogId=492021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492021
]
ASF GitHub Bot logged work on BEAM-10962:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Sep/20 15:15
Start Date: 28/Sep/20 15:15
Worklog Time Spent: 10m
Work Description: InigoSJ commented on pull request #12930:
URL: https://github.com/apache/beam/pull/12930#issuecomment-700072389
I was discussing this PR with Pablo and maybe it would be better to add it
within the `ReadFromPubSub` itself rather than with a different `PTransform`. I
see advantages in both approaches:
**Modifying ReadFromPubSub**
- Just one `PTransfrom` that does both things
- Less documentation needed
**New PTransform**
- Easier to mantain, since `MultipleReadFromPubSub` would only expand from
`ReadFromPubSub`, all modifications from `ReadFromPubSub` would directly be
added.
- Easier on different runners: if I'm not mistaken, Dataflow performs some
overrides to `ReadFromPubSub`, so using `MultipleReadFromPubSub` would not be
affected by this (since it expands it). Considering Dataflow is probably the
main runner for this operation, we should consider this.
- Less overhead: Having `ReadFromPubSub` take both list of
topics/subscriptions and single topics/subscriptions may be a bit too much
It should not be hard to move this PR from a different `PTransform` to
inside `ReadFromPubSub`. So please let me know what do you think about it.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 492021)
Time Spent: 1.5h (was: 1h 20m)
> Multiple PubSub Source Reader
> ------------------------------
>
> Key: BEAM-10962
> URL: https://issues.apache.org/jira/browse/BEAM-10962
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
> Reporter: Inigo San Jose Visiers
> Priority: P2
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> A very common use case in Dataflow / Beam is reading from multiple PubSub
> topics/subscriptions and flatten them out.
> I would be nice to have a PTransform that does this without users having to
> built it themselves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
