[
https://issues.apache.org/jira/browse/BEAM-10962?focusedWorklogId=492407&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492407
]
ASF GitHub Bot logged work on BEAM-10962:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Sep/20 12:02
Start Date: 29/Sep/20 12:02
Worklog Time Spent: 10m
Work Description: InigoSJ commented on pull request #12930:
URL: https://github.com/apache/beam/pull/12930#issuecomment-700656615
> The usual pattern for sources is.
>
> (1) A transform that reads from a given source config
> (2) A "ReadAll" transform that reads a PCollection of configs.
>
> Given that PubSub is a native transform for Dataflow though, we cannot
really implement (2).
>
> I'm not really sure a composite that just wraps a Flatten adds much value
since pipeline authors can easily do that themselves (we can do that for every
other transform as well but that will just clutter the API in my opinion).
>
> (2) above will be more useful and will enable new use-cases. But we cannot
really do that for Dataflow.
I agree with you that adding a similar concept for all IOs would clutter the
API. The main idea here is that this use case is widely shared and a lot of
users are doing it themselves. This PTransform would speedup a lot their code
and it would help organizing the sources better (rather than a wide pipeline
graph, the sources are separated by topic/subs and by project).
Anyhow, I do understand your concern, let me know if I should proceed (by
fixing the errors) or close the PR.
Thanks again!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 492407)
Time Spent: 2h 20m (was: 2h 10m)
> Multiple PubSub Source Reader
> ------------------------------
>
> Key: BEAM-10962
> URL: https://issues.apache.org/jira/browse/BEAM-10962
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
> Reporter: Inigo San Jose Visiers
> Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A very common use case in Dataflow / Beam is reading from multiple PubSub
> topics/subscriptions and flatten them out.
> I would be nice to have a PTransform that does this without users having to
> built it themselves.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
