[
https://issues.apache.org/jira/browse/BEAM-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329848#comment-16329848
]
Thomas Groh commented on BEAM-2715:
-----------------------------------
Effectively, yes. Reading from a PubSub subscription either:
* Produces duplicates, as no message is ACKed
* Permanently removes the elements from the subscription
As a result, pulling from a real subscription has ill-defined behavior.
The actual change to support this method is relatively minor (but continues to
hide the Pubsub Source). Example diff (admittedly untested) at the following
link:
https://github.com/apache/beam/compare/master...tgroh:pubsub_max_num_records?expand=1
> Expose PubsubSource to create UnboundedSource and utilize withMaxNumRecords
> from BoundedReadFromUnboundedSource
> ---------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-2715
> URL: https://issues.apache.org/jira/browse/BEAM-2715
> Project: Beam
> Issue Type: New Feature
> Components: runner-direct
> Reporter: Adam Levy
> Assignee: Thomas Groh
> Priority: Major
> Labels: pubsub
>
> We are ingesting from a Pubsub Read and are attempting to mimic the
> maxNumRecords that was available in 0.6.0. In order to do this we would need
> to utilize withMaxNumRecords from the BoundedReadFromUnboundedSource class.
> We would need to utilize the PubsubSource class to create the UnboundedSource
> from Pubsub. Would it be possible to expose PubsubSource? Currently what is
> the recommended way to create a bounded read from Pubsub with a set number of
> records?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
