[
https://issues.apache.org/jira/browse/BEAM-8503?focusedWorklogId=335755&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335755
]
ASF GitHub Bot logged work on BEAM-8503:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Oct/19 21:08
Start Date: 29/Oct/19 21:08
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #9880:
[BEAM-8503] Improve TestBigQuery and TestPubsub
URL: https://github.com/apache/beam/pull/9880#discussion_r340326416
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
##########
@@ -172,6 +193,65 @@ public void publish(List<PubsubMessage> messages) throws
IOException {
pubsub.publish(eventsTopicPath, outgoingMessages);
}
+ /** Pull up to 100 messages from {@link #subscriptionPath()}. */
+ public List<PubsubMessage> pull() throws IOException {
+ return pull(100);
+ }
+
+ /** Pull up to {@code maxBatchSize} messages from {@link
#subscriptionPath()}. */
+ public List<PubsubMessage> pull(int maxBatchSize) throws IOException {
+ return pubsub.pull(0, subscriptionPath, 100, true).stream()
Review comment:
I don't see anything in the public pubsub API that saves me from writing my
own pulling loop. The closest thing to what I've done here is the [synchronous
pull example](https://cloud.google.com/pubsub/docs/pull#synchronous_pull), but
that just does a single pull, no loop to wait for N messages. It also
acknowledges the messages, which I wasn't doing.. I've added that now.
All I really want is a simple way to wait for the subscription to receive N
messages, and then make an assertion about their content, so I can fluently
test pipelines that write to pubsub. The `pull` methods could be a private
implementation detail, would that be better? Actually the one that defaults to
maxBatchSize = 100 could be removed entirely in that case.
I could also look at switching this from `PubsubClient` to the public
library, it doesn't look like it would be too hard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 335755)
Time Spent: 1h 10m (was: 1h)
> Improve TestBigQuery and TestPubsub
> -----------------------------------
>
> Key: BEAM-8503
> URL: https://issues.apache.org/jira/browse/BEAM-8503
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Add better support for E2E BigQuery and Pubsub testing:
> - TestBigQuery should have the ability to insert data into the underlying
> table before a test.
> - TestPubsub should have the ability to subcribe to the underlying topic and
> read messages that were written during a test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
