[
https://issues.apache.org/jira/browse/BEAM-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Parsons updated BEAM-7851:
----------------------------------
Description:
Hi all,
I have a potential issue I am debugging with
`org.apache.beam.sdk.io.gcp.pubsub`. When using the GCP PubSub service no
messages are received on the DirectRunner or the DataflowRunner when using
PubsubIO.readStrings(). When using the DirectRunner and the Pubsub Emulator it
works correctly locally.
I have been working with this issue for the past few days and believe it may be
related to the PubsubIO library.
I am listening to a topic (also tried a subscription) using:
{code:java}
PubsubIO.Read<String> read = PubsubIO
.readStrings()
.withTimestampAttribute("timestamp")
.fromTopic(options.getTopic());{code}
Alternatively this method works:
{code:java}
PubsubIO.Read<PubsubMessage> read = PubsubIO
.readMessagesWithAttributes()
.withTimestampAttribute("timestamp")
.fromTopic(options.getTopic()); {code}
The code works perfectly with the Pubsub Emulator using readStrings() and after
switching to readMessagesWithAttributes() everything works correctly on both
the emulator as well as GCP.
I am using Beam "2.13.0" with "beam-runners-google-cloud-dataflow-java" with
Java 8.
My "gcloud" versions:
Google Cloud SDK 256.0.0
beta 2019.05.17
bigtable
bq 2.0.46
core 2019.07.26
gsutil 4.41
pubsub-emulator 2019.04.26
was:
Hi all,
I have a potential issue I am debugging with
`org.apache.beam.sdk.io.gcp.pubsub`. When using the GCP PubSub service no
messages are received on the DirectRunner or the DataflowRunner when using
PubsubIO.readStrings(). When using the DirectRunner and the Pubsub Emulator it
works correctly locally.
I have been working with this issue for the past few days and believe it may be
related to the PubsubIO library.
I am listening to a topic (also tried a subscription) using:
{code:java}
PubsubIO.Read<String> read = PubsubIO
.readStrings()
.withTimestampAttribute("timestamp")
.fromTopic(options.getTopic());{code}
Alternatively this method works:
{code:java}
PubsubIO.Read<PubsubMessage> read = PubsubIO
.readMessagesWithAttributes()
.withTimestampAttribute("timestamp")
.fromTopic(options.getTopic());
{code}
The code works perfectly with the Pubsub Emulator using readStrings() and after
switching to readMessagesWithAttributes() everything works correctly on both
the emulator as well as GCP.
I am using Beam "2.13.0" with "beam-runners-google-cloud-dataflow-java" with
Java 8.
My "gcloud" versions:
Google Cloud SDK 256.0.0
beta 2019.05.17
bigtable
bq 2.0.46
core 2019.07.26
gsutil 4.41
pubsub-emulator 2019.04.26
> PubsubIO.readStrings() and DataflowRunner
> -----------------------------------------
>
> Key: BEAM-7851
> URL: https://issues.apache.org/jira/browse/BEAM-7851
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, runner-direct
> Reporter: Gregory Parsons
> Priority: Major
>
> Hi all,
> I have a potential issue I am debugging with
> `org.apache.beam.sdk.io.gcp.pubsub`. When using the GCP PubSub service no
> messages are received on the DirectRunner or the DataflowRunner when using
> PubsubIO.readStrings(). When using the DirectRunner and the Pubsub Emulator
> it works correctly locally.
> I have been working with this issue for the past few days and believe it may
> be related to the PubsubIO library.
>
> I am listening to a topic (also tried a subscription) using:
> {code:java}
> PubsubIO.Read<String> read = PubsubIO
> .readStrings()
> .withTimestampAttribute("timestamp")
> .fromTopic(options.getTopic());{code}
>
> Alternatively this method works:
> {code:java}
> PubsubIO.Read<PubsubMessage> read = PubsubIO
> .readMessagesWithAttributes()
> .withTimestampAttribute("timestamp")
> .fromTopic(options.getTopic()); {code}
> The code works perfectly with the Pubsub Emulator using readStrings() and
> after switching to readMessagesWithAttributes() everything works correctly on
> both the emulator as well as GCP.
>
> I am using Beam "2.13.0" with "beam-runners-google-cloud-dataflow-java" with
> Java 8.
> My "gcloud" versions:
> Google Cloud SDK 256.0.0
> beta 2019.05.17
> bigtable
> bq 2.0.46
> core 2019.07.26
> gsutil 4.41
> pubsub-emulator 2019.04.26
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
