Yes, that is where Iam getting stuck. I wrote the complete pipeline in Python which reads from the BQ table and published it as a PubSub message. I'm able to force it as a streaming application by passing --streaming=True But for my project, they want it in Go so I had to rewrite the complete logic in Go. I did the same, but stuck at the last point of publishing it to PubSub.
On Mon, Feb 6, 2023 at 11:07 PM Shivam Singhal <shivamsinghal5...@gmail.com> wrote: > It depends on the input source: it will decide if your pipeline is a > streaming or a batch pipeline. > > Since you are querying over a BQ table, the input is finite and in result, > your pipeline is a batch pipeline. > I am not sure there is a straightforward way where you can convert this > pipeline into a streaming pipeline. > > > On Mon, 6 Feb 2023 at 17:32, Ashok KS <ashok7...@gmail.com> wrote: > >> Hi Shivam, >> >> Thanks for that. How can run the pipeline as a streaming pipeline? In >> python I could just run the pipeline by passing —streaming=True in the >> command line, but I couldn’t find anything similar in Go. >> >> Any pointers would be appreciated. >> >> Regards, >> Ashok >> >> On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal < >> shivamsinghal5...@gmail.com> wrote: >> >>> The issue is not yet verified by the maintainers but I think the >>> pubsubio connector's Write method doesn't work in Batch pipelines. >>> >>> But I am pretty sure that pubsubio Write doesn't work for Batch >>> Pipelines because it's mentioned in the code comments. Check the below >>> issue for the details: >>> https://github.com/apache/beam/issues/25326 >>> >>> On Mon, 6 Feb 2023 at 17:26, Ashok KS <ashok7...@gmail.com> wrote: >>> >>>> Hi Shivam, >>>> >>>> Thanks a lot for your response. Yes it is a batch pipeline. My task is >>>> to read a big query table, process the data and publish the Rows as a >>>> PubSub message. >>>> >>>> Regards, >>>> Ashok >>>> >>>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal < >>>> shivamsinghal5...@gmail.com> wrote: >>>> >>>>> Hey Ashok KS, >>>>> >>>>> Is this a batch pipeline? >>>>> >>>>> On Mon, 6 Feb 2023 at 09:27, Ashok KS <ashok7...@gmail.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> Just sending a reminder in case anyone could help. I haven't received >>>>>> any response to my issue. >>>>>> >>>>>> Regards, >>>>>> Ashok >>>>>> >>>>>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS <ashok7...@gmail.com> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I'm new to using Apache Beam using Go. >>>>>>> >>>>>>> pubsubio.Write(scope, "project", "topic", ppMessages) >>>>>>> When I try to publish a message in a topic I get an error message >>>>>>> "Could not find the sink for pubsub, Check that the sink library >>>>>>> specifies alwayslink = 1 >>>>>>> >>>>>>> I found a StackOverFlow post for the same issue but it doesn't solve >>>>>>> the problem. >>>>>>> >>>>>>> Stackoverflow Link >>>>>>> <https://stackoverflow.com/questions/69651665/go-apache-beam-gcp-dataflow-could-not-find-the-sink-for-pubsub-check-that-th> >>>>>>> >>>>>>> Can someone please help? >>>>>>> >>>>>>> Regards, >>>>>>> Ashok >>>>>>> >>>>>>
