Not sure if there is any solution other than fixing the Go pubsubio package.
On Mon, 6 Feb 2023 at 17:41, Ashok KS <ashok7...@gmail.com> wrote: > Yes, that is where Iam getting stuck. I wrote the complete pipeline in > Python which reads from the BQ table and published it as a PubSub message. > I'm able to force it as a streaming application by passing --streaming=True > But for my project, they want it in Go so I had to rewrite the complete > logic in Go. > I did the same, but stuck at the last point of publishing it to PubSub. > > On Mon, Feb 6, 2023 at 11:07 PM Shivam Singhal < > shivamsinghal5...@gmail.com> wrote: > >> It depends on the input source: it will decide if your pipeline is a >> streaming or a batch pipeline. >> >> Since you are querying over a BQ table, the input is finite and in >> result, your pipeline is a batch pipeline. >> I am not sure there is a straightforward way where you can convert this >> pipeline into a streaming pipeline. >> >> >> On Mon, 6 Feb 2023 at 17:32, Ashok KS <ashok7...@gmail.com> wrote: >> >>> Hi Shivam, >>> >>> Thanks for that. How can run the pipeline as a streaming pipeline? In >>> python I could just run the pipeline by passing —streaming=True in the >>> command line, but I couldn’t find anything similar in Go. >>> >>> Any pointers would be appreciated. >>> >>> Regards, >>> Ashok >>> >>> On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal < >>> shivamsinghal5...@gmail.com> wrote: >>> >>>> The issue is not yet verified by the maintainers but I think the >>>> pubsubio connector's Write method doesn't work in Batch pipelines. >>>> >>>> But I am pretty sure that pubsubio Write doesn't work for Batch >>>> Pipelines because it's mentioned in the code comments. Check the below >>>> issue for the details: >>>> https://github.com/apache/beam/issues/25326 >>>> >>>> On Mon, 6 Feb 2023 at 17:26, Ashok KS <ashok7...@gmail.com> wrote: >>>> >>>>> Hi Shivam, >>>>> >>>>> Thanks a lot for your response. Yes it is a batch pipeline. My task is >>>>> to read a big query table, process the data and publish the Rows as a >>>>> PubSub message. >>>>> >>>>> Regards, >>>>> Ashok >>>>> >>>>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal < >>>>> shivamsinghal5...@gmail.com> wrote: >>>>> >>>>>> Hey Ashok KS, >>>>>> >>>>>> Is this a batch pipeline? >>>>>> >>>>>> On Mon, 6 Feb 2023 at 09:27, Ashok KS <ashok7...@gmail.com> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Just sending a reminder in case anyone could help. I haven't >>>>>>> received any response to my issue. >>>>>>> >>>>>>> Regards, >>>>>>> Ashok >>>>>>> >>>>>>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS <ashok7...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I'm new to using Apache Beam using Go. >>>>>>>> >>>>>>>> pubsubio.Write(scope, "project", "topic", ppMessages) >>>>>>>> When I try to publish a message in a topic I get an error message >>>>>>>> "Could not find the sink for pubsub, Check that the sink library >>>>>>>> specifies alwayslink = 1 >>>>>>>> >>>>>>>> I found a StackOverFlow post for the same issue but it doesn't >>>>>>>> solve the problem. >>>>>>>> >>>>>>>> Stackoverflow Link >>>>>>>> <https://stackoverflow.com/questions/69651665/go-apache-beam-gcp-dataflow-could-not-find-the-sink-for-pubsub-check-that-th> >>>>>>>> >>>>>>>> Can someone please help? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Ashok >>>>>>>> >>>>>>>
