I just tried the SINGLE_BEAM option but it didn't work. Will try the Beam consumer now.
Thanks Mono On Fri, Apr 22, 2022 at 8:57 AM monajit choudhury <monoji...@gmail.com> wrote: > Hi Hans, > > Going through the log files I realized it had something to do with > multithreaded executions. I tried using the Beam Kafka Consumer but the > issue is it only supports AVRO. I need to consume json messages > > Thanks > Mono > > On Fri, Apr 22, 2022 at 12:21 AM Hans Van Akelyen < > hans.van.akel...@gmail.com> wrote: > >> Hi Monajit, >> >> This is the auto scaling nature of Flink fighting against the requirement >> of having a single threaded pipeline for Kafka messages (as we need to know >> when messages are finished. When running on Flink the best solution would >> be to use the Beam Kafka Consumer. >> >> Another solution (but not yet tested here so not sure it will work) is to >> force it to a single thread by setting SINGLE_BEAM in the "number of >> copies". >> More information about this can be found on our documentation pages [1] >> >> Kind regards, >> Hans >> >> [1] >> https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html >> >> On Fri, 22 Apr 2022 at 06:50, monajit choudhury <monoji...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I am trying to test a simple kafka consumer using Apache Hop v1.2. When >>> I run the pipeline using the local runner, it works fine. But if I run it >>> using the flink runner I get the following error >>> >>> You can only have one copy of the injector transform 'output' to accept >>> the Kafka messages >>> >>> I have tried debugging the Hop code and looks like the root cause is the >>> initSubPipeline() method being invoked multiple times while using the Flink >>> runner. That's not the case when I use the local runner. Am I missing >>> something here? >>> >>> >>> Thanks >>> >>> Monajit Choudhury >>> >>> Linkedin <https://www.linkedin.com/in/monajit-choudhury-b1409a2/> >>> >>
