Re: Testing a pipeline with external transform
Sounds good, let me try that thank you again! Thanks, Sahith On Mon, Apr 24, 2023 at 2:11 PM Chamikara Jayalath wrote: > > > On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy > wrote: > >> Hello, >> >> Ah I missed that thank you! I am assuming for java jobs with java >> external transform, we run a Java Job Service? >> > > No, the job service is tied to the runner not to the pipeline SDK. The > Java quickstart guide gives instructions for starting up a job service for > Python portable runner for running Java jobs. > > - Cham > > > >> >> Thanks, >> >> Sahith >> >> On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath >> wrote: >> >>> Have you looked into quickstart guides (Java [1], Python [2]) which also >>> gives instructions for running using DirectRunner ? >>> >>> Thanks, >>> Cham >>> >>> [1] >>> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ >>> [2] >>> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ >>> >>> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev < >>> dev@beam.apache.org> wrote: >>> Hello, I was wondering the best way to test a pipeline with external transform. We were trying to use the DirectRunner but it seemed to hang on the external transform step. I was wondering if someone could point me to somewhere in the beam code where this is done or give some pointers on how to test a pipeline with an External step in it? I tried searching a bit but was unsuccessful, but maybe I was looking in the wrong place. Thanks, Sahith >>>
Re: Testing a pipeline with external transform
On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy wrote: > Hello, > > Ah I missed that thank you! I am assuming for java jobs with java external > transform, we run a Java Job Service? > No, the job service is tied to the runner not to the pipeline SDK. The Java quickstart guide gives instructions for starting up a job service for Python portable runner for running Java jobs. - Cham > > Thanks, > > Sahith > > On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath > wrote: > >> Have you looked into quickstart guides (Java [1], Python [2]) which also >> gives instructions for running using DirectRunner ? >> >> Thanks, >> Cham >> >> [1] >> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ >> [2] >> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ >> >> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev < >> dev@beam.apache.org> wrote: >> >>> Hello, >>> >>> I was wondering the best way to test a pipeline with external transform. >>> We were trying to use the DirectRunner but it seemed to hang on the >>> external transform step. I was wondering if someone could point me to >>> somewhere in the beam code where this is done or give some pointers on how >>> to test a pipeline with an External step in it? I tried searching a bit but >>> was unsuccessful, but maybe I was looking in the wrong place. >>> >>> Thanks, >>> >>> Sahith >>> >>
Re: Testing a pipeline with external transform
Hello, Ah I missed that thank you! I am assuming for java jobs with java external transform, we run a Java Job Service? Thanks, Sahith On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath wrote: > Have you looked into quickstart guides (Java [1], Python [2]) which also > gives instructions for running using DirectRunner ? > > Thanks, > Cham > > [1] > https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ > [2] > https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ > > On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev < > dev@beam.apache.org> wrote: > >> Hello, >> >> I was wondering the best way to test a pipeline with external transform. >> We were trying to use the DirectRunner but it seemed to hang on the >> external transform step. I was wondering if someone could point me to >> somewhere in the beam code where this is done or give some pointers on how >> to test a pipeline with an External step in it? I tried searching a bit but >> was unsuccessful, but maybe I was looking in the wrong place. >> >> Thanks, >> >> Sahith >> >
Re: Testing a pipeline with external transform
Have you looked into quickstart guides (Java [1], Python [2]) which also gives instructions for running using DirectRunner ? Thanks, Cham [1] https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ [2] https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/ On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev < dev@beam.apache.org> wrote: > Hello, > > I was wondering the best way to test a pipeline with external transform. > We were trying to use the DirectRunner but it seemed to hang on the > external transform step. I was wondering if someone could point me to > somewhere in the beam code where this is done or give some pointers on how > to test a pipeline with an External step in it? I tried searching a bit but > was unsuccessful, but maybe I was looking in the wrong place. > > Thanks, > > Sahith >
Testing a pipeline with external transform
Hello, I was wondering the best way to test a pipeline with external transform. We were trying to use the DirectRunner but it seemed to hang on the external transform step. I was wondering if someone could point me to somewhere in the beam code where this is done or give some pointers on how to test a pipeline with an External step in it? I tried searching a bit but was unsuccessful, but maybe I was looking in the wrong place. Thanks, Sahith