Re: Using Beam Built-in I/O Transforms with an external framework.
Hi Chamikara, Chad Thanks for your replies. @Chamikara I am already working on a Beam runner for Twister2, the runner is functional for the most part even though I have not fully tested it. What I was thinking was to find a way to transform just the sources using the Read primitive that is used in the Twister2 beam runner, since the read primitive transforms between Beam sources and the Twister2 sources. The missing part is a way to do the transform without having to use a beam pipeline. I will look into this a little more to get a better idea. @Chad Like I mentioned we are already working on a runner for Twister2, Since Beam supports a rich set of data sources it would be great to get them to work with Twister2 framework directly in addition to supporting Beam as a runner. Best Regards, Pulasthi On Wed, Sep 18, 2019 at 5:27 PM Chad Dombrova wrote: > Hi Pulasthi, > Just to mirror what Cham said, it would be a non-starter to try to use a > Beam IO source in another framework: to make them work, you'd have to build > something that executes them with their expected protocol, and that would > look an awful lot like a Beam runner. It makes more sense to think about > the problem in reverse: how do you make a transform (or suite of > transforms) that calls out to your Twister2 framework. That's something > that's hard to answer without knowing more about the framework, but > splittable DoFns may be a good place to start. But honestly, if I were you > I'd just invest the time in porting to Beam. We've recently gone through a > similar process and it's well worth the effort. > > -chad > > > > > > > > > On Wed, Sep 18, 2019 at 2:17 PM Chamikara Jayalath > wrote: > >> Hi Pulasthi, >> >> This might be possible but I don't know if anybody has done this. API of >> Beam sources are no different from other Beam PTransforms and we highly >> recommend hiding away various implementations of source framework related >> abstractions in a composite transform [1]. So what you are looking for is a >> way to use a Beam transform in a separate system. >> >> Thanks, >> Cham >> >> [1] >> https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else >> >> On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < >> pulasthi...@gmail.com> wrote: >> >>> Hi Dev's >>> >>> We have a big data processing framework named Twister2, and wanted to >>> know if there is any way we could leverage the I/O Transforms that are >>> built into Apache Beam externally. That is rather than using it in a Beam >>> pipeline just use them as data sources in our project. Just wanted to check >>> with the dev's if such an approach has been done by anyone before, so we >>> could get some pointers on how this could be done. Any pointers in the >>> right direction would be highly appreciated. >>> >>> Best Regards, >>> Pulasthi >>> >>> -- >>> Pulasthi S. Wickramasinghe >>> PhD Candidate | Research Assistant >>> School of Informatics and Computing | Digital Science Center >>> Indiana University, Bloomington >>> cell: 224-386-9035 <(224)%20386-9035> >>> >> -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
Re: Using Beam Built-in I/O Transforms with an external framework.
Hi Pulasthi, Just to mirror what Cham said, it would be a non-starter to try to use a Beam IO source in another framework: to make them work, you'd have to build something that executes them with their expected protocol, and that would look an awful lot like a Beam runner. It makes more sense to think about the problem in reverse: how do you make a transform (or suite of transforms) that calls out to your Twister2 framework. That's something that's hard to answer without knowing more about the framework, but splittable DoFns may be a good place to start. But honestly, if I were you I'd just invest the time in porting to Beam. We've recently gone through a similar process and it's well worth the effort. -chad On Wed, Sep 18, 2019 at 2:17 PM Chamikara Jayalath wrote: > Hi Pulasthi, > > This might be possible but I don't know if anybody has done this. API of > Beam sources are no different from other Beam PTransforms and we highly > recommend hiding away various implementations of source framework related > abstractions in a composite transform [1]. So what you are looking for is a > way to use a Beam transform in a separate system. > > Thanks, > Cham > > [1] > https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else > > On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Dev's >> >> We have a big data processing framework named Twister2, and wanted to >> know if there is any way we could leverage the I/O Transforms that are >> built into Apache Beam externally. That is rather than using it in a Beam >> pipeline just use them as data sources in our project. Just wanted to check >> with the dev's if such an approach has been done by anyone before, so we >> could get some pointers on how this could be done. Any pointers in the >> right direction would be highly appreciated. >> >> Best Regards, >> Pulasthi >> >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> >
Re: Using Beam Built-in I/O Transforms with an external framework.
Hi Pulasthi, This might be possible but I don't know if anybody has done this. API of Beam sources are no different from other Beam PTransforms and we highly recommend hiding away various implementations of source framework related abstractions in a composite transform [1]. So what you are looking for is a way to use a Beam transform in a separate system. Thanks, Cham [1] https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < pulasthi...@gmail.com> wrote: > Hi Dev's > > We have a big data processing framework named Twister2, and wanted to know > if there is any way we could leverage the I/O Transforms that are built > into Apache Beam externally. That is rather than using it in a Beam > pipeline just use them as data sources in our project. Just wanted to check > with the dev's if such an approach has been done by anyone before, so we > could get some pointers on how this could be done. Any pointers in the > right direction would be highly appreciated. > > Best Regards, > Pulasthi > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 <(224)%20386-9035> >
