Hi Pulasthi, Just to mirror what Cham said, it would be a non-starter to try to use a Beam IO source in another framework: to make them work, you'd have to build something that executes them with their expected protocol, and that would look an awful lot like a Beam runner. It makes more sense to think about the problem in reverse: how do you make a transform (or suite of transforms) that calls out to your Twister2 framework. That's something that's hard to answer without knowing more about the framework, but splittable DoFns may be a good place to start. But honestly, if I were you I'd just invest the time in porting to Beam. We've recently gone through a similar process and it's well worth the effort.
-chad On Wed, Sep 18, 2019 at 2:17 PM Chamikara Jayalath <chamik...@google.com> wrote: > Hi Pulasthi, > > This might be possible but I don't know if anybody has done this. API of > Beam sources are no different from other Beam PTransforms and we highly > recommend hiding away various implementations of source framework related > abstractions in a composite transform [1]. So what you are looking for is a > way to use a Beam transform in a separate system. > > Thanks, > Cham > > [1] > https://beam.apache.org/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else > > On Wed, Sep 18, 2019 at 1:52 PM Pulasthi Supun Wickramasinghe < > pulasthi...@gmail.com> wrote: > >> Hi Dev's >> >> We have a big data processing framework named Twister2, and wanted to >> know if there is any way we could leverage the I/O Transforms that are >> built into Apache Beam externally. That is rather than using it in a Beam >> pipeline just use them as data sources in our project. Just wanted to check >> with the dev's if such an approach has been done by anyone before, so we >> could get some pointers on how this could be done. Any pointers in the >> right direction would be highly appreciated. >> >> Best Regards, >> Pulasthi >> >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> >
