Hi Luke, That is the approach i am taking currently to handle the functions. I Might have to do the same for Coders as well since some coders have the same issue of not having default constructors.
I also initially considered converting the pipeline into a JSON format and sending that over to the workers, Will take a look at the option you have mentioned since we do plan to implement a Portable pipeline runner for Twister2 as well. Thanks for the information Best Regards, Pulasthi On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik <[email protected]> wrote: > You should create placeholders inside of your Twister2/OpenMPI > implementation that represent these functions and then instantiate actual > instances of them on the workers if you want to write your own pipeline > representation and format for OpenMPI/Twister2. > > Or consider converting the pipeline to its proto representation and > building a portable pipeline runner. This way you could run Go and Python > pipelines as well. The best example of this is the current Flink > integration[1] > > 1: > https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java > > On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe < > [email protected]> wrote: > >> Hi Dev's >> >> Currently, the Pipeline class in Beam is not Serializable. This is not a >> problem for the current runners since the pipeline is translated and >> submitted through a centralized Driver like model. However, if the runner >> has a decentralized model similar to OpenMPI (MPI), which is also the case >> with Twister2, which I am developing a runner currently, it would have been >> better if the pipeline itself was Serializable. >> >> Currently, I am trying to transform the Pipeline into a Twister2 graph >> and then send over to the workers, however since there are some functions >> such as "SystemReduceFn" that are not serializable this also is somewhat >> troublesome. >> >> Was the decision to make Pipelines not Serializable made due to some >> specific reason or because all the current use cases did not present any >> valid requirement to make them Serializable? >> >> Best Regards, >> Pulasthi >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> > -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
