Serializable classes are not required to have default, no-arg constructors.
Reuven On Fri, Nov 15, 2019 at 11:00 AM Pulasthi Supun Wickramasinghe < [email protected]> wrote: > Hi Luke, > > Aren't the coders supposed to be serializable? The doc on the Coder > interface has the following java doc comment, which seems to mean that they > should be, and most of the basic coders seem to serializable. > > " {@link Coder} instances are serialized during job creation and > deserialized before use. This will generally be performed by serializing > the object via Java Serialization." > > However "TimestampedValueCoder" does not have a default constructor so it > cannot be deserialized. adding a private non-arg constructor would solve > the problem, but this may not be the only class that has this issue. I can > work on a pull request to add the private non-args constructors to coders > that have them missing if this was not done intentionally. WDYT? > > Best Regards, > Pulasthi > > On Fri, Nov 15, 2019 at 12:05 AM Pulasthi Supun Wickramasinghe < > [email protected]> wrote: > >> Hi Luke, >> >> That is the approach i am taking currently to handle the functions. I >> Might have to do the same for Coders as well since some coders have the >> same issue of not having default constructors. >> >> I also initially considered converting the pipeline into a JSON format >> and sending that over to the workers, Will take a look at the option you >> have mentioned since we do plan to implement a Portable pipeline runner for >> Twister2 as well. Thanks for the information >> >> Best Regards, >> Pulasthi >> >> >> On Thu, Nov 14, 2019 at 2:30 PM Luke Cwik <[email protected]> wrote: >> >>> You should create placeholders inside of your Twister2/OpenMPI >>> implementation that represent these functions and then instantiate actual >>> instances of them on the workers if you want to write your own pipeline >>> representation and format for OpenMPI/Twister2. >>> >>> Or consider converting the pipeline to its proto representation and >>> building a portable pipeline runner. This way you could run Go and Python >>> pipelines as well. The best example of this is the current Flink >>> integration[1] >>> >>> 1: >>> https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java >>> >>> On Wed, Nov 13, 2019 at 7:44 PM Pulasthi Supun Wickramasinghe < >>> [email protected]> wrote: >>> >>>> Hi Dev's >>>> >>>> Currently, the Pipeline class in Beam is not Serializable. This is not >>>> a problem for the current runners since the pipeline is translated and >>>> submitted through a centralized Driver like model. However, if the runner >>>> has a decentralized model similar to OpenMPI (MPI), which is also the case >>>> with Twister2, which I am developing a runner currently, it would have been >>>> better if the pipeline itself was Serializable. >>>> >>>> Currently, I am trying to transform the Pipeline into a Twister2 graph >>>> and then send over to the workers, however since there are some functions >>>> such as "SystemReduceFn" that are not serializable this also is somewhat >>>> troublesome. >>>> >>>> Was the decision to make Pipelines not Serializable made due to some >>>> specific reason or because all the current use cases did not present any >>>> valid requirement to make them Serializable? >>>> >>>> Best Regards, >>>> Pulasthi >>>> -- >>>> Pulasthi S. Wickramasinghe >>>> PhD Candidate | Research Assistant >>>> School of Informatics and Computing | Digital Science Center >>>> Indiana University, Bloomington >>>> cell: 224-386-9035 <(224)%20386-9035> >>>> >>> >> >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 <(224)%20386-9035> >> > > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 <(224)%20386-9035> >
