Serialize the RexNode as Json format is a solution but I’m afraid it can not solve the problem completely. One problem with it is how to re-parse the json format back to RexNode, the current RelJsonReader can only re-parse the RelNode but not RexNode, and it needs the RelOptSchema to lookup the operators.
In the distributed scenarios of Beam, I’m afraid it is hard to get the RelOptSchema because it is execution, we usually see the RelOptSchema during SQL compile time. Best, Danny Chan 在 2020年7月8日 +0800 AM3:39，Roman Kondakov <[email protected]>，写道： > Hi Rui, > > AFAIK, RelNodes can be serialized to and deserialized from JSON format. > See test [1] as an example. If I understand it correct, RelNodes are > serialized along with enclosed RexNodes, so you can transfer them over > the network as plain strings. > > [1] > https://github.com/apache/calcite/blob/f64cdcbb9f6535650f0227da19640e736496a9c3/core/src/test/java/org/apache/calcite/plan/RelWriterTest.java#L88 > > -- > Roman Kondakov > > On 07.07.2020 22:13, Enrico Olivelli wrote: > > Rui > > > > Il Mar 7 Lug 2020, 20:30 Rui Wang <[email protected]> ha scritto: > > > > > Hi Community, > > > > > > In Apache Beam we are facing a use case where we need to keep RexNode in > > > our distributed primitives. Because of the nature of distributed > > > computing, > > > Beam requires the usage of those primitives be serializable (thus those > > > primitives can be sent over the network to backend/workers for > > > further execution). > > > > > > In the Java world this requirement means to make RexNode implement the > > > Java > > > Serializable interface. > > > > > > A workaround right now is to create a bunch of classes to "clone" RexNode > > > while making those classes implement the Serializable interface. > > > > > > > Did you evaluate to use some framework like Kryo that allows you to > > serialize Jon serializable classes? > > > > I think that in general Java serialisation is not efficient as it is too > > general purpose. > > It also brings in a few Security issues. > > > > Maybe an alternative idea is to add some serialisation ad-hoc mechanism in > > RexNode. > > We should also ensure that every RexNode will be able to be serialized and > > deserialized. > > > > Enrico > > > > > > > So what do you think of the idea that makes RexNode implement the > > > Serializable interface? > > > > > > > > > -Rui > > > > >
