Thanks everyone for your inputs. Now it sounds like RexNode serialization is not an easy effort (e.g. as easy as making classes implement Serializable). I will log a JIRA to document people's opinions.
Currently I am leaning to add serialize()/deserialize() methods to RexNode, and also see if we can improve RelToJson/JsonToRel to include RexNode serialization/deserialization (which gives JSON, thus String). -Rui On Wed, Jul 8, 2020 at 1:42 PM Julian Hyde <[email protected]> wrote: > Serializabilty is not very popular in Java right now. There are a > bunch of concerns, including security. Serializable classes are very > brittle, because it's very easy to add a non-serializable field value > in a sub-class. > > I strongly favor externalizing over serialization. Convert RexNode to > a serializable type (e.g. java.lang.String!), and then convert it back > on the other side. > > Julian > > > On Tue, Jul 7, 2020 at 7:25 PM Danny Chan <[email protected]> wrote: > > > > Serialize the RexNode as Json format is a solution but I’m afraid it can > not solve the problem completely. > > One problem with it is how to re-parse the json format back to RexNode, > the current RelJsonReader can only re-parse the RelNode but not RexNode, > and it needs the RelOptSchema to lookup the operators. > > > > In the distributed scenarios of Beam, I’m afraid it is hard to get the > RelOptSchema because it is execution, we usually see the RelOptSchema > during SQL compile time. > > > > Best, > > Danny Chan > > 在 2020年7月8日 +0800 AM3:39，Roman Kondakov <[email protected]>，写道： > > > Hi Rui, > > > > > > AFAIK, RelNodes can be serialized to and deserialized from JSON format. > > > See test [1] as an example. If I understand it correct, RelNodes are > > > serialized along with enclosed RexNodes, so you can transfer them over > > > the network as plain strings. > > > > > > [1] > > > > https://github.com/apache/calcite/blob/f64cdcbb9f6535650f0227da19640e736496a9c3/core/src/test/java/org/apache/calcite/plan/RelWriterTest.java#L88 > > > > > > -- > > > Roman Kondakov > > > > > > On 07.07.2020 22:13, Enrico Olivelli wrote: > > > > Rui > > > > > > > > Il Mar 7 Lug 2020, 20:30 Rui Wang <[email protected]> ha scritto: > > > > > > > > > Hi Community, > > > > > > > > > > In Apache Beam we are facing a use case where we need to keep > RexNode in > > > > > our distributed primitives. Because of the nature of distributed > computing, > > > > > Beam requires the usage of those primitives be serializable (thus > those > > > > > primitives can be sent over the network to backend/workers for > > > > > further execution). > > > > > > > > > > In the Java world this requirement means to make RexNode implement > the Java > > > > > Serializable interface. > > > > > > > > > > A workaround right now is to create a bunch of classes to "clone" > RexNode > > > > > while making those classes implement the Serializable interface. > > > > > > > > > > > > > Did you evaluate to use some framework like Kryo that allows you to > > > > serialize Jon serializable classes? > > > > > > > > I think that in general Java serialisation is not efficient as it is > too > > > > general purpose. > > > > It also brings in a few Security issues. > > > > > > > > Maybe an alternative idea is to add some serialisation ad-hoc > mechanism in > > > > RexNode. > > > > We should also ensure that every RexNode will be able to be > serialized and > > > > deserialized. > > > > > > > > Enrico > > > > > > > > > > > > > So what do you think of the idea that makes RexNode implement the > > > > > Serializable interface? > > > > > > > > > > > > > > > -Rui > > > > > > > > > >
