Re: Correctly serializing "Number" as state in ProcessFunction

2021-04-26 Thread Robert Metzger
Quick comment on the kryo type registration and the messages you are seeing: The messages are expected: What the message is saying is that we are not serializing the type using Flink's POJO serializer, but we are falling back to Kryo. Since you are registering all the instances of Number that you

Re: Correctly serializing "Number" as state in ProcessFunction

2021-04-23 Thread Miguel Araújo
Thanks for your replies. I agree this is a somewhat general problem. I posted it here as I was trying to register the valid subclasses in Kryo but I couldn't get the message to go away, i.e., everything worked correctly but there was the complaint that GenericType serialization was being used.

Re: Correctly serializing "Number" as state in ProcessFunction

2021-04-21 Thread Arvid Heise
Hi Miguel, as Klemens said this is a rather general problem independent of Flink: How do you map Polymorphism in serialization? Flink doesn't have an answer on its own, as it's discouraged (A Number can have arbitrary many subclasses: how do you distinguish them except by classname? That adds a

Re: Correctly serializing "Number" as state in ProcessFunction

2021-04-20 Thread Klemens Muthmann
Hi, I guess this is more of a Java Problem than a Flink Problem. If you want it quick and dirty you could implement a class such as: public class Value { private boolean isLongSet = false; private long longValue = 0L; private boolean isIntegerSet = false; private int intValue =

Correctly serializing "Number" as state in ProcessFunction

2021-04-20 Thread Miguel Araújo
Hi everyone, I have a ProcessFunction which needs to store different number types for different keys, e.g., some keys need to store an integer while others need to store a double. I tried to use java.lang.Number as the type for the ValueState, but I got the expected "No fields were detected for