Vishal, If you decide you can't live with dropping that state, [1] is a complete example showing how to migrate from Kryo by using the state processor API.
David [1] https://www.docs.immerok.cloud/docs/cookbook/migrating-state-away-from-kryo/ On Fri, Sep 16, 2022 at 8:32 AM Vishal Santoshi <vishal.santo...@gmail.com> wrote: > Thank you for the clarification. I thought so to, > > Unfortunately my state are generics based and those are definitely not > treated as a POJO , though it has all the constructs ( no arg constructor, > getters/setters etc ). I will likely take an at least once hit by > > Changing the uid of that specific Operator, and restart with Allow > non-restored state ... This will ignore state that cannot be restored ( > for the previous uid ) , construct state for the new uid and not affect > other operators ( including the kafka consumer operators ). I can live with > it, I think. > > On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias < > matthias.schwa...@viseca.ch> wrote: > >> Hi Vishal, >> >> >> >> Good news and bad news đ: >> >> >> >> - Bad: Kryo serializer cannot be used for schema evolution, see [1] >> - Good: not all is lost here, >> - If you happen to have state that you cannot afford to lose, you >> can transcode it by means of the savepoint API [2], >> - However, this takes quite some effort >> - In general, if you ever plan to migrate/extend your schemas, choose >> a data type that supports schema migration [1], >> - In your case, PoJo types would be the closest to your original >> implementation >> - You can disable Kryo in configuration to avoid this situation in >> the future, by the way, >> - Kryo serializer is quite slow compared to the other options and I >> believe it is only there as a (emergency) fallback solution: [3] >> >> >> >> Feel free to ask for clarification đ >> >> >> >> Thias >> >> >> >> >> >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution >> >> [2] >> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ >> >> [3] >> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html >> >> >> >> >> >> >> >> *From:* Vishal Santoshi <vishal.santo...@gmail.com> >> *Sent:* Friday, September 16, 2022 1:17 AM >> *To:* user <user@flink.apache.org> >> *Subject:* Re: A question about restoring state with an additional >> variable with kryo >> >> >> >> â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â >> >> >> >> The exception thrown is as follows. I realize that it is trying to read >> the long value. How do I signal to kryo that it is OK and that he object >> can have a default value >> >> >> >> Caused by: java.io.EOFException: No more bytes left. >> >> at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput >> .require(NoFetchingInput.java:80) >> >> at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690) >> >> at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685) >> >> at com.esotericsoftware.kryo.serializers. >> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133) >> >> at com.esotericsoftware.kryo.serializers. >> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123) >> >> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >> >> at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField >> .java:113) >> >> at com.esotericsoftware.kryo.serializers.FieldSerializer.read( >> FieldSerializer.java:528) >> >> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >> >> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer >> .deserialize(KryoSerializer.java:354) >> >> at org.apache.flink.api.common.typeutils.CompositeSerializer >> .deserialize(CompositeSerializer.java:156) >> >> at org.apache.flink.contrib.streaming.state.RocksDBValueState.value( >> RocksDBValueState.java:89) >> >> >> >> On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi < >> vishal.santo...@gmail.com> wrote: >> >> << How do I make sure that when reconstituting the state, kryo does not >> complain? It tries to map the previous state to the new definition of Class >> A and complains that it cannot read the value for `String b`. >> >> >> >> >> How do I make sure that when reconstituting the state, kryo does not >> complain? It tries to map the previous state to the new definition of Class >> A and complains that it cannot read the value for `long b`. >> >> >> >> Sorry a typo >> >> >> >> >> >> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi < >> vishal.santo...@gmail.com> wrote: >> >> I have state in rocksDB that represents say >> >> >> >> class A { >> >> String a >> >> } >> >> >> >> I now change my class and add another variable >> >> >> >> >> Class A { >> >> String a; >> >> long b = 0; >> >> } >> >> >> >> How do I make sure that when reconstituting the state, kryo does not >> complain? It tries to map the previous state to the new definition of Class >> A and complains that it cannot read the value for `String b`. >> >> >> >> Unfortunately the state is not using POJO serializer. >> >> >> >> Thanks and Regards. >> >> >> >> Vishal >> >> >> >> >> >> >> >> >> >> Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und >> beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die >> Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, >> ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und >> Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir >> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie >> eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung >> dieser Informationen ist streng verboten. >> >> This message is intended only for the named recipient and may contain >> confidential or privileged information. As the confidentiality of email >> communication cannot be guaranteed, we do not accept any responsibility for >> the confidentiality and the intactness of this message. If you have >> received it in error, please advise the sender by return e-mail and delete >> this message and any attachments. Any unauthorised use or dissemination of >> this information is strictly prohibited. >> >
