Hi, We are facing a new issue related to RockDb when deploying a new version of our job, which is adding 3 more operators. We are using flink 1.17.1 with RockDb on Java 11.
We get an exception from another pre-existing operator during its initialization. That operator and the new ones have different ids and they are not in the same path of execution. So we are struggling to correlate the new operators with this error. In particular the error message is *our-operator-name -> Timestamps/Watermarks (12/12)#4 (49136e1eafd63fcf33f7574ed87c768a_6c6dc6fa91330c57e3390709d51035b1_11_4) switched from INITIALIZING to FAILED with failure cause:"* and the stack trace of the error is *o.r.RocksDBException: end key comes before start keyat org.rocksdb.RocksDB.deleteRange(RocksDB.java)at org.rocksdb.RocksDB.deleteRange(RocksDB.java:1493)at o.a.f.c.s.s.RocksDBIncrementalCheckpointUtils.deleteRange(RocksDBIncrementalCheckpointUtils.java:153)at o.a.f.c.s.s.RocksDBIncrementalCheckpointUtils.clipDBWithKeyGroupRange(RocksDBIncrementalCheckpointUtils.java:129)at o.a.f.c.s.s.r.RocksDBIncrementalRestoreOperation.initDBWithRescaling(RocksDBIncrementalRestoreOperation.java:388)... 21 common frames omittedWrapped by: o.a.f.r.s.BackendBuildingException: Failed to clip DB after initialization.at o.a.f.c.s.s.r.RocksDBIncrementalRestoreOperation.initDBWithRescaling(RocksDBIncrementalRestoreOperation.java:397)at o.a.f.c.s.s.r.RocksDBIncrementalRestoreOperation.restoreWithRescaling(RocksDBIncrementalRestoreOperation.java:295)at o.a.f.c.s.s.r.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:167)at o.a.f.c.s.s.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:329)at o.a.f.c.s.s.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)at o.a.f.c.s.s.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)at o.a.f.s.a.o.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336)at o.a.f.s.a.o.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)at o.a.f.s.a.o.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)... 13 common frames omittedWrapped by: o.a.f.u.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_6c6dc6fa91330c57e3390709d51035b1_(12/12) from any of the 1 provided restore options.at o.a.f.s.a.o.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)at o.a.f.s.a.o.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)at o.a.f.s.a.o.StreamTaskStateInitializerImpl.stream...* This RockDb ticket https://github.com/facebook/rocksdb/issues/8239 could be correlated to this problem. Any help would be really appreciated. Many Thanks
