Re: Troubleshooting checkpoint expiration

2024-08-07 Thread Alexis Sarda-Espinosa
I must ask again if anyone at least knows if Flink's file system can expose more detailed exceptions when things go wrong, Azure support is asking for specific exception messages to decide how to troubleshoot. Regards, Alexis. Am Di., 23. Juli 2024 um 13:39 Uhr schrieb Alexis Sarda-Espinosa < sar

Can we share states across tasks/operators

2024-08-07 Thread Sachin Mittal
Hi, I have a stream which starts from a source and is keyed by a field f. With the stream process function, I can emit the processed record downstream and also update state based on the records it received for the same key. Now I have another stream which starts from another source and is of the s

RE: Can we share states across tasks/operators

2024-08-07 Thread Schwalbe Matthias
Hi Sachin, Just as an idea, while you cannot easily share state across operators, you can do so within the same operator: * For two such input streams you could connect() the two streams into a ConnectedStreams and then process() by means of a KeyedCoProcessFunction * For more than two

Re: Can we share states across tasks/operators

2024-08-07 Thread Sachin Mittal
Hi, Thanks for pointing me in the right direction. I think I can union the two streams as they have the same type and in the process function have a shared state since now both are the same stream. I will figure this around connecting streams and using keyed state stores. Sachin On Wed, Aug 7,

Restore rocksDB from savepoint exception

2024-08-07 Thread Bjarke Tornager
Hi, I am doing stateful streaming (keyed coprocessing) using rocksDB. When my job restarts to adjust parallelism, and restores from savepoint, I quite often get the following error: 2024-08-07 19:43:09 java.lang.Exception: Exception while creating StreamOperatorStateContext. at org.apache.flink.s