Good morning Lorenzo,
You may want to implement
org.apache.flink.streaming.api.checkpoint.CheckpointedFunction interface in
your KeyedProcessFunction.
Btw. By the time initializeState(…) is called, the state backend is fully
initialized and can be read and written to (which is not the case for
Hello Kanchi,
It's recommended to submit a separate request or issue for the problem
you're encountering, as the data pipeline is distinct from the one Neha
raised. This will help ensure that each issue can be addressed individually
and efficiently.
Hello Neha,
Not sure about the issue you are
Hello Alexis,
I don't think data in RocksDB resides in JVM even with function calls.
For more details, check the link below:
https://github.com/facebook/rocksdb/wiki/RocksDB-Overview#3-high-level-architecture
RocksDB has three main components - memtable, sstfile and WAL(not used in
Flink as
Hi!
We just encountered a similar issue.
This is usually caused by: 1) Akka failed sending the message silently, due
to problems like oversized payload or serialization failures. In that case,
you should find detailed error information in the logs. 2) The recipient
needs more time for
Hi Asimansu
The memory RocksDB manages is outside the JVM, yes, but the mentioned
subsets must be bridged to the JVM somehow so that the data can be exposed
to the functions running inside Flink, no?
Regards,
Alexis.
On Thu, 15 Feb 2024, 14:06 Asimansu Bera, wrote:
> Hello Alexis,
>
>
Hello everyone,
I have a convoluted problem.
I am implementing a KeyedProcessFunction that keeps some non-serializable
"state" in memory, in a transient Map (key = stream key, value = the
non-serializable "state").
I can extract a serializable representation to put in Flink state, and I
can
Hello Alexis,
RocksDB resides off-heap and outside of JVM. The small subset of data ends
up on the off-heap in the memory.
For more details, check the following link:
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/memory/mem_setup_tm/#managed-memory
I hope this
Hello,
Most info regarding RocksDB memory for Flink focuses on what's needed
independently of the JVM (although the Flink process configures its limits
and so on). I'm wondering if there are additional special considerations
with regards to the JVM heap in the following scenario.
Assuming a key
