Hi Team, We are trying to shift from HDFS State Manager to Rocks DB State Manager, but while doing POC we realised it is using much more off-heap space than expected. Because of this, the executors get killed with : *out of** physical memory exception.*
Could you please help in understanding, why is there a massive increase in off-heap space, and what can we do about it? We are using, SPARK 3.2.1 with 1 executor and 1 executor core, to understand the memory requirements - 1. Rocks DB Run - took 3.5 GB heap and 11.5 GB Res Memory 2. Hdfs State Manager - took 5 GB heap and 10 GB Res Memory. Thanks, Akshit Thanks and regards - Akshit Marwah
