itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-521134445 > cool. is the issue here [#24922 (comment)](https://github.com/apache/spark/pull/24922#issuecomment-510327508) resolved? Yes this is now resolved. I have made the following changes to resolve it - There was a typo in setting dataBlockSize. Instead of KBs, I was setting it to Bytes. - For range scan, I was creating a RocksIterator which was not closed cleanly. - I fixed some of the rocksDB configs. Taken help from [here](https://github.com/facebook/rocksdb/issues/4112#issuecomment-470269235). In particular, set Max open files, strictly limited caches (No pinning of metadata in cache), Disable FillCache for range scan operations. Overall the memory usage is now contained and it's around 1-3 GB depending upon the data.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
