itsvikramagr commented on issue #24922: [SPARK-28120][SS]  Rocksdb state 
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-521134445
 
 
   > cool. is the issue here [#24922 
(comment)](https://github.com/apache/spark/pull/24922#issuecomment-510327508) 
resolved?
   
   Yes this is now resolved. I have made the following changes to resolve it 
   - There was a typo in setting dataBlockSize. Instead of KBs, I was setting 
it to Bytes. 
   - For range scan, I was creating a RocksIterator which was not closed 
cleanly.
   - I fixed some of the rocksDB configs. Taken help from 
[here](https://github.com/facebook/rocksdb/issues/4112#issuecomment-470269235). 
In particular, set Max open files, strictly limited caches (No pinning of 
metadata in cache), Disable FillCache for range scan operations.
   
   Overall the memory usage is now contained and it's around 1-3 GB depending 
upon the data.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to