itsvikramagr edited a comment on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-510327508 @HeartSaVioR, @gaborgsomogyi - thanks again for such a thorough review. I will soon resolve all your comments. Meanwhile, I was doing the performance analysis for various use-cases. While for complete output mode, I was able to ingest 2x-3x data than memory based backend in 1 executor and 5G container memory setup, I was not seeing encouraging results when the outputMode was "Append". I found that there is some memory leak (similar to one reported [here](https://github.com/facebook/rocksdb/issues/3216)) I am digging deeper to find the root cause. The fix might be to tune some rocksdb config or fall back to use an older version of rocksdb. (As @skonto commented earlier, code path for deletion of older states might need refactoring to get best out of rocksdb)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
