HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505290444 Thanks for the hard work, @itsvikramagr ! I agree keeping state in memory is not scalable, and the result looks promising. It might be better to have another kind of benchmark here, like stress test, to see the performance on stateful operations and let end users guide whether they're mostly encouraged to use this implementation, or use this selectively. What I did for my patch was following: https://issues.apache.org/jira/browse/SPARK-21271 https://github.com/apache/spark/pull/21733#issuecomment-411207042 Btw, it would take some time to review your patch as the diff is 2000+ lines, as well as I also have some works on my plate. You might want to spend time to get familiar with style guide if you haven't - there're some rules which are not checked via scala style check but reviewers will point out.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
