[GitHub] [spark] HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-521863843 Recently I also start working on some task so it might take time for me back to review this. I'll try to review (even partially) when I get some time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505293839 FYI: Just think out loud since I'm being cc-ed first, I'm just a one of contributors, not committers or PMC members of Apache Spark. In case of you might get confused due to "MEMBER" badge - the badge just means I'm one of committers in "any" of ASF projects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505291092 And please take a deep look at build result if it fails, and try to fix if build failure is related to your patch. In some cases, build output log has a guide message to fix the issue, like this case. ``` Spark's published dependencies DO NOT MATCH the manifest file (dev/spark-deps). To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'. diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/pr-deps/spark-deps-hadoop-2.7 index 62b00f3..7e33e82 100644 --- a/dev/deps/spark-deps-hadoop-2.7 +++ b/dev/pr-deps/spark-deps-hadoop-2.7 @@ -171,6 +171,7 @@ parquet-jackson-1.10.1.jar protobuf-java-2.5.0.jar py4j-0.10.8.1.jar pyrolite-4.23.jar +rocksdbjni-6.0.1.jar scala-compiler-2.12.8.jar scala-library-2.12.8.jar scala-parser-combinators_2.12-1.1.0.jar ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation
HeartSaVioR commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505290444 Thanks for the hard work, @itsvikramagr ! I agree keeping state in memory is not scalable, and the result looks promising. It might be better to have another kind of benchmark here, like stress test, to see the performance on stateful operations and let end users guide whether they're mostly encouraged to use this implementation, or use this selectively. What I did for my patch was following: https://issues.apache.org/jira/browse/SPARK-21271 https://github.com/apache/spark/pull/21733#issuecomment-411207042 Btw, it would take some time to review your patch as the diff is 2000+ lines, as well as I also have some works on my plate. You might want to spend time to get familiar with style guide if you haven't - there're some rules which are not checked via scala style check but reviewers will point out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
