[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-09-12 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-531092730 ping @gaborgsomogyi @dongjoon-hyun @HeartSaVioR I have a comment from @gaborgsomogyi to resolve. Is there anything

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-08-28 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-525823628 @gaborgsomogyi - I have addressed some of your comments and replied to remaining ones. Thanks again for reviewing the PR

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-08-14 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-521134445 > cool. is the issue here [#24922 (comment)](https://github.com/apache/spark/pull/24922#issuecomment-510327508) resolved?

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-08-07 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-519130078 Longevity run results **Setup** Same as above Executor Instance type = C5d.2xlarge cores per

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-08-01 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-517556064 > I agree keeping state in memory is not scalable, and the result looks promising. It might be better to have another kind

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-07-31 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-516792658 @HeartSaVioR, @gaborgsomogyi - I was able to fix the memory leaks and address a lot of your comments. I also ran

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-07-10 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-510327508 @HeartSaVioR, @gaborgsomogyi - thanks again for such a thorough review. I will soon resolve all your comments.

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-07-03 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-508068035 @gaborgsomogyi - Thanks for your review. Let me get back after working on your suggestions/feedbacks. Will improve the

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-30 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-507108250 Thanks, @HeartSaVioR for the review. Let me work on your comments. Also, I am looking into generating performance numbers

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-25 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505325772 Looking at the unit test failures. It's related to the rocksDbPath folder name. Will make it configurable and update the

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-24 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505292778 Thanks, @HeartSaVioR. I understand it a very big change. As suggested let me create a stress test suite and paste some

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-24 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-505062488 Thanks @gaborgsomogyi - Will fix the style problem asap and update the PR - In my test setup, I was able to

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-22 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-504650735 ping @arunmahadevan This is an automated message from

[GitHub] [spark] itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation

2019-06-22 Thread GitBox
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state storage implementation URL: https://github.com/apache/spark/pull/24922#issuecomment-504649951 ping @HeartSaVioR @tdas @jose-torres This is an automated