itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-543620437
> > @PingHao - Both the issue- stuck executor, core dump - might be due to
the same reasons. Will debug and fix it.
> >
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-543619888
@marmbrus / @gatorsmile - Got your point on making it an external package.
I will close the PR and corresponding JIRA.
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-540070575
@PingHao - Both the issue- stuck executor, core dump - might be due to the
same reasons. Will debug and fix it.
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-539838772
> 1. we using flatMapGroupsWithState, it cause it fail at begining
Will update the PR with the fix
> 2. Rocksdb
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-533396055
> @itsvikramagr are you planning to resolve the remaining comments or
waiting on second opinion? I think the config is not
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-531092730
ping @gaborgsomogyi @dongjoon-hyun @HeartSaVioR
I have a comment from @gaborgsomogyi to resolve. Is there anything
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-525823628
@gaborgsomogyi - I have addressed some of your comments and replied to
remaining ones. Thanks again for reviewing the PR
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-521134445
> cool. is the issue here [#24922
(comment)](https://github.com/apache/spark/pull/24922#issuecomment-510327508)
resolved?
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-519130078
Longevity run results
**Setup**
Same as above
Executor Instance type = C5d.2xlarge
cores per
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-517556064
> I agree keeping state in memory is not scalable, and the result looks
promising. It might be better to have another kind
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-516792658
@HeartSaVioR, @gaborgsomogyi - I was able to fix the memory leaks and
address a lot of your comments.
I also ran
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-510327508
@HeartSaVioR, @gaborgsomogyi - thanks again for such a thorough review. I
will soon resolve all your comments.
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-508068035
@gaborgsomogyi - Thanks for your review. Let me get back after working on
your suggestions/feedbacks. Will improve the
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-507108250
Thanks, @HeartSaVioR for the review. Let me work on your comments. Also, I
am looking into generating performance numbers
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-505325772
Looking at the unit test failures. It's related to the rocksDbPath folder
name. Will make it configurable and update the
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-505292778
Thanks, @HeartSaVioR. I understand it a very big change. As suggested let me
create a stress test suite and paste some
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-505062488
Thanks @gaborgsomogyi
- Will fix the style problem asap and update the PR
- In my test setup, I was able to
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-504650735
ping @arunmahadevan
This is an automated message from
itsvikramagr commented on issue #24922: [SPARK-28120][SS] Rocksdb state
storage implementation
URL: https://github.com/apache/spark/pull/24922#issuecomment-504649951
ping @HeartSaVioR @tdas @jose-torres
This is an automated
19 matches
Mail list logo
