Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16374
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
I don't use anymore approach above.
To unpersist unnecessary RDD, I hacked MapWithStateDStream a little bit by
calling unpersist for previously generated RDDs in internalMapWithStateStream.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16374
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16374
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user wobuxiangtong commented on the issue:
https://github.com/apache/spark/pull/16374
Excuse meï¼
I am using mapwithstate to storage data in sparkstreamingãWhat
confused me is that rememberDuration must more than checkpointDurationãI read
the code of MapWithStateRD
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
DStream.clearMetadata allows to keep metadata for up to
checkpoint_interval_multiplier RDDs.
This PR clears metadata as soon as possible. As the result applicatin uses
less memory and less GC o
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16374
> scala.NotImplementedError: put() should not be called on an EmptyStateMap
This is another issue and not related to the memory issue. Could you create
a new ticket, please?
For thi
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
cc @zsxwing, @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
We use spark 2.0.0.
Changing spark.memory.fraction (we tried 0.3 and 0.4) (also tried to
increase memoryOverhead to 20% of executor memory) does not help.
The application is not s
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
We dont use windows operations (Kafka streaming) - so we don't need 'old'
RDDs :)
I need to double check if spark.memory.fraction help to us.
---
If your project is set up for it, you can
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16374
Some operators (e.g., window) may still need to use these RDD, dropping
them will slow your application. That's why `DStream` uses `rememberDuration`
to filter RDDs.
You can [tune the memor
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user vpchelko commented on the issue:
https://github.com/apache/spark/pull/16374
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and w
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16374
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
14 matches
Mail list logo
