[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795061#comment-17795061
]
wang fanming commented on SPARK-44900:
--
Of course, the following is my execution in the spark3.5.1
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794817#comment-17794817
]
Dongjoon Hyun commented on SPARK-44900:
---
Could you try this with Apache Spark 3.5.0, please?
>
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792872#comment-17792872
]
王范明 commented on SPARK-44900:
-
I have analyzed the program execution details in the logs and it seems that
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760700#comment-17760700
]
Varun Nalla commented on SPARK-44900:
-
[~yxzhang] / [~yao] any update for us ?
> Cached DataFrame
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759821#comment-17759821
]
Yauheni Audzeichyk commented on SPARK-44900:
[~yxzhang] looks like it is just disk usage
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759818#comment-17759818
]
Yuexin Zhang commented on SPARK-44900:
--
Hi [~varun2807] [~yaud] did you check the actual cached
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759703#comment-17759703
]
Varun Nalla commented on SPARK-44900:
-
[~yao] hope you got a chance to look into what [~yaud]
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759130#comment-17759130
]
Yauheni Audzeichyk commented on SPARK-44900:
[~yao] we tried
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758868#comment-17758868
]
Kent Yao commented on SPARK-44900:
--
Set spark.cleaner.periodicGC.interval to a smaller value(3min)
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758674#comment-17758674
]
Varun Nalla commented on SPARK-44900:
-
We do not keep on adding , they are existing cached RDD's
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758376#comment-17758376
]
Kent Yao commented on SPARK-44900:
--
Please note that there is a storage limit in place. Adding items
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758316#comment-17758316
]
Varun Nalla commented on SPARK-44900:
-
[~yao] Thanks for the comment. However we can't release as
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758315#comment-17758315
]
Kent Yao commented on SPARK-44900:
--
How about releasing the cached rdds if you never touch it again
>
[
https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758161#comment-17758161
]
Varun Nalla commented on SPARK-44900:
-
[~yao] , is there a way I could prioritize this issue as it's
14 matches
Mail list logo
