[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856470#comment-16856470 ] Iris Shaibsky commented on SPARK-25380: --- We are facing it also on spark 2.4.2, I see that the PR

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769724#comment-16769724 ] t oo commented on SPARK-25380: -- i am also facing issue on spark2.3 > Generated plans occupy over 50% of

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702362#comment-16702362 ] Dave DeCaprio commented on SPARK-25380: --- I've created a PR that should address this.  It limits

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691519#comment-16691519 ] Dave DeCaprio commented on SPARK-25380: --- I would like to comment that we are also seeing this. 

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633145#comment-16633145 ] Marcelo Vanzin commented on SPARK-25380: We can provide ways to diminish the effect of large

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631313#comment-16631313 ] Jungtaek Lim commented on SPARK-25380: -- Btw, reproducer still helps when we tackle it with only UI

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631295#comment-16631295 ] Jungtaek Lim commented on SPARK-25380: -- IMHO it depends on how we see the issue and how we would

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631236#comment-16631236 ] Marcelo Vanzin commented on SPARK-25380: If all you want is see this live, why do you need a

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630053#comment-16630053 ] Jungtaek Lim commented on SPARK-25380: -- [~spektom] I guess physical plan description being 200M is

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626790#comment-16626790 ] Michael Spector commented on SPARK-25380: - It's really difficult to extract a working

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623087#comment-16623087 ] Jungtaek Lim commented on SPARK-25380: -- I thought about this as edge case which we might be unsure

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616675#comment-16616675 ] Nir Hedvat commented on SPARK-25380: Same problem here (using Spark 2.3.1) > Generated plans occupy

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612433#comment-16612433 ] Marcelo Vanzin commented on SPARK-25380: Yep. That's a 200MB plan description string... >

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611606#comment-16611606 ] Michael Spector commented on SPARK-25380: - [~vanzin] Here's the breakdown: !Screen Shot

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611012#comment-16611012 ] Marcelo Vanzin commented on SPARK-25380: Another bit of information that would be useful is the

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608045#comment-16608045 ] Michael Spector commented on SPARK-25380: - Originally reported in SPARK-23670   > Generated