[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188998#comment-14188998
]
Apache Spark commented on SPARK-3466:
-
User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/3003
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Davies Liu
Priority: Critical
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178025#comment-14178025
]
Matei Zaharia commented on SPARK-3466:
--
Ah, I see, that concern makes sense if the total size of the results is large,
even though each result might be small.
For large task results, the driver should be fetching them from executors using
the block manager. This means that once they get stored into the block manager
on each worker, the driver can choose whether it wants to fetch them all at
once or not. I'd go for a solution like the following:
- Each task only adds a result to their local block store if it's smaller than
the limit (otherwise it can throw an error right there).
- The result fetcher in the driver is updated to track total size; this might
be trickier, since I believe it can currently fetch stuff concurrently.
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Matthew Cheah
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178857#comment-14178857
]
Patrick Wendell commented on SPARK-3466:
I spoke with Matt today and I'm re-assigning this to [~davies] who has cycles
to look at it.
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Davies Liu
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177826#comment-14177826
]
Matt Cheah commented on SPARK-3466:
---
I got caught up in some other things, so I haven't had a chance to deeply look
into this. Off the top of my head though I was considering communicating back
the size of the partitions to the driver, using ResultTask, or something
similar. We'd want a way for the driver to know what the combined size of the
computation results will be.
I'm still unfamiliar with the codebase so I would have to do some more
investigation for that approach, but if you believe your approach to also be
sound feel free to point me to places in the code to look. However, my hunch is
that limiting the serialization stream on the executors might not always work.
If the data distribution across the executors at the end of the job is
extremely unbalanced, then one executor might serialize back a giant block that
won't fit in the stream even though that block could fit inside the driver
program. Or (more unlikely if this were configured correctly) if the combined
bytes in all of the executors is more than the driver can handle, despite the
fact that the serialization stream of the individual executors are only
near-capacity, then we could still run into this issue.
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Matthew Cheah
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174148#comment-14174148
]
Matt Cheah commented on SPARK-3466:
---
I'll look into this. Someone please assign to me!
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146640#comment-14146640 ] Andrew Ash commented on SPARK-3466: --- How would you design this feature? I can imagine measuring the size of partitions / RDD elements while they are held in memory across the cluster, sending those sizes back to the driver, and having the driver throw an exception if the requested size exceeds the threshold. Otherwise proceed as normal. Is that how you were envisioning implementation? Limit size of results that a driver collects for each action Key: SPARK-3466 URL: https://issues.apache.org/jira/browse/SPARK-3466 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Matei Zaharia Right now, operations like collect() and take() can crash the driver if they bring back too many data. We should add a spark.driver.maxResultSize setting (or something like that) that will make the driver abort a job if its result is too big. We can set it to some fraction of the driver's memory by default, or to something like 100 MB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
