[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matei Zaharia updated SPARK-3466:
-
Priority: Critical (was: Major)
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Davies Liu
Priority: Critical
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-3466:
---
Assignee: Davies Liu (was: Matthew Cheah)
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Davies Liu
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Ash updated SPARK-3466:
--
Assignee: Matthew Cheah
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Assignee: Matthew Cheah
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action
[
https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Ash updated SPARK-3466:
--
Description: Right now, operations like {{collect()}} and {{take()}} can
crash the driver with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will make
the driver abort a job if its result is too big. We can set it to some fraction
of the driver's memory by default, or to something like 100 MB. (was: Right
now, operations like collect() and take() can crash the driver if they bring
back too many data. We should add a spark.driver.maxResultSize setting (or
something like that) that will make the driver abort a job if its result is too
big. We can set it to some fraction of the driver's memory by default, or to
something like 100 MB.)
Limit size of results that a driver collects for each action
Key: SPARK-3466
URL: https://issues.apache.org/jira/browse/SPARK-3466
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Matei Zaharia
Right now, operations like {{collect()}} and {{take()}} can crash the driver
with an OOM if they bring back too many data. We should add a
{{spark.driver.maxResultSize}} setting (or something like that) that will
make the driver abort a job if its result is too big. We can set it to some
fraction of the driver's memory by default, or to something like 100 MB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
