[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248112#comment-17248112 ] Apache Spark commented on SPARK-22256: -- User 'dmcwhorter' has created a pull request for this issue: https://github.com/apache/spark/pull/30739 > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0, 2.3.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248110#comment-17248110 ] Apache Spark commented on SPARK-22256: -- User 'dmcwhorter' has created a pull request for this issue: https://github.com/apache/spark/pull/30739 > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0, 2.3.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947177#comment-16947177 ] David McWhorter commented on SPARK-22256: - I created this pull request https://github.com/pmackles/spark/pull/1 to update the branch for this pull request with changes on branch https://github.com/dmcwhorter/spark/tree/dmcwhorter-SPARK-22256. The updates do the following: 1. Rebase pmackles/paul-SPARK-22256 onto the latest apache/spark master branch 2. Add a test case for when the default value of spark.mesos.driver.memoryOverhead is 10% of driver memory as requested in apache#21006 My hope is this update will make this pull request ready to merge and include in the next spark release. > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0, 2.3.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430262#comment-16430262 ] paul mackles commented on SPARK-22256: -- created a PR on behalf of [~clehene] since he as moved on to other projects: https://github.com/apache/spark/pull/21006 > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0, 2.3.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Fix For: 2.3.1, 2.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430260#comment-16430260 ] Apache Spark commented on SPARK-22256: -- User 'pmackles' has created a pull request for this issue: https://github.com/apache/spark/pull/21006 > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0, 2.3.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200876#comment-16200876 ] Sean Owen commented on SPARK-22256: --- You don't need it assigned > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200855#comment-16200855 ] Cosmin Lehene commented on SPARK-22256: --- It seems I can't assign this issue to myself. [~srowen] can you make me a contributor, please? Thank you > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead
[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200852#comment-16200852 ] Cosmin Lehene commented on SPARK-22256: --- I have a patch, still testing it. I will go over the contribution guidelines and try to submit it ASAP. > Introduce spark.mesos.driver.memoryOverhead > > > Key: SPARK-22256 > URL: https://issues.apache.org/jira/browse/SPARK-22256 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 2.2.0 >Reporter: Cosmin Lehene >Priority: Minor > Labels: docker, memory, mesos > Original Estimate: 24h > Remaining Estimate: 24h > > When running spark driver in a container such as when using the Mesos > dispatcher service, we need to apply the same rules as for executors in order > to avoid the JVM going over the allotted limit and then killed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
