[ https://issues.apache.org/jira/browse/SPARK-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547888#comment-14547888 ]
Iulian Dragos commented on SPARK-3334: -------------------------------------- This was fixed in Mesos 0.21.0. Should we close this ticket? > Spark causes mesos-master memory leak > ------------------------------------- > > Key: SPARK-3334 > URL: https://issues.apache.org/jira/browse/SPARK-3334 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 1.0.2 > Environment: Mesos 0.16.0/0.19.0 > CentOS 6.4 > Reporter: Iven Hsu > > The {{akkaFrameSize}} is set to {{Long.MaxValue}} in MesosBackend to > workaround SPARK-1112, this causes all serialized task result is sent using > Mesos TaskStatus. > mesos-master stores TaskStatus in memory, and when running Spark, its memory > grows very fast, and will be OOM killed. > See MESOS-1746 for more. > I've tried to set {{akkaFrameSize}} to 0, mesos-master won't be killed, > however, the driver will block after success unless I use {{sc.stop()}} to > quit it manually. Not sure if it's related to SPARK-1112. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org