[ https://issues.apache.org/jira/browse/SPARK-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-10494: -------------------------------- Fix Version/s: 1.6.0 > Multiple Python UDFs together with aggregation or sort merge join may cause > OOM (failed to acquire memory) > ---------------------------------------------------------------------------------------------------------- > > Key: SPARK-10494 > URL: https://issues.apache.org/jira/browse/SPARK-10494 > Project: Spark > Issue Type: Bug > Components: PySpark, SQL > Affects Versions: 1.5.0 > Reporter: Davies Liu > Assignee: Reynold Xin > Priority: Critical > Fix For: 1.6.0, 1.5.1 > > > The RDD cache for Python UDF is removed in 1.4, then N Python UDFs in one > query stage could end up evaluate upstream (SparkPlan) 2^N times. > In 1.5, If there is aggregation or sort merge join in upstream SparkPlan, > they will cause OOM (failed to acquire memory). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org