fengtong created SPARK-51499:
--------------------------------
Summary: why copy child RDD in EvalPythonExec? it may lead into
risk of oom
Key: SPARK-51499
URL: https://issues.apache.org/jira/browse/SPARK-51499
Project: Spark
Issue Type: Question
Components: PySpark, SQL
Affects Versions: 3.2.0
Reporter: fengtong
I found
val inputRDD = child.execute().map(_.copy())
in "doExecute()" method of trait EvalPythonExec.
I'm concerning about the reason why a copy leveraged here, which may lead into
risk of oom
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
