[
https://issues.apache.org/jira/browse/SPARK-51499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-51499.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: Verified against master
(HEAD e9d2378b5a27). This is a "why does the code do X" design question, not a
defect or change request. EvalPythonExec.scala:73
(`child.execute().map(_.copy())`) is intentional and required for correctness:
EvalPythonEvaluatorFactory.scala buffers each input row via
`HybridRowQueue.add(inputRow.asInstanceOf[UnsafeRow])` (line 114) and drains it
later to reunite Python outputs with original inputs. Spark upstream operators
reuse a single mutable UnsafeRow per iteration, so queuing references without
copying would corrupt buffered rows. The reporter's OOM worry is explic
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> why copy child RDD in EvalPythonExec? it may lead into risk of oom
> ------------------------------------------------------------------
>
> Key: SPARK-51499
> URL: https://issues.apache.org/jira/browse/SPARK-51499
> Project: Spark
> Issue Type: Question
> Components: PySpark, SQL
> Affects Versions: 3.2.0
> Reporter: fengtong
> Priority: Major
>
> I found
> val inputRDD = child.execute().map(_.copy())
> in "doExecute()" method of trait EvalPythonExec.
> I'm concerning about the reason why a copy leveraged here, which may lead
> into risk of oom
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
