[
https://issues.apache.org/jira/browse/SPARK-60071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126605#comment-18126605
]
Dongjoon Hyun commented on SPARK-60071:
---------------------------------------
This is reverted from branch-4.2 due to the compilation failure.
-
https://github.com/apache/spark/commit/e8e8f37a7262b1081b4001c4a34fd1d4ac7caf9a
> Use identity equality for HybridRowQueue
> ----------------------------------------
>
> Key: SPARK-60071
> URL: https://issues.apache.org/jira/browse/SPARK-60071
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> HybridRowQueue is a case class, so two queues created in the same task with
> the same TaskMemoryManager, temp dir, number of fields, SerializerManager and
> lockFree flag are equal. Because HybridRowQueue is a MemoryConsumer (via
> HybridQueue), this breaks memory management:
> 1. TaskMemoryManager tracks consumers in a HashSet. A second queue that
> equals an already registered one is never added, so it is never offered for
> spilling when another consumer needs memory, and its memory is not attributed
> in the memory usage breakdown on OOM.
> 2. HybridQueue.spill(size, trigger) returns 0 when trigger == this, which
> uses equals. A queue therefore refuses to spill when an equal but distinct
> queue triggers the spill.
> This can happen in a real query, e.g. when several partitions evaluated by a
> Python UDF node are processed in one task (coalesce), or when both sides of a
> sort-merge join contain Python UDF eval nodes with the same input width.
> The fix is to give HybridRowQueue identity equality by making it a regular
> class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
