[
https://issues.apache.org/jira/browse/SPARK-60071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-60071:
-----------------------------------
Assignee: L. C. Hsieh
> Use identity equality for HybridRowQueue
> ----------------------------------------
>
> Key: SPARK-60071
> URL: https://issues.apache.org/jira/browse/SPARK-60071
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
>
> HybridRowQueue is a case class, so two queues created in the same task with
> the same TaskMemoryManager, temp dir, number of fields, SerializerManager and
> lockFree flag are equal. Because HybridRowQueue is a MemoryConsumer (via
> HybridQueue), this breaks memory management:
> 1. TaskMemoryManager tracks consumers in a HashSet. A second queue that
> equals an already registered one is never added, so it is never offered for
> spilling when another consumer needs memory, and its memory is not attributed
> in the memory usage breakdown on OOM.
> 2. HybridQueue.spill(size, trigger) returns 0 when trigger == this, which
> uses equals. A queue therefore refuses to spill when an equal but distinct
> queue triggers the spill.
> This can happen in a real query, e.g. when several partitions evaluated by a
> Python UDF node are processed in one task (coalesce), or when both sides of a
> sort-merge join contain Python UDF eval nodes with the same input width.
> The fix is to give HybridRowQueue identity equality by making it a regular
> class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
