L. C. Hsieh created SPARK-52809:
-----------------------------------
Summary: Don't hold reader and iterator references for all
partitions in task completion listeners for metric update
Key: SPARK-52809
URL: https://issues.apache.org/jira/browse/SPARK-52809
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.4
Reporter: L. C. Hsieh
For each partition iterator, currently we add one task completion listener used
to update final metrics if the task is stopped early. In the listener, the
reader and iterator are held. So if the partition is normally exhausted, the
references cannot be released early. It is a problem especially if the
references are heavy as reported by
https://github.com/apache/iceberg/issues/13297.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
