This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 838524f062df [SPARK-59762][PYTHON][FOLLOWUP] Stop UDTFWithPartitions
from keeping udtf_info alive for the whole task
838524f062df is described below
commit 838524f062dfa6ca5b1692c5739de5e378799946
Author: Constantin-Tiberiu Craiu
<[email protected]>
AuthorDate: Tue Sep 29 12:06:08 2026 -0700
[SPARK-59762][PYTHON][FOLLOWUP] Stop UDTFWithPartitions from keeping
udtf_info alive for the whole task
### What changes were proposed in this pull request?
Follow-up to https://github.com/apache/spark/pull/59013
Use the `partition_child_indexes` argument in `UDTFWithPartitions.__init__`
instead of the enclosing `udtf_info` and avoid `udtf_info` being keppt by the
closure.
### Why are the changes needed?
Reduces memory usage in UDTFs with PARTITION BY.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #59136 from craiuconstantintiberiu/udtf-partitions-release-init-info.
Authored-by: Constantin-Tiberiu Craiu
<[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 0bebce96e477fb4ecf893e72505a620644dc1f15)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index a957a5d673d4..80ac5ab95e24 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -599,7 +599,7 @@ def read_udtf(pickleSer, udtf_info, eval_type, runner_conf,
eval_conf):
self._create_udtf: Callable = create_udtf
self._udtf = create_udtf()
self._prev_arguments: list = list()
- self._partition_child_indexes: list =
udtf_info.partition_child_indexes
+ self._partition_child_indexes: list = partition_child_indexes
self._eval_raised_skip_rest_of_input_table: bool = False
def eval(self, *args, **kwargs) -> Iterator:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
