This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1a8c26c3f67e [SPARK-52829][PYTHON][FOLLOWUP] Remove unnecessary
special handling
1a8c26c3f67e is described below
commit 1a8c26c3f67e72542d678c355d030cb5ffcec5ff
Author: Takuya Ueshin <ues...@databricks.com>
AuthorDate: Tue Jul 22 11:29:57 2025 +0900
[SPARK-52829][PYTHON][FOLLOWUP] Remove unnecessary special handling
### What changes were proposed in this pull request?
Removes unnecessary special handling for empty schema in UDTF with Arrow
path.
### Why are the changes needed?
`LocalDataToArrowConversion.convert` handles the empty schema properly
after https://github.com/apache/spark/pull/51523.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51606 from ueshin/issues/SPARK-52829/udtf.
Authored-by: Takuya Ueshin <ues...@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/worker.py | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index d839cf00eb0a..eb8a60c7e428 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -1668,12 +1668,9 @@ def read_udtf(pickleSer, infile, eval_type):
pa.RecordBatch.from_pylist(data,
schema=pa.schema(list(arrow_return_type)))
]
try:
- ret = LocalDataToArrowConversion.convert(
+ return LocalDataToArrowConversion.convert(
data, return_type, prefers_large_var_types
).to_batches()
- if len(return_type.fields) == 0:
- return [pa.RecordBatch.from_struct_array(pa.array([{}]
* len(data)))]
- return ret
except Exception as e:
raise PySparkRuntimeError(
errorClass="UDTF_ARROW_TYPE_CONVERSION_ERROR",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
