This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 688a30b03b56 [SPARK-53433][TESTS][FOLLOW-UP] Make the test compatible
with PyArrow 15.0.0
688a30b03b56 is described below
commit 688a30b03b563ec6722c1a9efbd491379ec48d09
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Tue Sep 2 09:30:34 2025 +0900
[SPARK-53433][TESTS][FOLLOW-UP] Make the test compatible with PyArrow 15.0.0
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/52172 that
makes the test compatible with PyArrow 0.15.
### Why are the changes needed?
Tests fail with PyArrow 0.15
https://github.com/apache/spark/actions/runs/17355567623/job/49268068508
`StructArray.from_arrays(..., type=...)` is available from PyArrow 15.0.0
(https://arrow.apache.org/docs/18.0/python/generated/pyarrow.StructArray.html#pyarrow.StructArray.from_arrays).
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52186 from HyukjinKwon/SPARK-53433-followup.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
index 633e33bdb0d8..d6e010d8d2a9 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py
@@ -525,14 +525,13 @@ class ScalarArrowUDFTestsMixin:
pa.field("value", pa.binary(), nullable=False),
pa.field("metadata", pa.binary(), nullable=False,
metadata={b"variant": b"true"}),
]
- variant_type = pa.struct(fields)
@arrow_udf("variant")
def scalar_f(v: pa.Array) -> pa.Array:
assert isinstance(v, pa.Array)
v = pa.array([bytes([12, i.as_py()]) for i in v], pa.binary())
m = pa.array([bytes([1, 0, 0]) for i in v], pa.binary())
- return pa.StructArray.from_arrays([v, m], type=variant_type)
+ return pa.StructArray.from_arrays([v, m], fields=fields)
@arrow_udf("variant")
def iter_f(it: Iterator[pa.Array]) -> Iterator[pa.Array]:
@@ -540,7 +539,7 @@ class ScalarArrowUDFTestsMixin:
assert isinstance(v, pa.Array)
v = pa.array([bytes([12, i.as_py()]) for i in v])
m = pa.array([bytes([1, 0, 0]) for i in v])
- yield pa.StructArray.from_arrays([v, m], type=variant_type)
+ yield pa.StructArray.from_arrays([v, m], fields=fields)
df = self.spark.range(0, 10)
expected = [Row(l=i) for i in range(10)]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
