This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 9c3d0dd3d0b7 [SPARK-54486][PYTHON][TESTS][4.0] Backport the pyarrow
version check of `test_to_feather`
9c3d0dd3d0b7 is described below
commit 9c3d0dd3d0b718e45446afe3579d855bdbde2930
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Nov 26 15:09:49 2025 +0900
[SPARK-54486][PYTHON][TESTS][4.0] Backport the pyarrow version check of
`test_to_feather`
### What changes were proposed in this pull request?
Backport the pyarrow version check of `test_to_feather`
### Why are the changes needed?
to resolve test failure in master-server <> branch-4.0-client
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
will monitor the scheduled job
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #53228 from zhengruifeng/skip_test_to_feather.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/pandas/tests/io/test_feather.py | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/pandas/tests/io/test_feather.py
b/python/pyspark/pandas/tests/io/test_feather.py
index e4af29305b73..10638d915c0e 100644
--- a/python/pyspark/pandas/tests/io/test_feather.py
+++ b/python/pyspark/pandas/tests/io/test_feather.py
@@ -17,9 +17,10 @@
import unittest
import pandas as pd
-import os
+import sys
from pyspark import pandas as ps
+from pyspark.loose_version import LooseVersion
from pyspark.testing.pandasutils import PandasOnSparkTestCase, TestUtils
@@ -35,9 +36,16 @@ class FeatherMixin:
def psdf(self):
return ps.from_pandas(self.pdf)
- @unittest.skipIf(
- os.environ.get("SPARK_SKIP_CONNECT_COMPAT_TESTS") == "1",
"SPARK-54486: To be reenabled"
- )
+ has_arrow_21_or_below = False
+ try:
+ import pyarrow as pa
+
+ if LooseVersion(pa.__version__) < LooseVersion("22.0.0"):
+ has_arrow_21_or_below = True
+ except ImportError:
+ pass
+
+ @unittest.skipIf(not has_arrow_21_or_below, "SPARK-54068")
def test_to_feather(self):
with self.temp_dir() as dirpath:
path1 = f"{dirpath}/file1.feather"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
