This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 8c087f1a1ecc [SPARK-54065][PYTHON][TESTS] Enable
`test_in_memory_data_source` in Python 3.14
8c087f1a1ecc is described below
commit 8c087f1a1ecc1749cbab4300882f15ab0b5e7744
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Nov 9 23:11:35 2025 -0800
[SPARK-54065][PYTHON][TESTS] Enable `test_in_memory_data_source` in Python
3.14
### What changes were proposed in this pull request?
This PR aims to enable `test_in_memory_data_source` in Python 3.14.
### Why are the changes needed?
After upgrading to the latest `cloudpickle` 3.1.2, I verified that the test
passed on Python 3.14.0.
- #52964
```
$ python/run-tests --parallelism 1 --testnames
pyspark.sql.tests.test_python_datasource --python-executables python3
Running PySpark tests. Output is in
/Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python tests:
['pyspark.sql.tests.test_python_datasource']
python3 python_implementation is CPython
python3 version is: Python 3.14.0
Starting test(python3): pyspark.sql.tests.test_python_datasource (temp
output:
/Users/dongjoon/APACHE/spark-merge/python/target/e771105c-5304-4f4f-bea6-b8d896b1b250/python3__pyspark.sql.tests.test_python_datasource__2wyrv5n4.log)
Finished test(python3): pyspark.sql.tests.test_python_datasource (29s)
Tests passed in 29 seconds
```
### Does this PR introduce _any_ user-facing change?
No. This is a test change.
### How was this patch tested?
Pass the CIs and manual tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52967 from dongjoon-hyun/SPARK-54065.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 79d0c427878f3fc9a95375679934ce6d59595014)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/sql/tests/test_python_datasource.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/python/pyspark/sql/tests/test_python_datasource.py
b/python/pyspark/sql/tests/test_python_datasource.py
index cfedf1cf075b..ee2d0ac60d90 100644
--- a/python/pyspark/sql/tests/test_python_datasource.py
+++ b/python/pyspark/sql/tests/test_python_datasource.py
@@ -258,7 +258,6 @@ class BasePythonDataSourceTestsMixin:
with self.assertRaisesRegex(PythonException,
"DATA_SOURCE_INVALID_RETURN_TYPE"):
df.collect()
- @unittest.skipIf(sys.version_info > (3, 13), "SPARK-54065")
def test_in_memory_data_source(self):
class InMemDataSourceReader(DataSourceReader):
DEFAULT_NUM_PARTITIONS: int = 3
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
