HyukjinKwon commented on pull request #35391:
URL: https://github.com/apache/spark/pull/35391#issuecomment-1075922395
@LucaCanali thanks for addressing my comment. The test seems like failing:
```
======================================================================
ERROR [0.182s]: test_pandas_array_struct
(pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/__w/spark/spark/python/pyspark/sql/tests/test_pandas_udf_scalar.py", line
152, in test_pandas_array_struct
result = df.select(return_cols("array_struct_col"))
File "/__w/spark/spark/python/pyspark/sql/udf.py", line 276, in wrapper
return self(*args)
File "/__w/spark/spark/python/pyspark/sql/udf.py", line 249, in __call__
judf = self._judf
File "/__w/spark/spark/python/pyspark/sql/udf.py", line 215, in _judf
self._judf_placeholder = self._create_judf(self.func)
File "/__w/spark/spark/python/pyspark/sql/udf.py", line 224, in
_create_judf
wrapped_func = _wrap_function(sc, func, self.returnType)
File "/__w/spark/spark/python/pyspark/sql/udf.py", line 50, in
_wrap_function
pickled_command, broadcast_vars, env, includes =
_prepare_for_python_RDD(sc, command)
File "/__w/spark/spark/python/pyspark/rdd.py", line 3350, in
_prepare_for_python_RDD
pickled_command = ser.dumps(command)
File "/__w/spark/spark/python/pyspark/serializers.py", line 458, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 73, in dumps
cp.dump(obj)
File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 602, in dump
return Pickler.dump(self, obj)
File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py",
line 316, in _file_reduce
raise pickle.PicklingError(
_pickle.PicklingError: Cannot pickle files that are not opened for reading: w
```
Could we fix this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
