Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/3091#discussion_r19853296
--- Diff: python/pyspark/mllib/common.py ---
@@ -98,8 +98,13 @@ def _java2py(sc, r):
jrdd = sc._jvm.SerDe.javaToPython(r)
return RDD(jrdd, sc, AutoBatchedSerializer(PickleSerializer()))
- elif isinstance(r, (JavaArray, JavaList)) or clsName in
_picklable_classes:
+ if clsName in _picklable_classes:
r = sc._jvm.SerDe.dumps(r)
+ elif isinstance(r, (JavaArray, JavaList)):
+ try:
+ r = sc._jvm.SerDe.dumps(r)
+ except Py4JJavaError:
+ pass # not pickable
--- End diff --
The caller will handle it. The JavaArray/JavaList is iterable in Python,
caller can access the internal objects in this array/list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
