Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19085#discussion_r136081750
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -35,6 +35,16 @@ import org.apache.spark.rdd.RDD
/** Utilities for serialization / deserialization between Python and Java,
using Pickle. */
private[spark] object SerDeUtil extends Logging {
+ class ByteArrayConstructor extends
net.razorvine.pickle.objects.ByteArrayConstructor {
+ override def construct(args: Array[Object]): Object = {
+ // Deal with an empty byte array pickled by Python 3.
+ if (args.length == 0) {
--- End diff --
I see. It looks quite straightforward. I checked in Python 3:
```
>>> import pickle
>>> import pickletools
>>> print(pickletools.dis(pickle.dumps(bytearray())))
0: \x80 PROTO 3
2: c GLOBAL 'builtins bytearray'
22: q BINPUT 0
24: ) EMPTY_TUPLE
25: R REDUCE
26: q BINPUT 1
28: . STOP
```
which, up to my knowledge, gives new object[0] for `args`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
