Liang-Chi Hsieh created SPARK-27629:
---------------------------------------
Summary: Prevent Unpickler from intervening each unpickling
Key: SPARK-27629
URL: https://issues.apache.org/jira/browse/SPARK-27629
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 3.0.0
Reporter: Liang-Chi Hsieh
In SPARK-27612, one correctness issue was reported. When protocol 4 is used to
pickle Python objects, we found that unpickled objects were wrong. A temporary
fix was proposed by not using highest protocol.
It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4. It
is suspect to this issue.
A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of
Unpickler object. We use single Unpickler object to unpickle serialized Python
bytes. Stored objects intervenes next round of unpickling, if the map is not
cleared.
We has two options:
1. Continues to reuse Unpickler, but calls its close after each unpickling.
2. Not to reuse Unpickler and create new Unpickler object in each unpickling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
