[
https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang-Chi Hsieh updated SPARK-27629:
------------------------------------
Description:
In SPARK-27612, one correctness issue was reported. When protocol 4 is used to
pickle Python objects, we found that unpickled objects were wrong. A temporary
fix was proposed by not using highest protocol.
It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4. It
is suspect to this issue.
A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of
Unpickler object. We use single Unpickler object to unpickle serialized Python
bytes. Stored objects intervenes next round of unpickling, if the map is not
cleared.
We has two options:
1. Continues to reuse Unpickler, but calls its close after each unpickling.
2. Not to reuse Unpickler and create new Unpickler object in each unpickling.
Note: This issue is because internal object map in Pyrolite is not cleared
after op code STOP. If we use protocol 4 to pickle Python objects, op code
MEMOIZE will store objects in the map. We need to clear up it to make sure next
unpickling works on clear map. For now, we can manually clear the map.
was:
In SPARK-27612, one correctness issue was reported. When protocol 4 is used to
pickle Python objects, we found that unpickled objects were wrong. A temporary
fix was proposed by not using highest protocol.
It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4. It
is suspect to this issue.
A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of
Unpickler object. We use single Unpickler object to unpickle serialized Python
bytes. Stored objects intervenes next round of unpickling, if the map is not
cleared.
We has two options:
1. Continues to reuse Unpickler, but calls its close after each unpickling.
2. Not to reuse Unpickler and create new Unpickler object in each unpickling.
Note: This issue is because internal object map in Pyrolite is not cleared
after op code STOP. If we use protocol 4 to pickle Python objects, op code
MEMOIZE will store objects in the map. We need to clear up it to make sure next
unpickling works on clear map.
> Prevent Unpickler from intervening each unpickling
> --------------------------------------------------
>
> Key: SPARK-27629
> URL: https://issues.apache.org/jira/browse/SPARK-27629
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.0.0
> Reporter: Liang-Chi Hsieh
> Priority: Major
>
> In SPARK-27612, one correctness issue was reported. When protocol 4 is used
> to pickle Python objects, we found that unpickled objects were wrong. A
> temporary fix was proposed by not using highest protocol.
> It was found that Opcodes.MEMOIZE was appeared in the opcodes in protocol 4.
> It is suspect to this issue.
> A deeper dive found that Opcodes.MEMOIZE stores objects into internal map of
> Unpickler object. We use single Unpickler object to unpickle serialized
> Python bytes. Stored objects intervenes next round of unpickling, if the map
> is not cleared.
> We has two options:
> 1. Continues to reuse Unpickler, but calls its close after each unpickling.
> 2. Not to reuse Unpickler and create new Unpickler object in each unpickling.
> Note: This issue is because internal object map in Pyrolite is not cleared
> after op code STOP. If we use protocol 4 to pickle Python objects, op code
> MEMOIZE will store objects in the map. We need to clear up it to make sure
> next unpickling works on clear map. For now, we can manually clear the map.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
