[
https://issues.apache.org/jira/browse/SPARK-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davies Liu resolved SPARK-7891.
-------------------------------
Resolution: Duplicate
Assignee: Shixiong Zhu
Fix Version/s: 2.0.0
> Python class in __main__ may trigger AssertionError
> ---------------------------------------------------
>
> Key: SPARK-7891
> URL: https://issues.apache.org/jira/browse/SPARK-7891
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.2.0
> Environment: Linux, Python 2.7.3
> pickled by Python pickle Lib
> Reporter: flykobe cheng
> Assignee: Shixiong Zhu
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: demo_error.log, demo_pickle_error.py
>
>
> Callback functions for spark transformations and actions will be pickled.
> If the callback is instancemethod of __main__ module's class, and the class
> has more than one instancemethod which using class properties or
> classmethods, the class will be pickled twice, and 'pickle.memoize' twice,
> then trigger AssertionError.
> Demo code and traceback attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
