Repository: spark Updated Branches: refs/heads/master 8e7d5ba1a -> 9fd82dbbc
[SPARK-1687] [PySpark] fix unit tests related to pickable namedtuple serializer is imported multiple times during doctests, so it's better to make _hijack_namedtuple() safe to be called multiple times. Author: Davies Liu <[email protected]> Closes #1771 from davies/fix and squashes the following commits: 1a9e336 [Davies Liu] fix unit tests Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9fd82dbb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9fd82dbb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9fd82dbb Branch: refs/heads/master Commit: 9fd82dbbcb8b10debbe95f1acab53ae8b340f38e Parents: 8e7d5ba Author: Davies Liu <[email protected]> Authored: Mon Aug 4 15:54:52 2014 -0700 Committer: Josh Rosen <[email protected]> Committed: Mon Aug 4 15:54:52 2014 -0700 ---------------------------------------------------------------------- python/pyspark/serializers.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/9fd82dbb/python/pyspark/serializers.py ---------------------------------------------------------------------- diff --git a/python/pyspark/serializers.py b/python/pyspark/serializers.py index 1b52c14..a10f85b 100644 --- a/python/pyspark/serializers.py +++ b/python/pyspark/serializers.py @@ -297,8 +297,11 @@ def _hack_namedtuple(cls): def _hijack_namedtuple(): """ Hack namedtuple() to make it picklable """ - global _old_namedtuple # or it will put in closure + # hijack only one time + if hasattr(collections.namedtuple, "__hijack"): + return + global _old_namedtuple # or it will put in closure def _copy_func(f): return types.FunctionType(f.func_code, f.func_globals, f.func_name, f.func_defaults, f.func_closure) @@ -313,6 +316,7 @@ def _hijack_namedtuple(): collections.namedtuple.func_globals["_old_namedtuple"] = _old_namedtuple collections.namedtuple.func_globals["_hack_namedtuple"] = _hack_namedtuple collections.namedtuple.func_code = namedtuple.func_code + collections.namedtuple.__hijack = 1 # hack the cls already generated by namedtuple # those created in other module can be pickled as normal, --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
