[
https://issues.apache.org/jira/browse/ARROW-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264650#comment-16264650
]
ASF GitHub Bot commented on ARROW-1758:
---------------------------------------
pcmoritz commented on a change in pull request #1347: ARROW-1758: [Python]
Remove pickle=True option for object serialization
URL: https://github.com/apache/arrow/pull/1347#discussion_r152853811
##########
File path: python/pyarrow/serialization.py
##########
@@ -67,9 +68,11 @@ def _deserialize_default_dict(data):
serialization_context.register_type(
type(lambda: 0), "function",
- pickle=True)
+ custom_serializer=pickle.dumps, custom_deserializer=pickle.loads)
- serialization_context.register_type(type, "type", pickle=True)
+ serialization_context.register_type(type, "type",
+ custom_serializer=pickle.dumps,
+ custom_deserializer=pickle.loads)
Review comment:
In Ray, lambdas are handled with cloudpickle as are builtin types like
"type"; most user defined types are handled via __dict__; namedtuples are
special cased.
One reasonable way to handle this is to provide default callbacks for types
we encounter that don't work with __dict__ (using cloudpickle or a custom
serializer/deserializer), there are already examples of this in the repo, and
use __dict__ for anything else. That worked well for us so far. If this
strategy doesn't work for the user, they have full flexibility by providing
their own serialization context.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Python] Remove pickle=True option for object serialization
> -----------------------------------------------------------
>
> Key: ARROW-1758
> URL: https://issues.apache.org/jira/browse/ARROW-1758
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Philipp Moritz
> Assignee: Licht Takeuchi
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> As pointed out in
> https://github.com/apache/arrow/pull/1272#issuecomment-340738439, we don't
> really need this option, it can already be done with pickle.dumps as the
> custom serializer and pickle.loads as the deserializer.
> This has the additional benefit that it will be very clear to the user which
> pickler will be used and the user can use a custom pickler easily.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
