[issue28053] parameterize what serialization is used in multiprocessing

[Pierre Glaser]

Change by Pierre Glaser :


--
pull_requests: +14807
pull_request: https://github.com/python/cpython/pull/15058

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +9306

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Igor]

Change by Igor :


--
nosy: +i3v

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I'd like to know if the work here will be completed soon :-) This currently 
lacks documentation but also tests.

In particular, looking at the code (and that is supported by Will S' comment 
above), I feel the API isn't working as intended (i.e. setting the "reducer" 
property won't actually change the underlying parameters since the "reduction" 
module is imported eagerly).

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Will S]

Will S added the comment:

Just to follow up in case anyone comes across my last message later:

I just had to change the last line from

multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

to

multiprocessing.context._default_context.reducer = 
pickle2reducer.Pickle2Reducer()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Will S]

Will S added the comment:

Documentation would be appreciated. I have a project that uses BaseManager, 
Client, and Listener to create some servers and clients. I would like to update 
the project to work with Python 3 and would prefer to update the clients and 
the servers separately (i.e. switch the client to Python 3 while the server is 
run with Python 2.7). However, BaseManager uses connection.Client which uses 
connection._ConnectionBase which uses reduction.ForkingPickler without a 
protocol argument. It seems the default protocol is 3 on Python 3.6 and 2 on 
Python 2.7 (contrary to the comment above about v2 being used). I just want to 
set the protocol version to 2 in Python 3.6. Can I do that with the changes 
added by this patch?

I tried creating pickle2reducer.py like this:

from multiprocessing.reduction import ForkingPickler, AbstractReducer

class ForkingPickler2(ForkingPickler):
def __init__(self, *args):
if len(args) > 1:
args[1] = 2
else:
args.append(2)
super().__init__(*args)

@classmethod
def dumps(cls, obj, protocol=2):
return ForkingPickler.dumps(obj, protocol)


def dump(obj, file, protocol=2):
ForkingPickler2(file, protocol).dump(obj)


class Pickle2Reducer(AbstractReducer):
ForkingPickler = ForkingPickler2
register = ForkingPickler2.register
dump = dump

and then putting

import pickle2reducer
multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

at the top of my module before

import multiprocessing.connection

but I still see "ValueError: unsupported pickle protocol: 3" on the server when 
I connect with a Python 3.6 client.

--
nosy: +Will S

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Davin Potts]

Davin Potts added the comment:

Docs need updating still.

--
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7381b1b50e00 by Davin Potts in branch 'default':
Issue #28053: Applying refactorings, docs and other cleanup to follow.
https://hg.python.org/cpython/rev/7381b1b50e00

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Eric Snow]

Eric Snow added the comment:

LGTM

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Davin Potts]

Davin Potts added the comment:

Attaching patch containing refactorizations but missing update to docs for the 
purposes of review.

Introduces three new things:
* a function to get the current serializer/reducer
* a function to set the serializer/reducer
* an abstract base class to assist others in rolling their own reducers

--
keywords: +patch
nosy: +eric.snow
stage:  -> patch review
Added file: http://bugs.python.org/file44511/issue_28053_missingdocs.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Davin Potts]

Changes by Davin Potts :


--
components: +Library (Lib)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28053] parameterize what serialization is used in multiprocessing

[Davin Potts]

New submission from Davin Potts:

Currently multiprocessing uses the pickle module for its serialization of 
objects to be communicated between processes.  Specifically v2 of the pickle 
protocols is now exclusively used to provide maximum compatibility, motivated 
by the desire for multiple versions of Python to be used simultaneously with 
multiprocessing.

Per conversations in issue26507, issue23403, and others, multiprocessing should 
offer the option to specify what serialization is to be used for the transport 
of data between processes.  Besides supporting requests to use a different 
version of the pickle protocol or using 3rd party tools like dill, a hook to 
specify the means for reducing objects to a transmittable form opens a door for 
other creative or higher performance strategies.

Ultimately, this is not an enhancement to add functionality but rather to 
reorganize the existing internals of multiprocessing to permit better control 
over its use of serialization.

--
assignee: davin
messages: 275437
nosy: davin
priority: normal
severity: normal
status: open
title: parameterize what serialization is used in multiprocessing
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com