[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Closed as a duplicate of issue17560.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Justin Ting]

Justin Ting added the comment:

Actually, on further inspection, I seem to be having a slightly different 
problem with the same error that I initially described now.

Even after modifying my code so that each python forked off to another process 
was only given the following arguments:
args = [(None, models_shape, False, None, [start, end], 
'data/qp_red_features.npy') for start, end in jobs] 

where models_shape, start, and end are only single integers, the same error 
still comes up as a result. Within each process, I'm reading in a (relatively 
small, only 12MB) .npy ndarray and taking the [start:end] slice.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Justin Ting]

Justin Ting added the comment:

Ah, should have picked that up, coding at 3:30am doesn't do wonders for
keeping a clear head.

Thanks Tim, I'll keep that in mind!

*Justin Ting*
*E* justingl...@gmail.com |  *M* +61 424 751 665 | *L*
*https://au.linkedin.com/in/justinyting
* | *G *https://github.com/jyting

On Sun, Oct 23, 2016 at 3:48 AM, Tim Peters  wrote:

>
> Tim Peters added the comment:
>
> This has nothing to do with the _values_ you're passing - it has to do
> with the length of the pickle string:
>
> def _send_bytes(self, buf):
> n = len(buf)
> # For wire compatibility with 3.2 and lower
> header = struct.pack("!i", n)  IT'S BLOWING UP HERE
> if n > 16384:
> ...
> self._send(header)
> self._send(buf)
>
> where the traceback shows it's called here:
>
> self._send_bytes(ForkingPickler.dumps(obj))
>
> Of course the less data you're passing, the smaller the pickle, and that's
> why it doesn't blow up if you pass subsets of the data.
>
> I'd suggest rethinking how you're sharing data, as pushing two-gigabyte
> pickle strings around is bound to be the least efficient way possible even
> if it didn't blow up ;-)
>
> --
> nosy: +tim.peters
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This looks as a duplicate of issue17560.

--
nosy: +serhiy.storchaka
resolution:  -> duplicate
superseder:  -> problem using multiprocessing with really big objects?

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Tim Peters]

Tim Peters added the comment:

This has nothing to do with the _values_ you're passing - it has to do with the 
length of the pickle string:

def _send_bytes(self, buf):
n = len(buf)
# For wire compatibility with 3.2 and lower
header = struct.pack("!i", n)  IT'S BLOWING UP HERE
if n > 16384:
...
self._send(header)
self._send(buf)

where the traceback shows it's called here:

self._send_bytes(ForkingPickler.dumps(obj))

Of course the less data you're passing, the smaller the pickle, and that's why 
it doesn't blow up if you pass subsets of the data.

I'd suggest rethinking how you're sharing data, as pushing two-gigabyte pickle 
strings around is bound to be the least efficient way possible even if it 
didn't blow up ;-)

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Justin Ting]

New submission from Justin Ting:

Multiprocessing is throwing this error when dealing with large amounts of data 
(all floating points an integers), but none of which exceeds the number 
boundaries in the error that it throws:

  File "/root/anaconda3/lib/python3.5/multiprocessing/pool.py", line 268, in 
starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/root/anaconda3/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
  File "/root/anaconda3/lib/python3.5/multiprocessing/pool.py", line 385, in 
_handle_tasks
put(task)
  File "/root/anaconda3/lib/python3.5/multiprocessing/connection.py", line 206, 
in send
self._send_bytes(ForkingPickler.dumps(obj))
  File "/root/anaconda3/lib/python3.5/multiprocessing/connection.py", line 393, 
in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
> /root/anaconda3/lib/python3.5/multiprocessing/connection.py(393)_send_bytes()
-> header = struct.pack("!i", n)

It works fine on any number of subsets of this data, but not when put together.

--
components: Library (Lib)
messages: 279200
nosy: Justin Ting
priority: normal
severity: normal
status: open
title: Multiprocessing Pool starmap - struct.error: 'i' format requires 
-2e10<=n<=2e10
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com