[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-20 Thread Tim Peters
Sorry for the spam! A bunch of these were backed up in the moderation queue. I used the UI to set the list to auto-discard future messages from this address, but then clicked "Accept" in the mistaken sense of "yes, accept my request to auto-nuke this clown". But it took "Accept" to mean "sure

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-20 Thread joeevansjoe6
that is great to see this post . https://bit.ly/3C551OO ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Facundo Batista
El lun, 11 de oct. de 2021 a la(s) 06:50, Simon Cross (hodgestar+python...@gmail.com) escribió: > Multiprocessing sort of added support for this via multiprocessing.Array -- > see >

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Facundo Batista
El dom, 10 de oct. de 2021 a la(s) 15:20, Gregory P. Smith (g...@krypto.org) escribió: >> Is that because my particular case is very uncommon? Or maybe we *do* >> want this but we don't have it yet? Or do we already have a better way >> of doing this? >> >> Thanks! >> >> [0]

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Simon Cross
On Sun, Oct 10, 2021 at 4:23 PM Facundo Batista wrote: > > struct.pack_into(format, buffer, offset, v1, v2, ...) I've encountered this wart with pack and pack_into too. The current interface makes sense when if v1, v2 are a small number of items from a data record, but it becomes a bit

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Antoine Pitrou
On Sun, 10 Oct 2021 11:19:44 -0300 Facundo Batista wrote: > Hello everyone! > > I need to pack a long list of numbers into shared memory, so I thought > about using `struct.pack_into`. > > Its signature is > > struct.pack_into(format, buffer, offset, v1, v2, ...) > > I have a long list of

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Serhiy Storchaka
11.10.21 01:35, MRAB пише: > Maybe what's needed is to add, say, '*' to the format string to indicate > that multiple values should come from an iterable, e.g.: > >     struct.pack_into(f'{len(nums)}*Q', buf, 0, nums) > > in this case len(nums) from the nums argument. See

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread MRAB
On 2021-10-10 19:20, Gregory P. Smith wrote: On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista mailto:facundobati...@gmail.com>> wrote: Hello everyone! I need to pack a long list of numbers into shared memory, so I thought about using `struct.pack_into`. Its signature is

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Serhiy Storchaka
10.10.21 19:05, Patrick Reader пише: > This still isn't a completely direct write - you're still creating an array > in between which is then copied into shm, whereas struct.pack_into writes > directly into the shared memory with no intermediate buffer. > > And unfortunately you can't do > >

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Gregory P. Smith
On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista wrote: > Hello everyone! > > I need to pack a long list of numbers into shared memory, so I thought > about using `struct.pack_into`. > > Its signature is > > struct.pack_into(format, buffer, offset, v1, v2, ...) > > I have a long list of nums

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Patrick Reader
You can take a memory view of the array directly: memoryview(array.array("Q", range(1000))) If your exact use-case is writing to a SharedMemory, then I don't think there is any simple way to do it without creating some intermediate memory buffer. (other than using struct.pack_into, or packing

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Patrick Reader
This still isn't a completely direct write - you're still creating an array in between which is then copied into shm, whereas struct.pack_into writes directly into the shared memory with no intermediate buffer. And unfortunately you can't do shm.buf[l_offset:r_offset].cast('Q')[:] = nums

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Serhiy Storchaka
10.10.21 18:18, Facundo Batista пише: > You mean `array` from the `array` module? The only way I see using it > is like the following: > shm = shared_memory.SharedMemory(create=True, size=total_size) a = array.array('Q', nums) shm.buf[l_offset:r_offset] = a.tobytes() > > But I

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Guido van Rossum
Maybe instead of tobytes() you can use memoryview(). On Sun, Oct 10, 2021 at 08:21 Facundo Batista wrote: > El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka > (storch...@gmail.com) escribió: > > > > 10.10.21 17:19, Facundo Batista пише: > > > I have a long list of nums (several

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Facundo Batista
El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka (storch...@gmail.com) escribió: > > 10.10.21 17:19, Facundo Batista пише: > > I have a long list of nums (several millions), ended up doing the following: > > > > struct.pack_into(f'{len(nums)}Q', buf, 0, *nums) > > Why not use

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-10 Thread Serhiy Storchaka
10.10.21 17:19, Facundo Batista пише: > I have a long list of nums (several millions), ended up doing the following: > > struct.pack_into(f'{len(nums)}Q', buf, 0, *nums) Why not use array('Q', nums)? ___ Python-Dev mailing list --