[Python-ideas] Re: Default behavior for random.sample when no k

[raymond . hettinger]

Steven D'Aprano wrote:
> > This is easily solved with a three-line helper:
> def shuffled(iterable):
 ...
> I have implemented this probably a half a dozen times, and I expect 
> others have too.

FWIW, we've already documented a clean way to do it, 
https://docs.python.org/3/library/random.html#random.shuffle , "To shuffle an 
immutable sequence and return a new shuffled list, use sample(x, k=len(x)) 
instead."

>>> data = 'random module'
>>> ''.join(sample(data, len(data)))
'uaemdor odmln'

Given that we already have shuffle() and sample(), I really don't think we need 
a third way to it.  How about we save API extensions for ideas that add genuine 
new, useful capabilities.

Raymond
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VVKZU6ABPBYZORXMURCIHBZZRNRREMIS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Filtered wildcard imports?

[Steven D'Aprano]

On Sun, Aug 02, 2020 at 03:36:44PM +0200, Sebastian M. Ernst wrote:
> Hi all,
> 
> yet another (possibly bad?) idea from day-to-day work ...
> 
> I occasionally need to import a lot of "stuff" from certain modules. The
> "stuff" is usually following a pattern. E.g. I have modules that
> (mostly) collect special exception classes and I usually need them all
> in one push - but nothing else from those modules.

When I have a similar requirement, I do something like this:

stuff_to_import = ['SomeError', 'AnotherError']
exec("from somemodule import %s" % ', '.join(stuff_to_import))
del stuff_to_import

(by memory, so the details may not be exactly correct).

Another possibility is to offer the stuff to import in the library 
module:

# somemodule.py
errors = ['SomeError', 'AnotherError']

# caller
from somemodule import errors as errors
exec("from somemodule import %s" % ', '.join(errors))
del errors


The list of stuff to import can even be generated programmatically, by 
name or functionality:

# the very end of somemodule.py
errors = [name for name, obj in globals().items() 
  if isinstance(obj, type) and issubclass(obj, Exception)]


Another option:

# Caller
import somemodule
globals().update({name: obj for name, obj in vars(somemodule).items()
  if isinstance(obj, type) and issubclass(obj, Exception)})


The bottom line here is that we can be far more flexible about filtered 
imports than mere wild cards in the names, and in a lot fewer lines of 
code than you've been using.



-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/V6J43IRZWYDM2OWOZYN3M3B2W4U62OEQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Wes Turner]

How is this a different problem than the cache coherency problem?
https://en.wikipedia.org/wiki/Cache_coherence

Perhaps that's an unhelpful abstraction?

This hasn't gone anywhere:
https://en.wikipedia.org/wiki/Distributed_shared_memory#Directory_memory_coherence

Here's a great comparison chart for message passing vs distributed shared
memory:
https://en.wikipedia.org/wiki/Distributed_shared_memory#Message_Passing_vs._DSM

Could there be a multiprocessing.MemoryPool that tracks allocations,
refcounts, and also locks?

A combined approach might have an IPC channel/stream/source/sinks for
messages that instruct workers to invalidate/re-fetch object_id/references,
but consistency and ordering result in the same issues encountered with the
cache coherence problem.

Then, what is the best way to enqueue changes to shared global state (in
shared memory on one node, in this instance)?

(... "Ask HN: Learning about distributed systems?"
https://news.ycombinator.com/item?id=23931730 )

A solution for this could help accelerate dask and dask.distributed (which
already address many parallel issues in multiprocess and distributed
systems in pure python)
"Accelerate intra-node IPC with shared memory"
https://github.com/dask/dask/issues/6267



On Sun, Aug 2, 2020, 3:11 PM Eric V. Smith  wrote:

>
> On 8/2/2020 12:20 PM, Eric V. Smith wrote:
>
>
> On Sat, 1 Aug 2020 at 22:42, Eric V. Smith  wrote:
>
>> While they're immutable at the Python level, strings (and all other
>> objects) are mutated at the C level, due to reference count updates. You
>> need to consider this if you're sharing objects without locking or other
>> synchronization.
>>
>>
> This is interesting. What if you want to have a language that uses only
> immutable objects and garbage collection? Could smart pointers address this
> problem?
>
> Yes, garbage collection changes the picture entirely, with or without
> immutable objects. But the original topic was cross-processs shared memory,
> and I don't know of any cross-process aware garbage collectors that support
> shared memory. Although such a thing could easily exist without my
> knowledge.
>
> Note that I'm talking about putting Python objects into this shared
> memory. If that's not what people are contemplating, then my observations
> don't apply.
>
> Eric
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/KT7URLYEMMHTSB7NIBLF3667OYOCF67I/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AAQ5PIGMKSUAGRDVIZ5H2ABGFXGHQ6O6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Eric V. Smith]

On 8/2/2020 12:20 PM, Eric V. Smith wrote:


On Sat, 1 Aug 2020 at 22:42, Eric V. Smith > wrote:


While they're immutable at the Python level, strings (and all other
objects) are mutated at the C level, due to reference count
updates. You
need to consider this if you're sharing objects without locking
or other
synchronization.


This is interesting. What if you want to have a language that uses 
only immutable objects and garbage collection? Could smart pointers 
address this problem?


Yes, garbage collection changes the picture entirely, with or without 
immutable objects. But the original topic was cross-processs shared 
memory, and I don't know of any cross-process aware garbage collectors 
that support shared memory. Although such a thing could easily exist 
without my knowledge.


Note that I'm talking about putting Python objects into this shared 
memory. If that's not what people are contemplating, then my 
observations don't apply.


Eric

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KT7URLYEMMHTSB7NIBLF3667OYOCF67I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Eric V. Smith]

On Sat, 1 Aug 2020 at 22:42, Eric V. Smith > wrote:


While they're immutable at the Python level, strings (and all other
objects) are mutated at the C level, due to reference count
updates. You
need to consider this if you're sharing objects without locking or
other
synchronization.


This is interesting. What if you want to have a language that uses 
only immutable objects and garbage collection? Could smart pointers 
address this problem?


Yes, garbage collection changes the picture entirely, with or without 
immutable objects. But the original topic was cross-processs shared 
memory, and I don't know of any cross-process aware garbage collectors 
that support shared memory. Although such a thing could easily exist 
without my knowledge.


Eric

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5OI66KS2XOHH5Z25DEXZYJPJOKH4TJXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Vinay Sharma via Python-ideas]

sharedctypes can only be used with related processes. There is no way that you 
can pass a sharedctype to an unrelated process. multiprocessing.shared_memory 
was created to handle this i.e. allow usage of shared memory IPC across 
unrelated processes.

> On 02-Aug-2020, at 9:42 PM, Marco Sulla  wrote:
> 
> There's also the possibility to use shared ctypes:
> https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects 
> 
> 
> Operations like += which involve a read and write are not atomic. So if, for 
> instance, you want to atomically increment a shared value it is insufficient 
> to just do
> counter.value += 1
>  
> Assuming the associated lock is recursive (which it is by default)
> you can instead do
>   
> with counter.get_lock():
> counter.value += 1
> 
> Notice that they use a lock anyway. Maybe the solution of Wes Turner is 
> better. See also RLock:
> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock 
> 
> On Sat, 1 Aug 2020 at 22:42, Eric V. Smith  > wrote:
> While they're immutable at the Python level, strings (and all other 
> objects) are mutated at the C level, due to reference count updates. You 
> need to consider this if you're sharing objects without locking or other 
> synchronization.
> 
> 
> This is interesting. What if you want to have a language that uses only 
> immutable objects and garbage collection? Could smart pointers address this 
> problem?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/QVQWSNWJMMMRVFA6BTXDNJHIGAUBPWVX/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FIRY7P7VVW4IOFNPIY3OEYE3V4BISBKE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Marco Sulla]

There's also the possibility to use shared ctypes:
https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects

Operations like += which involve a read and write are not atomic. So if,
> for instance, you want to atomically increment a shared value it is
> insufficient to just do
>
> counter.value += 1
>
> Assuming the associated lock is recursive (which it is by default)
> you can instead do
> with counter.get_lock():
> counter.value += 1
>
> Notice that they use a lock anyway. Maybe the solution of Wes Turner is 
> better. See also RLock:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.RLock

On Sat, 1 Aug 2020 at 22:42, Eric V. Smith  wrote:

> While they're immutable at the Python level, strings (and all other
> objects) are mutated at the C level, due to reference count updates. You
> need to consider this if you're sharing objects without locking or other
> synchronization.
>
>
This is interesting. What if you want to have a language that uses only
immutable objects and garbage collection? Could smart pointers address this
problem?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QVQWSNWJMMMRVFA6BTXDNJHIGAUBPWVX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

[Wes Turner]

https://github.com/python/cpython/blob/master/Objects/odictobject.c :

"""
In order to preserve O(1) performance for node removal (finding nodes), we
must do better than just looping through the linked-list.  Here are options
we've considered:
1. use a second dict to map keys to nodes (a la the pure Python version).
2. keep a simple hash table mirroring the order of dict's, mapping each key
   to the corresponding node in the linked-list.
3. use a version of shared keys (split dict) that allows non-unicode keys.
4. have the value stored for each key be a (value, node) pair, and adjust
   __getitem__(), get(), etc. accordingly.
The approach with the least performance impact (time and space) is #2,
mirroring the key order of dict's dk_entries with an array of node pointers.
While lookdict() and friends (dk_lookup) don't give us the index into the
array, we make use of pointer arithmetic to get that index.  **An
alternative
would be to refactor lookdict() to provide the index, explicitly exposing
the implementation detail.**  We could even just use a custom lookup
function
for OrderedDict that facilitates our need.  However, both approaches are
significantly more complicated than just using pointer arithmetic.
The catch with mirroring the hash table ordering is that we have to keep
the ordering in sync through any dict resizes.  However, that order only
matters during node lookup.  We can simply defer any potential resizing
until we need to do a lookup.
"""

On Sat, Aug 1, 2020 at 10:06 PM Wes Turner  wrote:

> Is there any reason that these features couldn't be added to OrderedDict
> (which is a linked list)?
> https://github.com/python/cpython/blob/master/Objects/odictobject.c
>
> On Sat, Aug 1, 2020, 9:13 PM Inada Naoki  wrote:
>
>> On Sun, Aug 2, 2020 at 2:34 AM Christopher Barker 
>> wrote:
>> >
>> > On Sat, Aug 1, 2020 at 2:28 AM Marco Sulla <
>> marco.sulla.pyt...@gmail.com> wrote:
>> >>
>> >> On Sat, 1 Aug 2020 at 03:00, Inada Naoki 
>> wrote:
>> >>>
>> >>> Please teach me if you know any algorithm which has no hole, O(1)
>> >>> deletion, preserving insertion order, and efficient and fast as array.
>> >
>> >
>> > I would think the goal here would be to re-order once in a while to
>> remove the holes. But that would take time, of course, so you wouldn't want
>> to do it on every deletion. But when?
>> >
>> > One option: maybe too specialized, but it could re-pack the array when
>> an indexing operation is made -- since that operation is O(N) anyway. And
>> that would then address the issue of performance for multiple indexing
>> operations -- if you made a bunch of indexing operation in a row without
>> deleting (which would be the case, if this is an alternative to making a
>> copy in a Sequence first), then the first one would repack the internal
>> array (presumably faster than making a copy) and the rest would have O(1)
>> access.
>> >
>>
>> Repacking is mutation, and mutating dict while iterating it breaks the
>> iterator.
>> But `d.items()[42]` don't looks like mutation.
>>
>> > Given that this use case doesn't appear to be very important, I doubt
>> it's worth it, but it seems it would be possible.
>> >
>> > Another thought -- could the re-packing happen whenever the entire dict
>> is iterated through? Though maybe there's no way to know when that's going
>> to happen -- all you get are the individual calls for the next one, yes?
>> >
>>
>> You are right. it couldn't.
>>
>>
>> --
>> Inada Naoki  
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-ideas@python.org/message/ED2GRWD4RARR2LGP45PK4M6R3MLTAF75/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XUH3AKC5TVPWJBJSI7UGYWTQDCDME2F5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Wes Turner]

It's best to avoid those synchronization barriers if possible.

If you have all of the data in SHM (RAM) on one node, and you need to
notify processes / wait for other workers to be available to perform a task
that requires that data, you need a method for IPC: a queue, channel
subscriptions, a source/sink, over-frequent polling that's more resilient
against dropped messages. (But you only need to scale to one node).

There needs to be a shared structure that tracks allocations, right? What
does it need to do lookups by.

[
[obj_id_or_shm_pointer, [subscribers]]
]

Does the existing memory pool solve for that?

And there also needs to be an instruction pipeline; a queue/channel/source
of messages for each worker or only some workers to process.

...

https://distributed.dask.org/en/latest/journey.html

https://distributed.dask.org/en/latest/work-stealing.html

"Accelerate intra-node IPC with shared memory"
https://github.com/dask/dask/issues/6267


On Sun, Aug 2, 2020, 3:21 AM Vinay Sharma  wrote:

> I understand that I won’t need locks with immutable objects at some level,
> but I don’t understand how they can be used to synchronise shared memory
> segments.
>
> For every change in an immutable object, a copy is created which will have
> a different address. Now, for processes to use this updated object they
> will have to remap a new address in their address space for them to see any
> changes, and this remap will have to occur whenever a change takes place,
> which is obviously not feasible.
>
> So, changes in the shared memory segment should be done in the shared
> memory segment itself, therefore shared memory segments should be mutable.
>
> On 02-Aug-2020, at 5:11 AM, Wes Turner  wrote:
>
>
> https://docs.dask.org/en/latest/shared.html#known-limitations :
>
> > Known Limitations
> > The shared memory scheduler has some notable limitations:
> >
> > - It works on a single machine
> > - The threaded scheduler is limited by the GIL on Python code, so if
> your operations are pure python functions, you should not expect a
> multi-core speedup
> > - The multiprocessing scheduler must serialize functions between
> workers, which can fail
> > - The multiprocessing scheduler must serialize data between workers and
> the central process, which can be expensive
> > - The multiprocessing scheduler cannot transfer data directly between
> worker processes; all data routes through the master process.
>
> ...
> https://distributed.dask.org/en/latest/memory.html#difference-with-dask-compute
>
> (... https://github.com/dask/dask-labextension )
>
> On Sat, Aug 1, 2020 at 7:34 PM Wes Turner  wrote:
>
>> PyArrow Plasma object ids, "sealing" makes an object immutable, pyristent
>>
>> https://arrow.apache.org/docs/python/plasma.html#object-ids
>> https://arrow.apache.org/docs/python/plasma.html#creating-an-object-buffer
>>
>> > Objects are created in Plasma in two stages. First, they are created,
>> which allocates a buffer for the object. At this point, the client can
>> write to the buffer and construct the object within the allocated buffer.
>> >
>> > To create an object for Plasma, you need to create an object ID, as
>> well as give the object’s maximum size in bytes.
>> > ```python
>> > # Create an object buffer.
>> > object_id = plasma.ObjectID(20 * b"a")
>> > object_size = 1000
>> > buffer = memoryview(client.create(object_id, object_size))
>> >
>> > # Write to the buffer.
>> > for i in range(1000):
>> >   buffer[i] = i % 128
>> > ```
>> >
>> > When the client is done, the client seals the buffer, making the object
>> immutable, and making it available to other Plasma clients.
>> >
>> > ```python
>> > # Seal the object. This makes the object immutable and available to
>> other clients.
>> > client.seal(object_id)
>> > ```
>>
>> https://pypi.org/project/pyrsistent/ also supports immutable structures
>>
>> On Sat, Aug 1, 2020 at 4:44 PM Eric V. Smith  wrote:
>>
>>> On 8/1/2020 1:25 PM, Marco Sulla wrote:
>>> > You don't need locks with immutable objects. Since they're immutable,
>>> > any operation that usually will mutate the object, generate another
>>> > immutable instead. The most common example is str: the sum of two
>>> > strings in Python (and in many other languages) produces a new string.
>>>
>>> While they're immutable at the Python level, strings (and all other
>>> objects) are mutated at the C level, due to reference count updates. You
>>>
>>> need to consider this if you're sharing objects without locking or other
>>>
>>> synchronization.
>>>
>>> Eric
>>>
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/FEJEHFKBK7TMH6KIYJBPLBYBDU4IA4EB/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> __

[Python-ideas] Re: Filtered wildcard imports?

[William Pickard]

Why not just import the module they're defined in and then just do attribute 
access syntax?

from mod.modb.dtx import somemod
somemod.SomeError
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2ANGDC6RQ25YT3CAFZVE7XFO3MQSLS7D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

[Marco Sulla]

On Sat, 1 Aug 2020 at 20:25, Stestagg  wrote:

> I wrote some (better than the previously shared) benchmarks for this
> change a while ago.
>

I think that you could speed up the algorithm if you check if
dict->ma_keys->dk_lookup
== lookdict_unicode_nodummy. If so, the dict is a combined dict with only
string keys (quite common), and no deletion was done before, so there's no
hole in ma_keys.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZP3QVG4ZPALST5OXEVKXNHGCRC57J34G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Filtered wildcard imports?

[Sebastian M. Ernst]

Hi all,

yet another (possibly bad?) idea from day-to-day work ...

I occasionally need to import a lot of "stuff" from certain modules. The
"stuff" is usually following a pattern. E.g. I have modules that
(mostly) collect special exception classes and I usually need them all
in one push - but nothing else from those modules.

So a general wildcard import is a bad idea in a lot of ways:

```python
from somemodule import *
```

What I usually do instead is something along the following lines:

```python
import somemodule as _somemodule
_globals = globals()

for _attr in dir(_somemodule): # looking for "stuff"
if not _attr.endswith('Error'): # the filter - this is not "stuff"
continue
_globals[_attr] = getattr(_somemodule, _attr) # the "import"

del _globals, _attr, _somemodule # some cleanup
```

The above selects and "imports" everything ending on "Error" into the
global namespace.

What I would love to do instead is something like a "filtered wildcard
import", maybe through regular expressions or shell-like matching:

```python
from somemodule import *Error
```

Best regards,
Sebastian
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JT27NVYZVXLICUMI6XX57QTVCPL2KGDC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

[Marco Sulla]

On Sat, 1 Aug 2020 at 19:34, Christopher Barker  wrote:

> I would think the goal here would be to re-order once in a while to remove
> the holes. But that would take time, of course, so you wouldn't want to do
> it on every deletion. But when?
>

You should have a separate process that does this, so "normal" operations
will be not blocked. The process could also cache objects and do garbage
collection. A sort of domestic worker. Don't know if this is possible.


> > About the hole, I was thinking that in theory the problem can be
> circumvented using a modified version of lookdict.
>>
>> lookdict searches for a key and returns its position in the ma_keys
>> array. I suppose it's possible to do the contrary: search for the index and
>> return the key.
>> What do you think (theoretically speaking)?
>>
>
> but isn't searching for the index going to require iterating through the
> array until you find it? i.e. that O(N) operation we're trying to avoid?
>

O(n) is an+b. If a and b are small, the algorithm is fast for not-so-big n.

To sum up the alternatives:
1.  rearrange the items array at positional lookup. As Inada said, it will
mutate the dict while reading, that is quite unexpected
2. rearrange the items array at deletion. It will slow down deletion.
Probably not an option... a NAO
3. no positional indexing. It seems the more simple solution. A number of
alternatives are offered (by Tim Peters too)
4. something else.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VPMKJEP4A4SN6QTYWVZRR36AIDTOLSPN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Default behavior for random.sample when no k

[Ram Rachum]

I submitted a patch now, but Serhiy showed me that it's already been
proposed before, and rejected by Raymond Hettinger and Terry Reedy in
issues 26393 and 27964.

On Sun, Aug 2, 2020 at 8:05 AM Steven D'Aprano  wrote:

> On Sat, Aug 01, 2020 at 08:54:16PM +0300, Ram Rachum wrote:
>
> > When writing some code now, I needed to produce a shuffled version of
> > `range(10, 10 ** 5)`.
> >
> > This is one way to do it:
> >
> > shuffled_numbers = list(range(10, 10 ** 5))
> > random.shuffle(shuffled_numbers)
> >
> >
> > I don't like it because (1) it's too imperative and (2) I'm calling the
> > list "shuffled" even before it's shuffled.
>
> This is easily solved with a three-line helper:
>
> def shuffled(iterable):
> L = list(iterable)
> random.shuffle(L)
> return L
>
> I have implemented this probably a half a dozen times, and I expect
> others have too. I agree with Alex that this would make a nice addition
> to the random module.
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/47JMNMYPEETQFKPDK4OVLGM2IXCQ4GIA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QSJGCZLIGPELSSBKX7TJJIT7ZPPV4LXT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

[Vinay Sharma via Python-ideas]

I understand that I won’t need locks with immutable objects at some level, but 
I don’t understand how they can be used to synchronise shared memory segments.

For every change in an immutable object, a copy is created which will have a 
different address. Now, for processes to use this updated object they will have 
to remap a new address in their address space for them to see any changes, and 
this remap will have to occur whenever a change takes place, which is obviously 
not feasible.

So, changes in the shared memory segment should be done in the shared memory 
segment itself, therefore shared memory segments should be mutable.

> On 02-Aug-2020, at 5:11 AM, Wes Turner  wrote:
> 
> 
> https://docs.dask.org/en/latest/shared.html#known-limitations 
>  : 
> 
> > Known Limitations
> > The shared memory scheduler has some notable limitations:
> >
> > - It works on a single machine
> > - The threaded scheduler is limited by the GIL on Python code, so if your 
> > operations are pure python functions, you should not expect a multi-core 
> > speedup
> > - The multiprocessing scheduler must serialize functions between workers, 
> > which can fail
> > - The multiprocessing scheduler must serialize data between workers and the 
> > central process, which can be expensive
> > - The multiprocessing scheduler cannot transfer data directly between 
> > worker processes; all data routes through the master process.
> 
> ... 
> https://distributed.dask.org/en/latest/memory.html#difference-with-dask-compute
>  
> 
> 
> (... https://github.com/dask/dask-labextension 
>  )
> 
> On Sat, Aug 1, 2020 at 7:34 PM Wes Turner  > wrote:
> PyArrow Plasma object ids, "sealing" makes an object immutable, pyristent
> 
> https://arrow.apache.org/docs/python/plasma.html#object-ids 
> 
> https://arrow.apache.org/docs/python/plasma.html#creating-an-object-buffer 
> 
> 
> > Objects are created in Plasma in two stages. First, they are created, which 
> > allocates a buffer for the object. At this point, the client can write to 
> > the buffer and construct the object within the allocated buffer.
> >
> > To create an object for Plasma, you need to create an object ID, as well as 
> > give the object’s maximum size in bytes.
> > ```python
> > # Create an object buffer.
> > object_id = plasma.ObjectID(20 * b"a")
> > object_size = 1000
> > buffer = memoryview(client.create(object_id, object_size))
> >
> > # Write to the buffer.
> > for i in range(1000):
> >   buffer[i] = i % 128
> > ```
> >
> > When the client is done, the client seals the buffer, making the object 
> > immutable, and making it available to other Plasma clients.
> >
> > ```python
> > # Seal the object. This makes the object immutable and available to other 
> > clients.
> > client.seal(object_id)
> > ```
> 
> https://pypi.org/project/pyrsistent/  
> also supports immutable structures
> 
> On Sat, Aug 1, 2020 at 4:44 PM Eric V. Smith  > wrote:
> On 8/1/2020 1:25 PM, Marco Sulla wrote:
> > You don't need locks with immutable objects. Since they're immutable, 
> > any operation that usually will mutate the object, generate another 
> > immutable instead. The most common example is str: the sum of two 
> > strings in Python (and in many other languages) produces a new string.
> 
> While they're immutable at the Python level, strings (and all other 
> objects) are mutated at the C level, due to reference count updates. You 
> need to consider this if you're sharing objects without locking or other 
> synchronization.
> 
> Eric
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org 
> 
> To unsubscribe send an email to python-ideas-le...@python.org 
> 
> https://mail.python.org/mailman3/lists/python-ideas.python.org/ 
> 
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/FEJEHFKBK7TMH6KIYJBPLBYBDU4IA4EB/
>  
> 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/IRDFSJP7CIQ