Re: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it
It seems that even those originally asking for set retrieval have gone silent, so I guess this isn't going anywhere. However, for the benefit of future discussions (because I'm sure this question will be raised again), I'd like to answer a couple of points raised by Stephen. On Sat, 31 Oct 2009 01:42:46 pm Stephen J. Turnbull wrote: If folks prefer a different name, by all means suggest it. I like the name suggested by Wikipedia, pick. Pick has a connotation of removal in many contexts: Pick a card, any card. And in just as many, there is no connotation of removal. Pick a colour. For what it's worth, Forth and similar stack-based languages usually have a function pick equivalent to pop-without-removal. pick seems to be a standard term for this behaviour. Like get, to me it has a flavor of according to some criterion: a band of picked men. I would expect a pick or get operation to take an optional predicate. I think you are underestimating the power of context here. In practice, method names are interpreted in the context of the data being operated on. We don't get confused that ConfigParser.get() has a different signature to dict.get(), which is different again from Queue.get(). list.pop() takes an index; dict.pop() takes a key and an optional second argument; set.pop() takes no argument at all. Is anyone actually confused by this? If not, why would set.get() be more confusing? I think far too many people say this is confusing when what they really mean is I don't like this. The use case I have so far observed people to have in mind is a cast from an equivalence class to a representative member. The difficulty is that we actually have two related, but different, behaviours, and sadly we've conflated the two of them by using the same name get for both. One behaviour is what Wikipedia calls pick: set.pick() - return an arbitrary element from set without removing it This is not useful for the cast you describe. For that, you need a method that takes an argument. I'm going to call it retrieve, to avoid the baggage of get: set.retrieve(x) - return the element from set equal to x In the first case, it seems self-evident to me that having arbitrary mean the same element each time it is called really would condemn pick() to be unused, but I don't care enough to argue. In the second case, the retrieval isn't arbitrary, so this is not a problem that needs solving. No. Since sets are unordered, there's no way to seek to a specific element. I think people will realize that in fact *these* sets are ordered and they will demand a seek function for various practical purposes. We can iterate over dicts, and the order is arbitrary but consistent. Where are the people clamouring for dict.seek()? -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Hello again, Brett Cannon brett at python.org writes: I think it's worth it. Removal of the GIL is a totally open-ended problem with no solution in sight. This, on the other hand, is a performance benefit now. I say move forward with this. If it happens to be short-lived because some actually figures out how to remove the GIL then great, but is that really going to happen between now and Python 3.2? I doubt it. Based on this whole discussion, I think I am going to merge the new GIL work into the py3k branch, with priority requests disabled. If you think this is premature or uncalled for, or if you just want to review the changes before making a judgement, please voice up :) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it
Am Sonntag, 1. November 2009 12:21:15 schrieben Sie: It seems that even those originally asking for set retrieval have gone silent Nope. Stilll following and waiting for the verdict of the community after having filed the corpus delicti [1] wr [1]: http://bugs.python.org/issue7212 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Bizarre mtime behaviour
Hello, I wondered if someone had a clue about the following behaviour. While debugging an erratic test_mailbox failure on RDM's buildbot (and other machines), it turned out that the system sometimes set the wrong mtime on a directory: $ date python -c 'import os; os.link(setup.py, t/c)' stat t date Sun Nov 1 09:49:04 EST 2009 File: `t' Size: 144 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 223152 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 1001/ pitrou) Gid: ( 1005/ pitrou) Access: 2009-11-01 09:10:11.0 -0500 Modify: 2009-11-01 09:49:03.0 -0500 Change: 2009-11-01 09:49:03.0 -0500 Sun Nov 1 09:49:04 EST 2009 As you see above, the mtime for directory 't' is set to a full second before the actual modification has happened. Sprinkling traces of time.time() and os.path.getmtime() on Lib/mailbox.py shows this is exactly what trips up test_mailbox. I've got posted a patch to fix it (see issue #6896), but I would like to know if such OS behaviour is normal. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
On Sun, Nov 1, 2009 at 08:23, Antoine Pitrou solip...@pitrou.net wrote: Hello, I wondered if someone had a clue about the following behaviour. While debugging an erratic test_mailbox failure on RDM's buildbot (and other machines), it turned out that the system sometimes set the wrong mtime on a directory: $ date python -c 'import os; os.link(setup.py, t/c)' stat t date Sun Nov 1 09:49:04 EST 2009 File: `t' Size: 144 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 223152 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 1001/ pitrou) Gid: ( 1005/ pitrou) Access: 2009-11-01 09:10:11.0 -0500 Modify: 2009-11-01 09:49:03.0 -0500 Change: 2009-11-01 09:49:03.0 -0500 Sun Nov 1 09:49:04 EST 2009 As you see above, the mtime for directory 't' is set to a full second before the actual modification has happened. Sprinkling traces of time.time() and os.path.getmtime() on Lib/mailbox.py shows this is exactly what trips up test_mailbox. I've got posted a patch to fix it (see issue #6896), but I would like to know if such OS behaviour is normal. Looks like an OS bug to me. Linux I'm guessing? -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
Adam Olsen rhamph at gmail.com writes: Looks like an OS bug to me. Linux I'm guessing? Yes, but only on certain boxes. I could never reproduce on my home box. RDM (David)'s buildbot is a Gentoo vserver with a reiserfs filesystem. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
Antoine Pitrou wrote: Adam Olsen rhamph at gmail.com writes: Looks like an OS bug to me. Linux I'm guessing? Yes, but only on certain boxes. I could never reproduce on my home box. RDM (David)'s buildbot is a Gentoo vserver with a reiserfs filesystem. You'll occasionally see something similar on Windows boxes running FAT, because the timestamps stored in the filesystem have a 2 second granularity there. Not sure if this might be similar, or just a reiserfs issue. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] EC2 buildslaves
On 31 Oct, 08:13 pm, solip...@pitrou.net wrote: Martin v. L�wis martin at v.loewis.de writes: Not sure whether it's still relevant after the offers of individually donated hardware. We'll see, indeed. However, if you want to look into this, feel free to set up EC2 slaves. I only know to setup mainstream Linux distros though (Debian- or Redhat-lookalikes :-)). I've just played a bit and, after the hassle of juggling with a bunch of different keys and credentials, setting up an instance and saving an image for future use is quite easy. Starting with a mainstream distro doesn't seem like a bad idea. For example, there isn't currently a 32bit Ubuntu (any version) slave. That would be a nice gap to fill in, right? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
On Sun, 2009-11-01 at 12:03 -0500, Eric Smith wrote: Antoine Pitrou wrote: Adam Olsen rhamph at gmail.com writes: Looks like an OS bug to me. Linux I'm guessing? Yes, but only on certain boxes. I could never reproduce on my home box. RDM (David)'s buildbot is a Gentoo vserver with a reiserfs filesystem. You'll occasionally see something similar on Windows boxes running FAT, because the timestamps stored in the filesystem have a 2 second granularity there. Not sure if this might be similar, or just a reiserfs issue. Some years ago there was a bug where the dentry cache held timestamps more granular than those that the filesystem could represent. We saw this cause build failures which were troubling to reproduce :) I haven't seen Linux arbitrarily invent time travel so far:). The FAT rounding issue is a possibility, but I didn't think reiserfs was short that much precision. I'd check that the work area you had really was reiser, not a mounted AT partition, and if its not look up the ReiserFS precision for mtime. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bizarre mtime behaviour
Le lundi 02 novembre 2009 à 08:02 +1100, Robert Collins a écrit : The FAT rounding issue is a possibility, but I didn't think reiserfs was short that much precision. I'd check that the work area you had really was reiser, not a mounted AT partition, and if its not look up the ReiserFS precision for mtime. Yes it was reiserfs. As for the mtime granularity, since some displayed mtimes were even and some were odd, I assume it's one second or less. Another possibility would be that the filesystem would use a slightly different clock than time.time(), but why? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
On Sun, Nov 1, 2009 at 03:33, Antoine Pitrou solip...@pitrou.net wrote: Hello again, Brett Cannon brett at python.org writes: I think it's worth it. Removal of the GIL is a totally open-ended problem with no solution in sight. This, on the other hand, is a performance benefit now. I say move forward with this. If it happens to be short-lived because some actually figures out how to remove the GIL then great, but is that really going to happen between now and Python 3.2? I doubt it. Based on this whole discussion, I think I am going to merge the new GIL work into the py3k branch, with priority requests disabled. This will be a nice Py3K carrot! If you think this is premature or uncalled for, or if you just want to review the changes before making a judgement, please voice up :) I know I personally trust you to not mess it up, Antoine, but that might also come from mental exhaustion and laziness. =) -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nonlocal keyword in 2.x?
2009/10/29 Nick Coghlan ncogh...@gmail.com: Lennart Regebro wrote: 2009/10/28 Antoine Pitrou solip...@pitrou.net: skip at pobox.com writes: So 2.7 support will for the most part be a case not of supporting Python versions, but Python *users*. Antoine That's still not a good reason to backport nonlocal. The same Antoine reasoning could be used to backport new features to the 2.6 Antoine branch after all. No, because 2.6 is in feature freeze (bug fixes only). 2.7 is the current version of 2.x where new features are allowed to be added. That was precisely my point. Then I don't understand what you are saying. Obviously we shouldn't backport to the 2.6 branch, it's in bugfix mode. This is about 2.7. I don't see what 2.6 has to do with it. There are development practices which mitigate the idea that backporting is always helpful to the user. And those are? You said it above yourself: bugfix mode Again: Then I don't understand what you are saying. Obviously we shouldn't backport to the 2.6 branch, it's in bugfix mode. This is about 2.7. I don't see what 2.6 has to do with it. That's all Antoine's point was - backporting of new features to previous branches is not automatically a good idea. Obviously we shouldn't backport to the 2.6 branch, it's in bugfix mode. This is about 2.7. I don't see what 2.6 has to do with it. In the case of 3.2 - 2.7 backports, there are issues with the initial development time investment to do the backport, future double-keying of additional maintenance issues, consideration of possible poor interaction with legacy features in the 2.x series. It's a bunch of additional work that isn't going to happen without someone volunteering to do it. Yes? I feel that you are saying things that are obvious, and that you then expect me to draw a conclusion from this, that you probably find obvious too, but I don't. So I still don't understand what you are saying, or how this in any way contradicts what I said, or clarified or expanded on the matter. As I said: Python 2 support is not only about supporting old versions of Python, but also supporting users of Python2-only modules. So 2.7 support will for the most part be a case not of supporting Python versions, but Python *users*. And contrary to what Antoine said, that *is* a good reason to backport it. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nonlocal keyword in 2.x?
As I said: Python 2 support is not only about supporting old versions of Python, but also supporting users of Python2-only modules. So 2.7 support will for the most part be a case not of supporting Python versions, but Python *users*. And contrary to what Antoine said, that *is* a good reason to backport it. FWIW, I support backporting the nonlocal-keyword in 2.7. All of the reasons for introducting nonlocal to 3.x also apply to 2.x. Using the nonlocal keyword in clear and explicit, especially when compared to the existing workarounds which are not pretty. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Antoine Pitrou wrote: Based on this whole discussion, I think I am going to merge the new GIL work into the py3k branch, with priority requests disabled. If you think this is premature or uncalled for, or if you just want to review the changes before making a judgement, please voice up :) +1 from me. I trust you like Brett does. How much work would it cost to make your patch optional at compile time? For what it's worth we could compare your work on different machines and on different platforms before it gets enabled by default. Can you imagine scenarios where your implementation might be slower than the current GIL implementation? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Christian Heimes lists at cheimes.de writes: +1 from me. I trust you like Brett does. How much work would it cost to make your patch optional at compile time? Quite a bit, because it changes the logic for processing asynchronous pending calls (signals) and asynchronous exceptions in the eval loop. The #defines would get quite convoluted, I think; I'd prefer not to do that. For what it's worth we could compare your work on different machines and on different platforms before it gets enabled by default. Can you imagine scenarios where your implementation might be slower than the current GIL implementation? I don't really think so. The GIL is taken and released much more predictably than it was before. The thing that might be worth checking is a workload with many threads (say 50 or 100). Does anyone have that? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Antoine Pitrou wrote: Christian Heimes lists at cheimes.de writes: +1 from me. I trust you like Brett does. How much work would it cost to make your patch optional at compile time? Quite a bit, because it changes the logic for processing asynchronous pending calls (signals) and asynchronous exceptions in the eval loop. The #defines would get quite convoluted, I think; I'd prefer not to do that. Based on the new piece of information I totally agree. I don't really think so. The GIL is taken and released much more predictably than it was before. The thing that might be worth checking is a workload with many threads (say 50 or 100). Does anyone have that? I don't have an application that works on Python 3 and uses that many threads, sorry. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] nonlocal keyword in 2.x?
On Sun, Nov 1, 2009 at 13:39, Raymond Hettinger pyt...@rcn.com wrote: As I said: Python 2 support is not only about supporting old versions of Python, but also supporting users of Python2-only modules. So 2.7 support will for the most part be a case not of supporting Python versions, but Python *users*. And contrary to what Antoine said, that *is* a good reason to backport it. FWIW, I support backporting the nonlocal-keyword in 2.7. All of the reasons for introducting nonlocal to 3.x also apply to 2.x. Using the nonlocal keyword in clear and explicit, especially when compared to the existing workarounds which are not pretty. We are voting, I'm -0. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Integer behaviour in Python 2.6.4
Why does this happen? type(2**31-1) type 'long' It seems to have broken NumPy's RNG on Win32. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integer behaviour in Python 2.6.4
On Sun, Nov 1, 2009 at 8:22 PM, Sturla Molden stu...@molden.no wrote: Why does this happen? type(2**31-1) type 'long' Does that not happen on non-Windows platforms? 2**31 can't be represented as a 32-bit signed integer, so it's automatically promoted to a long. -- Curt Hagenlocher c...@hagenlocher.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integer behaviour in Python 2.6.4
Curt Hagenlocher skrev: Does that not happen on non-Windows platforms? 2**31 can't be represented as a 32-bit signed integer, so it's automatically promoted to a long. Yes you are right. I've now traced down the problem to an integer overflow in NumPy. It seems to have this Pyrex code: cdef long lo, hi, diff [...] diff = hi - lo - 1 :-D Sturla ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
On 30Oct2009 20:43, Chris Bergstresser ch...@subtlety.com wrote: | On Fri, Oct 30, 2009 at 8:29 PM, Steven D'Aprano st...@pearwood.info wrote: | Iterating over an iterable is | what iterators are for. | | set.get(), or set.pick() as Wikipedia calls it, isn't for iterating over | sets. It is for getting an arbitrary element from the set. [...] | The purpose is to | return an arbitrary item each time it is called. If two threads get the | same element, that's not a problem; if one thread misses an element | because another thread grabbed it first, that's not a problem either. | If people prefer a random element instead, I have no problem with | that -- personally I think that's overkill, but maybe that's just me. | |I also think returning a random one is overkill, in the base set. | And I'd have the base implementation do the simplest thing possible: | return a fixed element (either the first returned when iterated over, | or the last stored, or whatever's easiest based on the internals). | For me, leaving the choice of *which* element to return on each call | is a feature. It allows subclasses to change the behavior to support | other use cases, like a random or round-robin behavior. Personally, I'm for the iteration spec in a lot of ways. Firstly, a .get()/.pick() that always returns the same element feels horrible. Is there anyone here who _likes_ it? Secondly, I think the thread-unsafe arguments are weak. Question: is the existing set iteration scheme thread safe? i.e. does it return a fresh iterator that a thread can happily use concurrently while another thread uses its own iterator? (Absent set modifications). If the answer is yes, then I don't buy the thread-unsafe complaints - there are already plenty of thread unsafe things much as lots of iterators will break in the face of changes to the data structures over which they're iterating. If iter(set) gets you a safe iterator (absent set modification and multiple users of that iterator) then thread safe methods exist and are easy to use. No presently thread-safe program is going to be broken by adding an iterating .get() method. Thirdly, an iteration-like .get() is dead easy to provide: you just keep a _single_, cycling, internal iterator made on demand the same way __iter__ already works. So why not do just do it? There's no need to construct it before anyone calls .get() the first time. Somewhat like: def get(self): for x in self: return x raise something here but not making a new iterator for every caller. Indeed, making a new iterater per caller, in addition to being expensive, might easily return the same element to every caller. Do anyone feel an iteration capable .get() unduely burdens subclasses that want to impement different .get()s? Both the suggested potential subclass choices (round robin and random) suggest iteration capable .get()s (presuming random means shuffle order rather than potentially returning the same element twice). Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ Why doesn't DOS ever say EXCELLENT command or filename? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
On Sun, Nov 1, 2009 at 3:33 AM, Antoine Pitrou solip...@pitrou.net wrote: Hello again, Brett Cannon brett at python.org writes: I think it's worth it. Removal of the GIL is a totally open-ended problem with no solution in sight. This, on the other hand, is a performance benefit now. I say move forward with this. If it happens to be short-lived because some actually figures out how to remove the GIL then great, but is that really going to happen between now and Python 3.2? I doubt it. Based on this whole discussion, I think I am going to merge the new GIL work into the py3k branch, with priority requests disabled. If you think this is premature or uncalled for, or if you just want to review the changes before making a judgement, please voice up :) +1 Good idea. Thats the best way to make sure this work gets anywhere. It can be iterated on from there if anyone has objections. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
