Re: [Python-Dev] Optimizing list.sort() by checking type in advance

Elliot Gorokhovsky wrote: I ran the benchmark a couple of times and the numbers seem to exactly line up something like one in five times; perhaps not that crazy considering they're executing nearly the same code? Could this be a result of a time being measured in seconds somewhere and then

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 11 October 2016 at 08:49, Guido van Rossum wrote: > On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings wrote: >> For now I'm going to leave the names as-is and just change the semantics >> everywhere. If this approach really works, and if we decide we want

Re: [Python-Dev] [Distutils] pythonhosted.org upload no longer works

I only mentioned it because the OP mentioned the theme as one of his reasons to prefer one over the other. On Mon, Oct 10, 2016 at 8:16 PM, wrote: > > >> -Original Message- >> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- >>

Re: [Python-Dev] [Distutils] pythonhosted.org upload no longer works

> -Original Message- > From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- > list=sdamon@python.org] On Behalf Of Guido van Rossum > Sent: Monday, October 10, 2016 12:27 PM > To: Donald Stufft > Cc: Distutils ; Giampaolo Rodola' >

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

Right - sorry about that last bit! I couldn't figure out how to use timeit/perf for this because the list has to be reinitialized each time it gets sorted. So with time.time() I can just wrap the sorting and cut the initialization out (and I could always throw it in a for loop to do it as many

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

So here's a simple attempt at taking lots of measurements just using time.time() with lists of ints. The results are great, if they are valid (which I leave to you to judge); even for lists with just one element, it's 16% faster! The columns are length, number of trials, and percent improvement:

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

[please restrict follow-ups to python-ideas] Let's not get hung up on meta-discussion here - I always thought "massive clusterf**k" was a precise technical term anyway ;-) While timing certainly needs to be done more carefully, it's obvious to me that this approach _should_ pay off significantly

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 3:27 PM, Greg Ewing wrote: > Nathaniel Smith wrote: >> >> IIRC to handle >> this gilectomy adds per-object mutexes that you have to hold whenever >> you're mucking around with that object's internals. > > > What counts as "mucking around with

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

So maybe someone should explain to Elliott *why* his own benchmarks are not trustworthy, rather than just repeat "use perf or timeit". Actually, there are two things: (a) when something new comes along it *always* needs to prove beyond a shadow of a doubt that it is actually an improvement and not

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On Mon, Oct 10, 2016 at 09:16:32PM +, Elliot Gorokhovsky wrote: > Anyway, benchmarking technique aside, the point is that it it works well > for small lists (i.e. doesn't affect performance). You've been shown that there is something suspicious about your benchmarking technique, something

[Python-Dev] [RELEASE] Python 3.6.0b2 is now available

On behalf of the Python development community and the Python 3.6 release team, I'm happy to announce the availability of Python 3.6.0b2. 3.6.0b2 is the second of four planned beta releases of Python 3.6, the next major release of Python, and marks the end of the feature development phase for 3.6.

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 9:52 AM, MRAB wrote: >> Also, I don't know when it would ever be safe to release the "memory >> deallocation lock". Just because it's safe for your thread doesn't mean >> it's safe for another thread. And if you do it on a thread-by-thread >>

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 22:14, Larry Hastings wrote: On 10/10/2016 09:36 PM, Chris Angelico wrote: Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory deallocation lock". While it is held (by any thread - it's a guard, more than a lock), Py_DECREF will not actually deallocate memory -

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings wrote: > > On 10/10/2016 06:37 PM, Guido van Rossum wrote: > > Modified +1: you can't change the behavior of the existing API, but > you can deprecate it and introduce a better one with a different name. > We'll have until Python

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Random832 wrote: Or is this a special kind of lock that you can "assert isn't locked" without locking it for yourself, and INCREF/DECREF does so? I don't think that would work. It might be unlocked at the moment you test it, but someone might lock it between then and the following

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Nathaniel Smith wrote: IIRC to handle this gilectomy adds per-object mutexes that you have to hold whenever you're mucking around with that object's internals. What counts as "mucking around with the object's internals", though? If I do the C equivalent of: x = somedict[5]

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore wrote: If you have an object that forever owns a reference to another object, it's safe to return borrowed references. Even if there are such cases, it seems we're going to have to be a lot more careful about identifying them.

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On Mon, Oct 10, 2016 at 2:16 PM, Elliot Gorokhovsky wrote: > Hm... that is strange, but I don't think there's anything wrong with the way > I'm timing, though I agree perf/timeit would be better. I ran the benchmark > a couple of times and the numbers seem to exactly

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings wrote: > But I don't think this fixes the problem. Consider: > > Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed reference. > Thread A then gets suspended, before it has a chance to Py_INCREF(Q). > Thread B

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On Tue, Oct 11, 2016 at 7:42 AM, Elliot Gorokhovsky wrote: > ChrisA suggested I also try "make test" or something to get a more realistic > benchmark. I will do that once I implement this as a patch, right now it's > an extension module that subclasses list, so I

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

Hm... that is strange, but I don't think there's anything wrong with the way I'm timing, though I agree perf/timeit would be better. I ran the benchmark a couple of times and the numbers seem to exactly line up something like one in five times; perhaps not that crazy considering they're executing

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 09:36 PM, Chris Angelico wrote: Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory deallocation lock". While it is held (by any thread - it's a guard, more than a lock), Py_DECREF will not actually deallocate memory - objects can fall to zero references without

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 06:37 PM, Guido van Rossum wrote: Modified +1: you can't change the behavior of the existing API, but you can deprecate it and introduce a better one with a different name. We'll have until Python 4.0 to carry through the deprecation anyways. And I doubt this is the only C API

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On Mon, Oct 10, 2016 at 1:42 PM, Elliot Gorokhovsky wrote: > *** 10 strings *** > F.fastsort(): 1.6689300537109375e-06 > F.sort(): 1.6689300537109375e-06 I think something has gone wrong with your timing harness... For accurately timing microbenchmarks, you should

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

I'd like to reply to both questions with one email, if that's all right. Bernardo Sulzbach asked, "How does this interfere with sorting very small lists?", and ChrisA asked, "How much slower are sorts of heterogeneous lists?". I modified the benchmark to answer both questions, the former at the

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 20:36, Chris Angelico wrote: On Tue, Oct 11, 2016 at 5:24 AM, Random832 wrote: On Mon, Oct 10, 2016, at 14:04, MRAB wrote: Instead of locking the object, could we keep the GIL, but have it normally released? A thread could then still call a function such

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 5:24 AM, Random832 wrote: > On Mon, Oct 10, 2016, at 14:04, MRAB wrote: >> Instead of locking the object, could we keep the GIL, but have it >> normally released? >> >> A thread could then still call a function such as PyWeakref_GetObject() >> that

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016, at 14:04, MRAB wrote: > Instead of locking the object, could we keep the GIL, but have it > normally released? > > A thread could then still call a function such as PyWeakref_GetObject() > that returns a borrowed reference, but only if it's holding the GIL. It > would be

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 18:50, Nathaniel Smith wrote: On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10 October 2016 at 18:50, Nathaniel Smith wrote: > I assume Larry is way ahead of us on this, Yeah, I'd imagine you're right on that :-) Paul ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and there's no GIL, another thread could delete the

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore wrote: > On 10 October 2016 at 17:49, MRAB wrote: >> If you lookup something in a dict, it'll be a borrowed reference. >> >> If the dict is globals() and there's no GIL, another thread could delete the

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Tue, Oct 11, 2016 at 3:49 AM, MRAB wrote: > On 2016-10-10 10:45, Chris Angelico wrote: >> >> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings >> wrote: >>> >>> Huh? In all other circumstances, a "borrowed" reference is exactly that: >>> X >>>

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10 October 2016 at 17:49, MRAB wrote: > If you lookup something in a dict, it'll be a borrowed reference. > > If the dict is globals() and there's no GIL, another thread could delete the > item while your code had the borrowed reference. > > It looks like there

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 2016-10-10 10:45, Chris Angelico wrote: On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings wrote: Huh? In all other circumstances, a "borrowed" reference is exactly that: X has a reference, and you are relying on X's reference to keep the object alive. Borrowing from a

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Modified +1: you can't change the behavior of the existing API, but you can deprecate it and introduce a better one with a different name. We'll have until Python 4.0 to carry through the deprecation anyways. And I doubt this is the only C API change needed for happy gil-free coding... On Mon,

Re: [Python-Dev] [Distutils] pythonhosted.org upload no longer works

Honestly I like readthedocs a lot, and I actually *don't* like docs that look too much like the standard Python docs -- it should be clear to readers (subliminally, through page style, not just be parsing the URL) that they're reading third party docs. On Mon, Oct 10, 2016 at 9:06 AM, Donald

Re: [Python-Dev] [Distutils] pythonhosted.org upload no longer works

You’re using the new PyPI (either by getting a new default from a new version of Python or Twine OR explicitly switching to it yourself) which has this disabled. You can still upload to legacy PyPI by switching the default back. That will restore the ability to upload docs to pythonhosted.org.

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On Mon, Oct 10, 2016 at 5:18 PM, Elliot Gorokhovsky wrote: > First, some simple benchmark results (numbers are seconds, check out the > extension module at https://github.com/embg/python-fast-listsort.git): > > *** 1e3 ints *** > F.fastsort(): 0.00018930435180664062

Re: [Python-Dev] Optimizing list.sort() by checking type in advance

On 10/10/2016 03:18 AM, Elliot Gorokhovsky wrote: Hi, I posted here a while back asking what you guys thought about implementing radix sort for strings in list.sort(). You gave me a lot of reasons why that would be a bad idea. However, it got me thinking, and I came up with something that you

Re: [Python-Dev] pythonhosted.org upload no longer works

Thanks Paul, no I wasn't aware of that, and I will subscribe to distutils-sig from now on. I don't remember ever receiving a warrant about this decision but it's entirely possible I may have forgot. =) So long story short is that I will have to move the doc on readthedocs, correct? Does

[Python-Dev] Optimizing list.sort() by checking type in advance

Hi, I posted here a while back asking what you guys thought about implementing radix sort for strings in list.sort(). You gave me a lot of reasons why that would be a bad idea. However, it got me thinking, and I came up with something that you may find interesting. First, some simple benchmark

Re: [Python-Dev] pythonhosted.org upload no longer works

On 10 October 2016 at 14:34, Giampaolo Rodola' wrote: > This is what I've bumped into just now: > > python setup.py upload_sphinx --upload-dir=docs/_build/html > running upload_sphinx > Submitting documentation to https://upload.pypi.org/legacy/ > Upload failed (410):

[Python-Dev] pythonhosted.org upload no longer works

This is what I've bumped into just now: python setup.py upload_sphinx --upload-dir=docs/_build/html running upload_sphinx Submitting documentation to https://upload.pypi.org/legacy/ Upload failed (410): Uploading documentation is no longer supported, we recommend using https://readthedocs.org/.

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 4:17 AM, Larry Hastings wrote: > Given that the weakref doesn't have a reference to the object--merely a weak > reference, different thing--whose reference is it borrowing? As others have said, it doesn't really matter who's reference it was; just that

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

> On 2016-10-10, at 11:05 , Devin Jeanpierre wrote: > The term "borrowed" is supposed to imply a sensible scope during which you're > free to use the object, and weakrefs don't have that (except for what is > granted by the GIL), so this does sound wacky. I bet it was

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

Larry Hastings wrote: In contrast, the "borrowed" reference returned by PyWeakRef_GetObject() seems to be "borrowed" from some unspecified entity. It's a delocalised quantum reference, borrowing a little bit from all strong references in existence. :-) -- Greg

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings wrote: > Huh? In all other circumstances, a "borrowed" reference is exactly that: X > has a reference, and you are relying on X's reference to keep the object > alive. Borrowing from a borrowed reference is simply a chain of

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

On 10/10/2016 11:05 AM, Devin Jeanpierre wrote: > whose reference is it borrowing? I think this is a red herring sort of question. "borrowed" only means "unowned". But anyway, we borrowed from the weakref. (Borrowing from somebody doesn't imply they own a reference -- we can borrow from a

Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

> It's my contention that this API is simply untenable under the Gilectomy Yeah, I agree with your contention. The only circumstance PyWeakref_GetObject would still be valid without a GIL is if you happened to know you had a reference elsewhere that was keeping it alive. But if you did, you

[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

The documentation for PyWeakref_GetObject() states: Return value: Borrowed reference. Return the referenced object from a weak reference, ref. If the referent is no longer live, returns Py_None. Given that the weakref doesn't have a reference to the object--merely a weak reference,