[Python-Dev] building Tcl/Tk to deploy on other platforms?
I built Python 2.5.2 on RedHat3 and wrapped it up with some other stuff that was deployed on RedHat4. When trying to fire up Idle on RedHat4, there is an error that states a usable init.tcl cannot be found. Python is built on RedHat3 against Tcl/Tk 8.3, so even installing Tcl/Tk on RedHat4 would not work, as it is 8.4. I've noticed when Python is installed on Windows, all the necessary stuff is also installed in the Python25 directory. Is there a similar way to do this on linux? Thanks, -Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] building Tcl/Tk to deploy on other platforms?
This may work on the deployed side, although I'm skeptical if I've built against a different version of Tcl/Tk than exists. I don't see how this helps on the build side, althought I'll look into it. On Wed, Nov 26, 2008 at 5:51 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: >> Adjust the TCL_LIBRARY and TK_LIBRARY as needed. > > We should also point out that the FixTk module already does > that, but is imported only on Windows. So if you arrange to > always import FixTk, then it should Just Work. > > Regards, > Martin > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Sun, Jun 22, 2014 at 8:00 AM, Steve Dower wrote: > We can always lie about the version in sys.version. Existing code is > unaffected and new code will have to use version_info (Windows developers > will know that Windows pulls tricks like this every other version... doesn't > make it a great idea, but it works). I'd prefer a change of format to an outright lie. Something like "2.7._10" which will sort after "2.7.9". But ideally, nothing at all - just go smoothly to "2.7.10" and let broken code be broken. It'll think it's running on 2.7.1, and if anything needs to distinguish between 2.7.1 and 2.7.x, hopefully it's using version_info. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Sun, Jun 22, 2014 at 7:37 AM, M.-A. Lemburg wrote: > There are no places in the stdlib that parse sys.version in a > way that would break wtih 2.7.10, AFAIK. I was just referring > to the statement that Nick quoted. sys.version *is* used for > parsing the Python version or using parts of it to build > e.g. filenames and that's really no surprise. Right, good to know. I thought you were implying that stuff would break. Yes, stuff definitely does parse out the version number from sys.version, lots of that happens. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
Not being a Python developer, I normally just lurk on Py-Dev, but I figured I'd throw this out there for this thread: Recent version of Maya embed Python 2.x, and the newer version of Maya (I believe 2012 was the first version) embeds a Python 2.7 compiled with VS 2010. From my experience, most C extensions work across compiler versions, however when they don't, it's generally a fairly difficult to debug issue - at least unless you know what to look for in the call stacks, and have access to the symbol files. Chris On Mon, Jun 23, 2014 at 2:07 PM, M.-A. Lemburg wrote: > On 23.06.2014 22:20, Donald Stufft wrote: > > > > On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg wrote: > > > >> On 23.06.2014 18:09, Donald Stufft wrote: > >>> > >>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis > wrote: > >>> > >>>>> > >>>>> * Should we make use of the potential breakage with 2.7.10 > >>>>> to introduce a new Windows compiler version for Python 2.7 ? > >>>> > >>>> Assuming it is a good idea to continue producing Windows binaries > >>>> for 2.7, I think it would be a bad idea to switch compilers. It will > >>>> cause severe breakage of 2.7 installations, much more problematic > >>>> than switching to two-digit version numbers. > >>> > >>> I agree with this, we’ve just finally started getting things to the > point where > >>> it makes a lot of sense for binary distributions for Windows. Breaking > all > >>> of them on 2.7 would be very bad. > > > > Err, sorry that “We” was with my pip hat on. > > > >> > >> Not sure what you mean. We've had binary wininst distributions > >> for Windows for more than a decade, and egg and msi distributions > >> for 8 years :-) > > > > Nonetheless, changing the compiler will not only break pip, but every > > automated installer tool (easy_install, buildout) that i’m aware of. The > > blow back for binary installation is going to be huge I think. > > > >> But without access to the VS 2008 compiler that is needed to > >> compile those extensions, it will become increasingly difficult > >> for package authors to provide such binary packages, so we have to > >> ask ourselves: > >> > >> What's worse: breaking old Windows binaries for Python 2.7 > >> or not having updated and new Windows binaries for Python 2.7 > >> at all in a few years ? > > > > At the risk of getting Guido to post his slide again, I still think the > > solution to the old compiler is to just roll a 2.8 with minimal changes. > > It could even be a good place to move to the ssl backport changes > > too since they were the riskier set of changes in PEP466. > > > > But either way, if a compiler does change in a 2.7 release we’ll need > > to update a lot of tooling to cope with that, so any plan to do that > should > > include that and a timeline for adoption of that. > > Sure, and we'd need to hash out possible solutions to minimize > breakage, but first we'll have to see whether we want to consider > this step or not. > > > BTW: It's strange that I'm arguing for breaking things. I'm usually > on the other side of such arguments :-) > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source > >>> Python/Zope Consulting and Support ...http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ > > > ::: Try our new mxODBC.Connect Python Database Interface for free ! > > >eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >Registered at Amtsgericht Duesseldorf: HRB 46611 >http://www.egenix.com/company/contact/ > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ckaynor%40zindagigames.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" wrote: > See my other message. It's actually heavier, since it requires changes > to distutils, PyPI, pip, buildout etc., all which know how to deal with > Python minor version numbers, but are unaware of the notion of competing > ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit). Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of things to handle the different compilers? So, for instance, there might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new architectures, just as if someone came out with a 128-bit Windows and built Python 2.7 for it. Would packaging be able to handle that more easily than a compiler change within the same architecture? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
On Thu, Jun 26, 2014 at 9:04 PM, Antoine Pitrou wrote: > For the same reason, I agree with Victor that we should ditch the > threading-disabled builds. It's too much of a hassle for no actual, > practical benefit. People who want a threadless unicodeless Python can > install Python 1.5.2 for all I care. Or some other implementation of Python. It's looking like micropython will be permanently supporting a non-Unicode build (although I stepped away from the project after a strong disagreement over what would and would not make sense, and haven't been following it since). If someone wants a Python that doesn't have stuff that the core CPython devs treat as essential, s/he probably wants something like uPy anyway. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
On Sat, Jun 28, 2014 at 11:05 PM, Akira Li <4kir4...@gmail.com> wrote: > Have you considered adding support for paths relative to directory > descriptors [1] via keyword only dir_fd=None parameter if it may lead to > more efficient implementations on some platforms? > > [1]: https://docs.python.org/3.4/library/os.html#dir-fd Potentially more efficient and also potentially safer (see 'man openat')... but an enhancement that can wait, if necessary. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summary of the scandir (PEP 471)
On Wed, Jul 2, 2014 at 7:20 AM, Paul Moore wrote: > I think that thin wrapper is needed - even > if the various bells and whistles are useful, they can be built on top > of a low-level version (whereas the converse is not the case). +1. Make everything as simple as possible (but no simpler). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 1:15 AM, Benjamin Peterson wrote: > Why do you think that? > > % python > Python 2.7.6 (default, May 29 2014, 22:22:15) > [GCC 4.7.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. class x(object): pass > ... class y(object): pass > ... x != y > True x == y > False Your analysis is flawed - you're testing the equality of the types, not of instances. But your conclusion's correct; testing instances does work the same way you're implying: rosuav@sikorsky:~$ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class x(object): pass ... >>> class y(object): pass ... >>> x() != y() True >>> x() == y() False >>> x() == x() False >>> z = x() >>> z == z True ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 11:59 AM, Rob Cliffe wrote: > If I came across an int object and had no concept of what an integer number > was, how would I know what its "value" is supposed to be? The value of an integer is the number it represents. In CPython, it's entirely possible to have multiple integer objects (ie objects with unique identities) with the same value, although AIUI there are Pythons for which that's not the case. The value of a float, Fraction, Decimal, or complex is also the number it represents, so when you compare 1==1.0, the answer is that they have the same value. They can't possibly have the same identity (every object has a single type), but they have the same value. But what *is* that value? It's not something that can be independently recognized, because casting to a different type might change the value: >>> i = 2**53+1 >>> f = float(i) >>> i == f False >>> f == int(f) True Ergo the comparison of a float to an int cannot be done by casting the int to float, nor by casting the float to int; it has to be done by comparing the abstract numbers represented. Those are the objects' values. But what's the value of a sentinel object? _SENTINEL = object() def f(x, y=_SENTINEL): do_something_with(x) if y is not _SENTINEL: do_something_with(y) I'd say this is a reasonable argument for the default object value to be identity. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 1:12 PM, Steven D'Aprano wrote: > Why? What value (pun intended) is there in adding an explicit statement > of value to every single class? > > "The value of a str is the str's sequence of characters." > "The value of a list is the list's sequence of items." > "The value of an int is the int's numeric value." > "The value of a float is the float's numeric value, or in the case of > INFs and NANs, that they are an INF or NAN." > "The value of a complex number is the ordered pair of its real and > imaginary components." > "The value of a re MatchObject is the MatchObject itself." > > I don't see any benefit to forcing all classes to explicitly document > this sort of thing. It's nearly always redundant and unnecessary. It's important where it's not obvious. For instance, two lists with the same items are equal, two tuples with the same items are equal, but a list and a tuple with the same items aren't. Doesn't mean it necessarily has to be documented, though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 5:01 PM, Stephen J. Turnbull wrote:
> I agree with Steven d'A that this rule is not part of the language
> definition and shouldn't be, but it's the rule of thumb I find hardest
> to imagine *ever* wanting to break in my own code (although I sort of
> understand why the IEEE 754 committee found they had to).
The reason NaN isn't equal to itself is because there are X bit
patterns representing NaN, but an infinite number of possible
non-numbers that could result from a calculation. Is
float("inf")-float("inf") equal to float("inf")/float("inf")? There
are three ways NaN equality could have been defined:
1) All NaNs are equal, as if NaN is some kind of "special number".
2) NaNs are equal if they have the exact same bit pattern, and unequal else.
3) All NaNs are unequal, even if they have the same bit pattern.
The first option is very dangerous, because it'll mean that "NaN
pollution" can actually result in unexpected equality. The second
looks fine - a NaN is equal to itself, for instance - but it suffers
from the pigeonhole problem, in that eventually you'll have two
numbers which resulted from different calculations and happen to have
the same bit pattern. The third is what IEEE went with. It's the
sanest option.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 5:53 PM, Stephen J. Turnbull wrote: > But you're missing at least two alternatives that > involve raising on some calculations involving NaN, as well as the > fact that forcing inequality of two NaNs produced by equivalent > calculations is arguably just as wrong as allowing equality of two > NaNs produced by the different calculations. This is off-topic for this thread, but still... The trouble is that your "arguably just as wrong" is an indistinguishable case. If you don't want two different calculations' NaNs to *ever* compare equal, the only solution is to have all NaNs compare unequal - otherwise, two calculations might happen to produce the same bitpattern, as there are only a finite number of them available. > That's where things get > fuzzy for me -- in Python I would expect that preserving invariants > would be more important than computational efficiency, but evidently > it's not. What invariant is being violated for efficiency? As I see it, it's one possible invariant (things should be equal to themselves) coming up against another possible invariant (one way of generating NaN is unequal to any other way of generating NaN). Raising an exception is, of course, the purpose of signalling NaNs rather than quiet NaNs, which is a separate consideration from how they compare. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Wed, Jul 9, 2014 at 3:00 AM, Steven D'Aprano wrote: > On Tue, Jul 08, 2014 at 04:58:33PM +0200, Anders J. Munch wrote: > >> For two NaNs computed differently to compare equal is no worse than 2+2 >> comparing equal to 1+3. You're comparing values, not their history. > > a = -23 > b = -42 > if log(a) == log(b): > print "a == b" That could also happen from rounding error, though. >>> a = 2.0**52 >>> b = a+1.0 >>> a == b False >>> log(a) == log(b) True Any time you do any operation on numbers that are close together but not equal, you run the risk of getting results that, in finite-precision floating point, are deemed equal, even though mathematically they shouldn't be (two unequal numbers MUST have unequal logarithms). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another case for frozendict
On Mon, Jul 14, 2014 at 12:04 AM, Jason R. Coombs wrote:
> I can achieve what I need by constructing a set on the ‘items’ of the dict.
>
set(tuple(doc.items()) for doc in res)
>
> {(('n', 1), ('err', None), ('ok', 1.0))}
This is flawed; the tuple-of-tuples depends on iteration order, which
may vary. It should be a frozenset of those tuples, not a tuple. Which
strengthens your case; it's that easy to get it wrong in the absence
of an actual frozendict.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x - list delegation to members?
On Mon, Jul 14, 2014 at 2:23 AM, Steven D'Aprano wrote:
>> We will see
>> later that that happens. Further, when comparing float NaNs of the same
>> identity, the list implementation forgot to special-case NaNs. Which
>> would be a bug, IMHO.
>
> "Forgot"? I don't think the behaviour of list comparisons is an
> accident.
Well, "forgot" is on the basis that the identity check is intended to
be a mere optimization. If that were the case ("don't actually call
__eq__ when you reckon it'll return True"), then yes, failing to
special-case NaN would be a bug. But since it's intended behaviour, as
explained further down, it's not a bug and not the result of
forgetfulness.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x - list delegation to members?
On Mon, Jul 14, 2014 at 4:11 AM, Nick Coghlan wrote: > What we've never figured out is a good place to *document* it. I > thought there was an open bug for that, but I can't find it right now. Yeah. The Py3 docs explain why "x in [x]" is True, but I haven't found a parallel explanation of sequence equality. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Sat, Aug 2, 2014 at 1:35 PM, David Wilson wrote: > > Repeated list and str concatenation both have quadratic O(N**2) > > performance, but people frequently build up strings with + > > join() isn't preferable in cases where it damages readability while > simultaneously providing zero or negative performance benefit, such as > when concatenating a few short strings, e.g. while adding a prefix to a > filename. > Good point -- I was trying to make the point about .join() vs + for strings in an intro python class last year, and made the mistake of having the students test the performance. You need to concatenate a LOT of strings to see any difference at all -- I know that O() of algorithms is unavoidable, but between efficient python optimizations and a an apparently good memory allocator, it's really a practical non-issue. > Although it's true that join() is automatically the safer option, and > especially when dealing with user supplied data, the net harm caused by > teaching rote and ceremony seems far less desirable compared to fixing a > trivial slowdown in a script, if that slowdown ever became apparent. > and it rarely would. Blocking sum( some_strings) because it _might_ have poor performance seems awfully pedantic. As a long-time numpy user, I think sum(a_long_list_of_numbers) has pathetically bad performance, but I wouldn't block it! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Mon, Aug 4, 2014 at 11:10 AM, Steven D'Aprano
wrote:
> On Mon, Aug 04, 2014 at 09:25:12AM -0700, Chris Barker wrote:
>
> > Good point -- I was trying to make the point about .join() vs + for
> strings
> > in an intro python class last year, and made the mistake of having the
> > students test the performance.
> >
> > You need to concatenate a LOT of strings to see any difference at all
>
> If only that were the case, but it isn't. Here's a cautionary tale for
> how using string concatenation can blow up in your face:
>
> Chris Withers asks for help debugging HTTP slowness:
> https://mail.python.org/pipermail/python-dev/2009-August/091125.html
Thanks for that -- interesting story. note that that was not suing sum() in
that case though, which is really the issue at hand.
It shouldn't be hard to demonstrate the difference between repeated
> string concatenation and join, all you need do is defeat sum()'s
> prohibition against strings. Run this bit of code, and you'll see a
> significant difference in performance, even with CPython's optimized
> concatenation:
>
well, that does look compelling, but what it shows is that
sum(a_list_of_strings) is slow compared to ''.join(a_list_of_stings). That
doesn't surprise me a bit -- this is really similar to why:
a_numpy_array.sum()
is going to be a lot faster than:
sum(a_numpy_array)
and why I'll tell everyone that is working with lots of numbers to use
numpy. ndarray.sum know what data type it's deaing with,a nd can do the
loop in C. similarly with ''.join() (though not as optimized.
But I'm not sure we're seeing the big O difference here at all -- but
rather the extra calls though each element in the list's __add__ method.
In the case where you already HAVE a big list of strings, then yes, ''.join
is the clear winner.
But I think the case we're often talking about, and I've tested with
students, is when you are building up a long string on the fly out of
little strings. In that case, you need to profile the full "append to list,
then call join()", not just the join() call:
# continued adding of strings ( O(n^2)? )
In [6]: def add_strings(l):
...: s = ''
...: for i in l:
...: s+=i
...: return s
Using append and then join ( O(n)? )
In [14]: def join_strings(list_of_strings):
: l = []
: for i in list_of_strings:
: l.append(i)
: return ''.join(l)
In [23]: timeit add_strings(strings)
100 loops, best of 3: 831 ns per loop
In [24]: timeit join_strings(strings)
10 loops, best of 3: 1.87 µs per loop
## hmm -- concatenating is faster for a small list of tiny strings
In [31]: strings = list('Hello World')* 1000
strings *= 1000
In [26]: timeit add_strings(strings)
1000 loops, best of 3: 932 µs per loop
In [27]: timeit join_strings(strings)
1000 loops, best of 3: 967 µs per loop
## now about the same.
In [31]: strings = list('Hello World')* 1
In [29]: timeit add_strings(strings)
100 loops, best of 3: 9.44 ms per loop
In [30]: timeit join_strings(strings)
100 loops, best of 3: 10.1 ms per loop
still about he same?
In [31]: strings = list('Hello World')* 100
In [32]: timeit add_strings(strings)
1 loops, best of 3: 1.27 s per loop
In [33]: timeit join_strings(strings)
1 loops, best of 3: 1.05 s per loop
there we go -- slight advantage to joining.
So this is why we've said that the common wisdom about string concatenating
isn't really a practical issue.
But if you already have the strings all in a list, then yes, join() is a
major win over sum()
In fact, I tried the above with sum() -- and it was really, really slow. So
slow I didn't have the patience to wait for it.
Here is a smaller example:
In [22]: strings = list('Hello World')* 1
In [23]: timeit add_strings(strings)
100 loops, best of 3: 9.61 ms per loop
In [24]: timeit sum( strings, Faker() )
1 loops, best of 3: 246 ms per loop
So why is sum() so darn slow with strings compared to a simple loop with +=
?
(and if I try it with a list 10 times as long it takes "forever")
Perhaps the http issue cited was before some nifty optimizations in current
CPython?
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Thu, Aug 7, 2014 at 4:01 PM, Ethan Furman wrote:
> I don't remember where, but I believe that cPython has an optimization
> built in for repeated string concatenation, which is probably why you
> aren't seeing big differences between the + and the sum().
>
Indeed -- clearly so.
A little testing shows how to defeat that optimization:
blah = ''
> for string in ['booyah'] * 10:
> blah = string + blah
>
> Note the reversed order of the addition.
>
thanks -- cool trick.
Oh, and the join() timings:
> --> timeit.Timer("blah = ''.join(['booya'] * 10)", "blah =
> ''").repeat(3, 1)
> [0.0014629364013671875, 0.0014190673828125, 0.0011930465698242188]
> So, + is three orders of magnitude slower than join.
only one if if you use the optimized form of + and not even that if you
need to build up the list first, which is the common use-case.
So my final question is this:
repeated string concatenation is not the "recommended" way to do this --
but nevertheless, cPython has an optimization that makes it fast and
efficient, to the point that there is no practical performance reason to
prefer appending to a list and calling join()) afterward.
So why not apply a similar optimization to sum() for strings?
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
On Sun, Aug 10, 2014 at 10:44 AM, Steven D'Aprano wrote: > Looking at your comment here: > >> [1]: https://news.ycombinator.com/item?id=8154471 > > there is a reply from zeckalpha, who says: > >"Actually, leaving out `object` is the preferred convention for > Python 3, as they are semantically equivalent." > > How does (s)he justify this claim? > >"Explicit is better than implicit." > > which is not logical. If you leave out `object`, that's implicit, not > explicit. The justification is illogical. However, I personally believe boilerplate should be omitted where possible; that's why we have a whole lot of things that "just work". Why does Python not have explicit boolification for if/while checks? REXX does (if you try to use anything else, you get a run-time error "Logical value not 0 or 1"), and that's more explicit - Python could require you to write "if bool(x)" for the case where you actually want the truthiness magic, to distinguish from "if x is not None" etc. But that's unnecessary boilerplate. Python could have required explicit nonlocal declarations for all names used in closures, but that's unhelpful too. Python strives to eliminate that kind of thing. So, my view would be: Py3-only tutorials can and probably should omit it, for the same reason that we don't advise piles of __future__ directives. You can always add stuff later for coping with Py2+Py3 execution; chances are any non-trivial code will have much bigger issues than accidentally making an old-style class. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python2.7 infinite recursion when loading pickled object
On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__pete...@web.de> wrote: > Read again. The OP tries to delegate attribute lookup to an (existing) > attribute. > > IMO the root cause of the problem is that pickle looks up __dunder__ methods > in the instance rather than the class. The recursion comes from the attempted lookup of self.item, when __init__ hasn't been called. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull wrote: > I'm referring to removing the unnecessary information that there's a > better way to do it, and simply raising an error (as in Python 3.2, > say) which is all a RealProgrammer[tm] should ever need! > I can't imagine anyone is suggesting that -- disallow it, but don't tell anyone why? The only thing that is remotely on the table here is: 1) remove the special case for strings -- buyer beware -- but consistent and less "ugly" 2) add a special case for strings that is fast and efficient -- may be as simple as calling "".join() under the hood --no more code than the exception check. And I doubt anyone really is pushing for anything but (2) Steven Turnbull wrote: > IMO we'd also want a homogeneous_iterable ABC Actually, I've thought for years that that would open the door to a lot of optimizations -- but that's a much broader question that sum(). I even brought it up probably over ten years ago -- but no one was the least bit iinterested -- nor are they now -- I now this was a rhetorical suggestion to make the point about what not to do Because obviously we'd want the > attractive nuisance of "if you have __add__, there's a default > definition of __sum__" now I'm confused -- isn't that exactly what we have now? It's possible that Python could provide some kind of feature that > would allow an optimized sum function for every type that has __add__, > but I think this will take a lot of thinking. does it need to be every type? As it is the common ones work fine already except for strings -- so if we add an optimized string sum() then we're done. *Somebody* will do it > (I don't think anybody is +1 on restricting sum() to a subset of types > with __add__). uhm, that's exactly what we have now -- you can use sum() with anything that has an __add__, except strings. Ns by that logic, if we thought there were other inefficient use cases, we'd restrict those too. But users can always define their own classes that have a __sum__ and are really inefficient -- so unless sum() becomes just for a certain subset of built-in types -- does anyone want that? Then we are back to the current situation: sum() can be used for any type that has an __add__ defined. But naive users are likely to try it with strings, and that's bad, so we want to prevent that, and have a special case check for strings. What I fail to see is why it's better to raise an exception and point users to a better way, than to simply provide an optimization so that it's a mute issue. The only justification offered here is that will teach people that summing strings (and some other objects?) is order(N^2) and a bad idea. But: a) Python's primary purpose is practical, not pedagogical (not that it isn't great for that) b) I doubt any naive users learn anything other than "I can't use sum() for strings, I should use "".join()". Will they make the leap to "I shouldn't use string concatenation in a loop, either"? Oh, wait, you can use string concatenation in a loop -- that's been optimized. So will they learn: "some types of object shave poor performance with repeated concatenation and shouldn't be used with sum(). So If I write such a class, and want to sum them up, I'll need to write an optimized version of that code"? I submit that no naive user is going to get any closer to a proper understanding of algorithmic Order behavior from this small hint. Which leaves no reason to prefer an Exception to an optimization. One other point: perhaps this will lead a naive user into thinking -- "sum() raises an exception if I try to use it inefficiently, so it must be OK to use for anything that doesn't raise an exception" -- that would be a bad lesson to mis-learn -Chris PS: Armin Rigo wrote: > It also improves a > lot the precision of sum(list_of_floats) (though not reaching the same > precision levels of math.fsum()). while we are at it, having the default sum() for floats be fsum() would be nice -- I'd rather the default was better accuracy loser performance. Folks that really care about performance could call math.fastsum(), or really, use numpy... This does turn sum() into a function that does type-based dispatch, but isn't python full of those already? do something special for the types you know about, call the generic dunder method for the rest. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Wed, Aug 13, 2014 at 11:11 PM, Isaac Morland wrote: > While I would not claim a Python sandbox is utterly impossible, I'm > suspicious that the whole "consenting adults" approach in Python is > incompatible with a sandbox. The whole idea of a sandbox is to absolutely > prevent people from doing things even if they really want to and know what > they are doing. It's certainly not *fundamentally* impossible to sandbox Python. However, the question becomes one of how much effort you're going to go to and how much you're going to restrict the code. I think I remember reading about something that's like ast.literal_eval, but allows name references; with that, plus some tiny features of assignment, you could make a fairly straight-forward evaluator that lets you work comfortably with numbers, strings, lists, dicts, etc. That could be pretty useful - but it wouldn't so much be "Python in a sandbox" as "an expression evaluator that uses a severely restricted set of Python syntax". If you start with all of Python and then start cutting out the dangerous bits, you're doomed to miss something, and your sandbox is broken. If you start with nothing and then start adding functionality, you're looking at a gigantic job before it becomes anything that you could call an applications language. So while it's theoretically possible (I think - certainly I can't say for sure that it's impossible), it's fairly impractical. I've had my own try at it, and failed quite badly (fortunately noisily and at a sufficiently early stage of development to shift). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Thu, Aug 14, 2014 at 2:58 AM, Steven D'Aprano wrote: >> It's certainly not *fundamentally* impossible to sandbox Python. >> However, the question becomes one of how much effort you're going to >> go to and how much you're going to restrict the code. > > I believe that PyPy has an effective sandbox, but to what degree of > effectiveness I don't know. """ A potential attacker can have arbitrary code run in the subprocess, but cannot actually do any input/output not controlled by the outer process. Additional barriers are put to limit the amount of RAM and CPU time used. Note that this is very different from sandboxing at the Python language level, i.e. placing restrictions on what kind of Python code the attacker is allowed to run (why? read about pysandbox). """ That's quite useful, but isn't the same thing as a Python-in-Python sandbox (or even what I was doing, Python-in-C++). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Tue, Aug 12, 2014 at 11:21 PM, Stephen J. Turnbull wrote: > Redirecting to python-ideas, so trimming less than I might. reasonable enough -- you are introducing some more significant ideas for changes. I've said all I have to say about this -- I don't seem to see anything encouraging form core devs, so I guess that's it. Thanks for the fun bike-shedding... -Chris > Chris Barker writes: > > On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull < > step...@xemacs.org> > > wrote: > > > > > I'm referring to removing the unnecessary information that there's a > > > better way to do it, and simply raising an error (as in Python 3.2, > > > say) which is all a RealProgrammer[tm] should ever need! > > > > > > > I can't imagine anyone is suggesting that -- disallow it, but don't tell > > anyone why? > > As I said, it's a regression. That's exactly the behavior in Python 3.2. > > > The only thing that is remotely on the table here is: > > > > 1) remove the special case for strings -- buyer beware -- but consistent > > and less "ugly" > > It's only consistent if you believe that Python has strict rules for > use of various operators. It doesn't, except as far as they are > constrained by precedence. For example, I have an application where I > add bytestrings bytewise modulo N <= 256, and concatenate them. In > fact I use function call syntax, but the obvious operator syntax is > '+' for the bytewise addition, and '*' for the concatenation. > > It's not in the Zen, but I believe in the maxim "If it's worth doing, > it's worth doing well." So for me, 1) is out anyway. > > > 2) add a special case for strings that is fast and efficient -- may be > as > > simple as calling "".join() under the hood --no more code than the > > exception check. > > Sure, but what about all the other immutable containers with __add__ > methods? What about mappings with key-wise __add__ methods whose > values might be immutable but have __add__ methods? Where do you stop > with the special-casing? I consider this far more complex and ugly > than the simple "sum() is for numbers" rule (and even that is way too > complex considering accuracy of summing floats). > > > And I doubt anyone really is pushing for anything but (2) > > I know that, but I think it's the wrong solution to the problem (which > is genuine IMO). The right solution is something generic, possibly a > __sum__ method. The question is whether that leads to too much work > to be worth it (eg, "homogeneous_iterable"). > > > > Because obviously we'd want the attractive nuisance of "if you > > > have __add__, there's a default definition of __sum__" > > > > now I'm confused -- isn't that exactly what we have now? > > Yes and my feeling (backed up by arguments that I admit may persuade > nobody but myself) is that what we have now kinda sucks[tm]. It > seemed like a good idea when I first saw it, but then, my apps don't > scale to where the pain starts in my own usage. > > > > It's possible that Python could provide some kind of feature that > > > would allow an optimized sum function for every type that has > > > __add__, but I think this will take a lot of thinking. > > > > does it need to be every type? As it is the common ones work fine > already > > except for strings -- so if we add an optimized string sum() then we're > > done. > > I didn't say provide an optimized sum(), I said provide a feature > enabling people who want to optimize sum() to do so. So yes, it needs > to be every type (the optional __sum__ method is a proof of concept, > modulo it actually being implementable ;-). > > > > *Somebody* will do it (I don't think anybody is +1 on restricting > > > sum() to a subset of types with __add__). > > > > uhm, that's exactly what we have now > > Exactly. Who's arguing that the sum() we have now is a ticket to > Paradise? I'm just saying that there's probably somebody out there > negative enough on the current situation to come up with an answer > that I think is general enough (and I suspect that python-dev > consensus is that demanding, too). > > > sum() can be used for any type that has an __add__ defined. > > I'd like to see that be mutable types with __iadd__. > > > What I fail to see is why it's better to raise an exception and > > point users to a better way, t
Re: [Python-Dev] Multiline with statement line continuation
On Sat, Aug 16, 2014 at 10:47 PM, Marko Rauhamaa wrote:
>
> You might be able to have it bothways. You could have:
>
>with (open(name) for name in os.listdir("config")) as files:
But that's not a tuple, it's a generator. Should generators be context
managers? Is anyone seriously suggesting this? I don't think so. Is
this solutions looking for problems?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray
On 08/17/2014 09:40 PM, Antoine Pitrou wrote: Le 17/08/2014 20:08, Nick Coghlan a écrit : On 18 Aug 2014 09:57, "Barry Warsaw" mailto:ba...@python.org>> wrote: > > On Aug 18, 2014, at 09:12 AM, Nick Coghlan wrote: > > >I'm talking more generally - do you *really* want to be explaining that > >"bytes" behaves like a tuple of integers, while "bytes.bytes" behaves like > >a tuple of bytes? > > I would explain it differently though, using concrete examples. > > data = bytes(...) > for i in data: # iterate over data as integers > for i in data.bytes: # iterate over data as bytes > > But whatever. I just wish there was something better than iterbytes. There's actually another aspect to your idea, independent of the naming: exposing a view rather than just an iterator. So that view would actually be the bytes object done right? Funny :-) Will it have lazy slicing? bytes.sorry()? ;-) - C ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray
On Sun, Aug 17, 2014 at 2:41 PM, Barry Warsaw wrote: > I think the biggest API "problem" is that default iteration returns > integers > instead of bytes. That's a real pain. > what is really needed for this NOT to be a pain is a byte scalar. numpy has a scalar type for every type it supports -- this is a GOOD THING (tm): In [53]: a = np.array((3,4,5), dtype=np.uint8) In [54]: a Out[54]: array([3, 4, 5], dtype=uint8) In [55]: a[1] Out[55]: 4 In [56]: type(a[1]) Out[56]: numpy.uint8 In [57]: a[1].shape Out[57]: () The lack of a character type is a major source of "type errors" in python (the whole list of strings vs a single string problem -- both return a sequence when you index into them or iterate over them) Anyway, the character ship has long since sailed, but maybe a byte scalar would be a good idea? And FWIW, I think the proposal does make for a better, cleaner API. Whether that's worth the deprecation is not clear to me, though as someone whose been on the verge of making the leap to 3.* for ages, this isn't going to make any difference. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes & bytearray
On Mon, Aug 18, 2014 at 1:06 PM, Terry Reedy wrote: > The byte scalar is an int in range(256). Bytes is an array of such. > then why the complaint about iterating over bytes producing ints? Ye,s a byte owuld be pretty much teh same as an int, but it would have restrictions - useful ones. numpy has a scalar type for every type it supports -- this is a GOOD >> THING (tm): >> In [56]: type(a[1]) >> Out[56]: numpy.uint8 >> >> In [57]: a[1].shape >> Out[57]: () >> >> >> The lack of a character type is a major source of "type errors" in >> python (the whole list of strings vs a single string problem -- both >> return a sequence when you index into them or iterate over them) >> > > This is exactly what iterbytes would do -- yields bytes of size 1. as I understand it, it would yield a bytes object of length one -- that is a sequence that _happens_ to only have one item in it -- not the same thing. Note above. In numpy, when you index out of a 1-d array you get a scalar -- with shape == () -- not a 1-d array of length 1. And this is useful, as it provide s clear termination point when you drill down through multiple dimensions. I often wish I could do that with nested lists with strings at the bottom. [1,2,3] is a sequence of numbers "this" is a sequence of characters -- oops, not it's not, it's a sequence of sequences of sequences of ... I think it would be cleaner if bytes was a sequence of a scalar byte object. This is a bigger deal for numpy, what with its n-dimensional arrays and many reducing operations, but the same principles apply. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
> > but disallowing them in higher level >> > explicitly cross platform abstractions like pathlib. >> > I think the trick here is that posix-using folks claim that filenames are just bytes, and indeed they can be passed around with a char*, so they seem to be. but you can't possible do anything other than pass them around if you REALLY think they are just bytes. So really, people treat them as "bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and maybe a couple others)-is-ascii-compatible" If you assume that, then you could write a pathlib that would work. And in practice, I expect a lot of designed only for posix code works that way. But of course, this gets ugly if you go to a platform where filenames are not "bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and maybe a couple others)-is-ascii-compatible", like windows. I'm not sure if it's worth having a pathlib, etc. that uses this assumption -- but it could help us all write code that actually works with this screwy lack of specification. Antoine Pitrou wrote: > To elaborate specifically about pathlib, it doesn't handle bytes paths > but allows you to generate them if desired: > https://docs.python.org/3/library/pathlib.html#operators but that uses os.fsencode: Encode filename to the filesystem encoding As I understand it, the whole problem with some posix systems is that there is NO filesystem encoding -- i.e. you can't know for sure what encoding a filename is in. So you need to be able to pass the bytes through as they are. (At least as I read Armin Ronacher's blog) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Wed, Aug 20, 2014 at 9:52 PM, Cameron Simpson wrote: > On 20Aug2014 16:04, Chris Barker - NOAA Federal > wrote: > >> > So really, people treat them as >>> >> "bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and >> maybe a couple others)-is-ascii-compatible" >> > > As someone who fought long and hard in the surrogate-escape listdir() > wars, and was won over once the scheme was thoroughly explained to me, I > take issue with these assertions: they are bogus or misleading. > > Firstly, POSIX filenames _are_ just byte strings. The only forbidden > character is the NUL byte, which terminates a C string, and the only > special character is the slash, which separates pathanme components. > so they are "just byte strings", oh, except that you can't have a null, and the "slash" had better be code 47 (and vice versa). How is that different than "bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-is-ascii-compatible"? (sorry about the "maybe a couple others", I was too lazy to do my research and be sure). But my point is that python users want to be able to work with paths, and paths on posix are not strictly strings with a clearly defined encoding, but they are also not quite "just arbitrary bytes". So it would be nice if we could have a pathlib that would work with these odd beasts. I've lost track a bit as to whether the surrogate-escape solution allows this to all work now. If it does, then great, sorry for the noise. Second, a bare low level program cannot do _much_ more than pass them > around. It certainly can do things like compute their basename, or other > path related operations. > only if you assume that pesky slash == 47 thing -- it's not much, but it's not raw bytes either. The "bytes in some arbitrary encoding where at least the slash character > (and > maybe a couple others) is ascii compatible" notion is completely bogus. > There's only one special byte, the slash (code 47). There's no OS-level > need that it or anything else be ASCII compatible. I think > characterizations such as the one quoted are activately misleading. > code 47 == "slash" is ascii compatible -- where else did the 47 value come from? > I think we'd all agree it is nice to have a system where filenames are all > Unicode, but since POSIX/UNIX predates it by decades it is a bit late to > ignore the reality for such systems. well, the community could have gone to "if you want anything other than ascii, make it utf-8 -- but always, we're all a bunch of independent thinkers. But none of this is relevant -- systems in the wild do what they do -- clearly we all want Python to work with them as best it can. > There's no _external_ "filesystem encoding" in the sense of something > recorded in the filesystem that anyone can inspect. But there is the > expressed locale settings, available at runtime to any program that cares > to pay attention. It is a workable situation. > I haven't run into it, but it seem the folks that have don't think relying on the locale setting is the least bit workable. If it were, we woldn't be havin this discussion -- use the locale setting to decide how to decode filenames -- done. Oh, and I reject Nick's characterisation of POSIX as "broken". It's > perfectly internally consistent. It just doesn't match what he wants. > (Indeed, what I want, and I'm a long time UNIX fanboy.) > bug or feature? you decide. Internal consistency is a good start, but it punts the whole encoding issue to the client software, without giving it the tools to do it right. I call that "really hard to work with" if not broken. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Fri, Aug 22, 2014 at 10:09 AM, Glenn Linderman wrote: > What encoding does have a text file (an HTML, to be precise) with > text in utf-8, ads in cp1251 (ad blocks were included from different > files) and comments in koi8-r? >Well, I must admit the HTML was rather an exception, but having a > text file with some strange characters (binary strings, or paragraphs > in different encodings) is not that exceptional. > > That's not a text file. That's a binary file containing (hopefully > delimited, and documented) sections of encoded text in different > encodings. > > Allow me to disagree. For me, this is a text file which I can (and > do) view with a pager, edit with a text editor, list on a console, > search with grep and so on. If it is not a text file by strict Python3 > standards then these standards are too strict for me. Either I find a > simple workaround in Python3 to work with such texts or find a different > tool. I cannot avoid such files because my reality is much more complex > than strict text/binary dichotomy in Python3. > > First -- we're getting OT here -- this thread was about file and path names, not the contents of files. But I suppose I brought that in when I talked about writing file names to files... The first I'll mention is the one that follows from my description of what > your file really is: Python3 allows opening files in binary mode, and then > decoding various sections of it using whatever encoding you like, using the > bytes.decode() operation on various sections of the file. Determination of > which sections are in which encodings is beyond the scope of this > description of the technique, and is application dependent. > right -- and you would have wanted to open such file in binary mode with py2 as well, but in that case, you's have the contents in py2 string object, which has a few more convenient ways to work with text (at least ascii-compatible) than the py3 bytes object does. The third is to specify the UTF-8 with the surrogate escape error handler. > This allows non-UTF-8 codes to be loaded into memory. You, or algorithms as > smart as you, could perhaps be developed to detect and manipulate the > resulting "lone surrogate" codes in meaningful ways, or could simply allow > them to ride along without interpretation, and be emitted as the original, > into other files. > Just so I'm clear here -- if you write that back out, encoded as utf-8 -- you'll get the exact same binary blob out as came in? I wonder if this would make it hard to preserve byte boundaries, though. By the way, IIUC correctly, you can also use the python latin-1 decoder -- anything latin-1 will come through correctly, anything not valid latin-1 will come in as garbage, but if you re-encode with latin-1 the original bytes will be preserved. I think this will also preserve a 1:1 relationship between character count and byte count, which could be handy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Thu, Aug 21, 2014 at 7:42 PM, Oleg Broytman wrote: > On Thu, Aug 21, 2014 at 05:30:14PM -0700, Chris Barker - NOAA Federal < > chris.bar...@noaa.gov> wrote: > > This brings up the other key problem. If file names are (almost) > > arbitrary bytes, how do you write one to/read one from a text file > > with a particular encoding? ( or for that matter display it on a > > terminal) > >There is no such thing as an encoding of text files. So we just > write those bytes to the file So I write bytes that are encoded one way into a text file that's encoded another way, and expect to be abel to read that later? you're kidding, right? Only if that's he only thing in the file -- usually not the case with my text files. or output them to the terminal. I often do > that. My filesystems are full of files with names and content in > at least 3 different encodings - koi8-r, utf-8 and cp1251. So I open a > terminal with koi8 or utf-8 locale and fonts and some file always look > weird. But however weird they are it's possible to work with them. > Not for me (or many other users) -- terminals are sometimes set with ascii-only encoding, so non-ascii barfs -- or you get some weird control characters that mess up your terminal -- dumping arbitrary bytes to a terminal does not always "just work". > > And people still want to say posix isn't broken in this regard? > >Not at all! And broken or not broken it's what I (for many different > reasons) prefer to use for my desktops, servers, notebooks, routers and > smartphones, Sorry -- that's a Red Herring -- I agree, "broken" or "simple and consistent" is irrelevant, we all want Python to work as well as it can on such systems. The point is that if you are reading a file name from the system, and then passing it back to the system, then you can treat it as just bytes -- who cares? And if you add the byte value of 47 thing, then you can even do basic path manipulations. But once you want to do other things with your file name, then you need to know the encoding. And it is very, very common for users to need to do other things with filenames, and they almost always want them as text that they can read and understand. Python3 supports this case very well. But it does indeed make it hard to work with filenames when you don't know the encoding they are in. And apparently that's pretty common -- or common enough that it would be nice for Python to support it well. This trick is how -- we'd like the "just pass it around and do path manipulations" case to work with (almost) arbitrary bytes, but everything else to work naturally with text (unicode text). Which brings us to the "what APIs should accept bytes" question. I think that's been pretty much answered: All the low-level ones, so that protocol and library programmers can write code that works on systems with undefined filename encodings. But: casual users still need to do the normal things with file names and paths, and ideally those should work the same way on all systems. I think the way to do this is to abstract the path concept, like pathlib does. Back in the day, paths were "just strings", and that worked OK with py2 strings, because you could put arbitrary bytes in them. But the "py2 strings were perfect" folks seem to not acknowledge that while they are nice for matching the posix filename model, they were a pain in the neck when you needed to do somethign else like write them in to a JSON file or something. From my personal experience, non-ascii filenames are much easier to deal with if I use unicode for filenames everywhere (py2). Somehow, I have yet to be bitten by mixed encoding in filenames. So will using a surrogate-escape error handling with pathlib make all this just work? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 6:17 AM, Glenn Linderman wrote: > "cp1251 of utf-8 encoding" is non-sensical. Either it is cp1251 or it is > utf-8, but it is not both. Maybe you meant "or" instead of "of". I'd assume "or" meant there, rather than "of", it's a common typo. Not sure why 1251, specifically, but it's not uncommon for boundary code to attempt a decode that consists of something like "attempt UTF-8 decode, and if that fails, attempt an eight-bit decode". For my MUD clients, that's pretty much required; one of the servers I frequent is completely bytes-oriented, so whatever encoding one client uses will be dutifully echoed to every other client. There are some that correctly use UTF-8, but others use whatever they feel like; and since those naughty clients are mainly on Windows, I can reasonably guess that they'll be using CP-1252. So that's what I do: UTF-8, fall-back on 1252. (It's also possible some clients will be using Latin-1, but 1252 is a superset of that.) But it's important to note that this is a method of handling junk. It's not a design intention; this is for a situation where I really want to cope with any byte stream and attempt to display it as text. And if I get something that's neither UTF-8 nor CP-1252, I will display it wrongly, and there's nothing can be done about that. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 8:26 AM, Oleg Broytman wrote: > On Sat, Aug 23, 2014 at 07:04:20AM +1000, Chris Angelico > wrote: >> On Sat, Aug 23, 2014 at 6:17 AM, Glenn Linderman >> wrote: >> > "cp1251 of utf-8 encoding" is non-sensical. Either it is cp1251 or it is >> > utf-8, but it is not both. Maybe you meant "or" instead of "of". >> >> I'd assume "or" meant there, rather than "of", it's a common typo. >> >> Not sure why 1251, specifically > >This is the encoding of Russian Windows. Files and emails in Russia > are mostly in cp1251 encoding; something like 60-70%, I think. The > second popular encoding is cp866 (Russian DOS); it's used by Windows as > OEM encoding. Yeah, that makes sense. In any case, you pick one "most likely" 8-bit encoding and go with it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 7:02 PM, Stephen J. Turnbull wrote: > Chris Barker writes: > > > So I write bytes that are encoded one way into a text file that's encoded > > another way, and expect to be abel to read that later? > > No, not you. Crap software does that. Your MUD server. Oleg's > favorite web pages with ads, or more likely the ad servers. Just to clarify: Presumably you're referring to my previous post regarding my MUD client's heuristic handling of broken encodings. It's "my server" in the sense of the one that I'm connecting to, and not in the sense that I control it. I do also run a MUD server, and it guarantees that everything it sends is UTF-8. (Incidentally, that server has the exact same set of heuristics for coping with broken encodings from other clients. There's no escaping it.) Your point is absolutely right: mess like that is to cope with the fact that there's broken stuff out there. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: > Not sure why. Just put another module named "ssl" in sys.modules directly. > You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou wrote: > On Mon, 1 Sep 2014 23:24:39 +1000 > Chris Angelico wrote: >> On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: >> > Not sure why. Just put another module named "ssl" in sys.modules directly. >> > You can also monkeypatch the genuine ssl module. >> >> That has to be done inside the same process. But imagine this >> scenario: You have a program that gets invoked as root (or some other >> user than yourself), and you're trying to fiddle with what it sees. >> You don't have root access, but you can manipulate the file system, to >> the extent that your userid has access. What can you do to affect this >> other program? > > If you're root you shouldn't run untrusted code. See > https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > Sorry, I haven't been a very good maintainer for that buildbot (the main > > reason it never graduated to the "stable" list). If you send me your public > > SSH key, I can add it (I think - if not, I can ask Luke to do it). > > Alternatively, CentOS 6 may exhibit the same problem. > > I wonder how many of these buildbots could be maintained by the kind > of folks who show up on core-mentorship asking "how can I help?" > > Just a thought -- I wouldn't be surprised if the reaction is universal > horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" > > And of course most want to write code, not sysadm. Maintaining a buildbot isn't hard. (Although one thing I'm not sure of: If my bot goes down for an extended period of time, is any sort of automated email sent to me? I don't often check their status.) But it does mean a measure of trust in some external entity, or else some very careful rules (mainly firewall), which not every coder will know about. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On Sun, Aug 31, 2014 at 3:00 PM, Nick Coghlan wrote: > > However, we still think we should start providing pip by default to Python > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > serious +1 here. Just last night I was writing up notes for an intro to python class on getting started. (py2, :-( ) The "go download get-pip.py and run it with python" step(s) is a serious hiccup. Better than it's been for years, but sure would be nice to just "install python" then "use pip to install packages" As for the feature freeze of py2.7 -- this is really an addition to teh installers, not to python itself (or it can be seen that way, anyway) Thanks Ned for offering to back port for OS-X ! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Thu, Sep 4, 2014 at 3:32 AM, francis wrote: >> does mean a measure of trust in some external entity, or else some >> very careful rules (mainly firewall), which not every coder will know >> about. > > > Just curious, is there a way to mount the infrastructure the oder way > around? One sets a system polling for sources changes, if so it starts > a build an sends per e-mail the results to some address. I'm more talking about how there's a (virtual) machine that I run, executing code sent to me by someone else (the Python devs). That machine needs a certain measure of access to the internet (to fetch code, and to run certain tests), and the easy and naive way to set it up is to give it full access to everything, which is the trust that I spoke of. Firewalling that box so it can't see anything it's not allowed to see is certainly possible, but that requires sysadmin skills, not coder skills, hence my comment. And it's very easy to make a mistake that you won't notice - everything works just fine, because normally that buildbot won't do anything it shouldn't. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closure
On Thu, Sep 11, 2014 at 12:55 PM, Li Tianqing wrote: > Hello, > Can someone explain me why gc(CPython) can not collect recursive > closure's cycle reference? There is no __del__ here, why gc can not > collect? Can you start by explaining what's not getting collected and what is? Where's the cycle you're looking at? Also, this thread would do better on python-l...@python.org rather than python-dev. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1
On Sun, Sep 14, 2014 at 8:13 PM, Brynjar Smári Bjarnason
wrote:
> I am using Python 3.4.1 installed with Anaconda. I tried the following
> (expecting an OrderedDict as result):
>
from collections import namedtuple
NT = namedtuple("NT",["a","b"])
nt = NT(1,2)
print(vars(nt))
> {}
>
> so the result is an empty dict. In Python 3.3.2 (downgraded in the
> same Anaconda environment) results in:
>
print(vars(nt))
> OrderedDict([('a', 1), ('b', 2)])
For what it's worth, I can't reproduce the issue on trunk CPython
(built from default branch on Aug 21, so it's a little old now), nor
on 3.4.1 as currently shipping with Debian Jessie, nor with 3.4.0 on
Windows. So this may be an Anaconda issue. Do you know if it's meant
to be a patched install of Python?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull wrote: > Jim J. Jewett writes: > > > In terms of best-effort, it is reasonable to treat the smuggled bytes > > as representing a character outside of your unicode repertoire > > I have to disagree. If you ever end up passing them to something that > validates or tries to reencode them without surrogateescape, BOOM! > These things are the text equivalent of IEEE NaNs. If all you know > (as in the stdlib) is that you have "generic text", the only fairly > safe things to do with them are (1) delete them, (2) substitute an > appropriate replacement character for them, (3) pass the text > containing them verbatim to other code, and (4) reencode them using > the same codec they were read with. Don't forget, these are *errors*. These are bytes that cannot be correctly decoded. That's not something that has any meaning whatsoever in text; so by definition, the only things you can do are the four you list there (as long as "codec" means both the choice of encoding and the use of the surrogateescape flag). It's like dealing with control characters when you need to print something visually, except that they have an official solution [1] and surrogateescape is unofficial. They're not real text, so you have to deal with them somehow. The bytes might each represent one character. Several of them together might represent a single character. Or maybe they don't mean anything at all, and they're just part of a chunked data format... like I was finding in the .cwk files that I was reading this weekend (it's mostly MacRoman encoding, but the text is divided into chunks separated by \0\0 and two more bytes - turns out the bytes are chunk lengths, so they don't mean any sort of characters at all). You can't know. ChrisA [1] http://www.unicode.org/charts/PDF/U2400.pdf ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray wrote: > That isn't the case in the email package. The smuggled bytes are not > errors[*], they are literally smuggled bytes. But they're not characters, which is what Stephen and I were saying - and contrary to what Jim said about treating them as characters. At best, they represent characters but in some encoding other than the one you're using, and you have no idea how many bytes form a character or anything. So you can't, for instance, word-wrap the text, because you can't know how wide these unknown bytes are, whether they represent spaces (wrap points), or newlines, or anything like that. You can't treat them as characters, so while you have them in your string, you can't treat it as a pure Unicode string - it''s a Unicode string with smuggled bytes. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray wrote: >> You can't treat them as characters, so while you have them in your >> string, you can't treat it as a pure Unicode string - it''s a Unicode >> string with smuggled bytes. > > Well, except that I do. The email header parsing algorithms all work > fine if I treat the surrogate escaped bytes as 'unknown junk' and just > parse based on the valid unicode. (Unless the header is so garbled that > it can't be parsed, of course, at which point it becomes an invalid > header). Do what, exactly? As I understand you, you treat the unknown bytes as completely opaque, not representing any characters at all. Which is what I'm saying: those are not characters. If you, instead, represented the header as a list with some str elements and some bytes, it would be just as valid (though much harder to work with); all your manipulations are done on the str parts, and the bytes just tag along for the ride. > You are right about the wrapping, though. If a header with invalid > bytes (and in this scenario we *are* talking about errors) needs to > be wrapped, we have to first decode the smuggled bytes and turn it > into an 'unknown-8bit' encoded word before we can wrap the header. Yeah, and that's going to be a bit messy. If you get 60 characters followed by 30 unknown bytes, where do you wrap it? Dare you wrap in the middle of the smuggled section? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 3:55 AM, Jim Baker wrote: > Of course, if you do actually have a smuggled isolated low surrogate > FOLLOWED by a smuggled isolated high surrogate - guess what, the only > interpretation is a codepoint. Or perhaps more likely garbage. Of course it > doesn't happen so often, so maybe we are fine with the occasional bug ;) > > I personally suspect that we will resolve this by also supporting UCS-4 as a > representation in Jython 3.x for such Unicode strings, albeit with the > limitation that we have simply moved the problem to when we try to call Java > methods taking java.lang.String objects. > That'll cost efficiency, of course, but it'll guarantee correctness. And maybe, just maybe, you'll be able to put some pressure on Java itself to start supporting UCS-4 natively... One can dream. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote:
> Yes. I thought you were saying that one could not treat the string with
> smuggled bytes as if it were a string. (It's a string that can't be
> encoded unless you use the surrogateescape error handler, but it is
> still a string from Python's POV, which is the point of the error
> handler).
>
> Or, to put it another way, your implication was that there were no
> string operations that could be usefully applied to a string containing
> smuggled bytes, but that is not the case. (I may well have read an
> implication that was not there; if so I apologize and you can ignore the
> rest of this :)
Ahh, I see where we are getting confused. What I said was that you
can't treat the string as a *pure* Unicode string. Parts of it are
Unicode text, parts of it aren't.
> Basically, we are pretending that the each smuggled
> byte is single character for string parsing purposes...but they don't
> match any of our parsing constants. They are all "any character" matches
> in the regexes and what have you.
This is slightly iffy, as you can't be sure that one byte represents
one character, but as long as you don't much care about that, it's not
going to be an issue. I'm fairly sure you're never going to find an
encoding in which one unknown byte represents two characters, but
there are cases where it takes more than one byte to make up a
character (or the bytes are just shift codes or something). Does that
ever throw off your regexes? It wouldn't be an issue to a .* between
two character markers, but if you ever say .{5} then it might match
incorrectly.
I think we're in agreement here, just using different words. :)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.4.2rc1 Mac OS X
On Tue, Sep 23, 2014 at 10:27 AM, Ned Deily wrote: > Thanks for the feedback, Carol. Let us know via bugs.python.org of any > issues you see. BTW, the new installer format will be coming to Python > 2.7.9 as well. > Are the supported platforms going to be the same? i.e.: 10.3+ -- Intel32+PPC32 (Or is this deprecated already?) 10.6+ -- Intel32_Intel64 I'm actually wondering if it's time for a pure Intel64, probably 10.7+ Building extensions for Universal builds and older OS versions really is a pain... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote: > Donald Stufft wrote: >> One thing about *nix is even though you can’t write to your normal Python >> install location without root, invoking pip with permissions (assuming you >> have >> them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows >> have >> an equivalent or do you need to launch a whole new shell? > > Unfortunately not. The "easy way" is for the executable to declare that it > needs administrative privileges, and then the OS will take over and let you > approve/reject/sign-in/etc. according to your settings. When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Thu, Sep 25, 2014 at 9:00 AM, Donald Stufft wrote: > 1) Just always default to —user and add a —system or similar flag, this > is super easy to change but is a backwards incompatible change and > would need to go through a deprecation window. > Maybe would have been the way to go to begin with, but I think backwards compatibility should probably trump "better" -- even with a deprecation window. > 2) Switch to —user based on if the user has permission to write to the > site-packages or not. > ouch -- no. Why not a clear error message if pip can't write to site-packages -- something like: """ pip doesn't have permissions to write to the system location. If you want to install this package system-wide, you need to run pip with admin priviledges (and example here if it's easy), if you want to install for this user only, pass the "--user" flag to pip install """ "In the face of ambiguity, refuse the temptation to guess." -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX
On Fri, Sep 26, 2014 at 9:53 AM, Antoine Pitrou wrote:
>> In other words, os.system is *already* an attack vector, unless you only
>> use it with trusted strings. I don't think the bash env vulnerability
>> adds to the attack surface.
>>
>> Have I missed something?
>
> The part where the attack payload is passed through the environment, not
> through hypothetical user-injected command-line arguments.
Which means this also affects anything that invokes shell scripts (if
they use bash, rather than sh), even if it doesn't use os.system().
I'm just in process of checking and patching my systems (most of them
are just 'apt-get update; apt-get upgrade' followed by a quick check),
and can confirm that it does happen in Python. All you have to do is
invoke bash, either explicitly or through your target's shebang.
>>> import os, subprocess
>>> os.environ["HAHA"]="() { :;}; echo This is crafted from the environment."
>>> subprocess.call(["./test.sh"])
This is crafted from the environment.
This is from my test script.
0
>>> open("./test.sh").read()
'#!/bin/bash\necho This is from my test script.\n\n'
>>> subprocess.call(["bash","-c","echo This is from the command line."])
This is crafted from the environment.
This is from the command line.
0
But this is a bash issue, not a Python one.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX
On Fri, Sep 26, 2014 at 10:29 AM, Devin Jeanpierre
wrote:
> As I understand it, if the attacker can help specify the environment
> (e.g. this is a CGI script), and you run os.system('echo hi'), you can
> get pwned. Even safe uses of os.system are vulnerable unless you point
> /bin/sh at a secure shell (e.g. patched bash).
/bin/sh may well not point to bash anyway - it doesn't on any of my
systems. Debian provides dash instead, much faster than bash. But if
you're invoking a script that calls for bash, then it's vulnerable.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Thu, Sep 25, 2014 at 5:38 PM, Donald Stufft wrote: > 2) Switch to —user based on if the user has permission to write to the >> site-packages or not. >> > > ouch -- no. Why not a clear error message if pip can't write to > site-packages -- something like: > > I fairly strongly believe that the current default is doing a great > disservice > to users. I believe that for *most* people --user is the correct option for > them to be using and the fact that it's not the default and requires opt in > is a historical artifact more than anything else. > OK -- fine -- I think that history may be important -- at least for Py2, but I agree that --user is a better default. But what I'm objecting to is is switching the install mode based on the permissions that the user happens to be running with at the time. None of us should be routinely running as admin. So what I do (and I image a LOT of people do) is try to do whatever I need to do, and only if I get an error to I switch to admin mode (or add sudo, or ???). What I'm suggesting is that folks that simply expect stuff to get installed into the usual system python is not thinking ahead of time "oh, I need be admin for this", but rather, simply do it, and then, if you get a permission error, make the switch to either admin in mode, or add --user. If we do switch the default to --user, then this issue goes away. In short -- too much magic is bad. All this is also making me think that virtualenv and friends is the real solution to user installed packages anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.4.2rc1 online docs: cannot access tkinter, idle pages
On Mon, Oct 6, 2014 at 6:49 AM, Terry Reedy wrote: > I can access every page I have tried except for the tkinter and idle pages, > which either give interminable 'Connecting...' or in one try, 404. > https://docs.python.org/3/library/tkinter.html > https://docs.python.org/3/library/idle.html Is this still the case? I just tried them and they work for me. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing 2.7.x
On Tue, Oct 7, 2014 at 4:33 AM, Skip Montanaro wrote: > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily wrote: >> So 2.7.x is not "security only" and wouldn't reach that stage until 2020 >> under current policy. > > Apparently no other 2.x release qualifies as "security only" at this > point? I would have expected at least 2.6 to fall into that category. Was until about a year ago. http://legacy.python.org/dev/peps/pep-0361/ ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
On Fri, Oct 10, 2014 at 11:52 AM, Jesus Cea wrote: > Actually, it doesn't work in Python 2 either. It never supported > international mailbox names. > > Should I dare to suggest to port this to 2.7, since 2.7 is special and > will be supported for a long time?. Or maybe this is something like > "Yes, Python 2 is broken, the real deal is Python 3"? :). That's ultimately up to the release manager, but IMO this sounds like a bug to be fixed, more than a feature being added. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
On Fri, Oct 10, 2014 at 3:05 PM, Glenn Linderman wrote: > There are still lots of idiotic web sites that assume everything in front of > the @ must be a letter, digit, dot, or hyphen, and even some that only > permit one dot after the @... even though for 30 years or so, the RFCs have > permitted a nice variety of other special characters, although not all of > them. And heaps that require a dot after the @, which is definitely not a requirement. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Sat, Oct 11, 2014 at 2:45 AM, Jesus Cea wrote: > On 03/09/14 02:37, Antoine Pitrou wrote: > >> I'm not sure that's an answer to the problem. What we need is not more >> machines, but dedicated buildbot maintainers. > > I would love to get an email if my buildbots are consistently RED for a > few hours. > > In the past Antoine, Victor and others pinged me about issues in my > buildbots (OpenIndiana). I feel shame and I think it is not very > productive either. Likewise. In the earliest days of my buildbots, I kept a very close eye on them (which was good; they had some occasional network glitches, which I had to resolve), but once they became stable, I stopped watching. Now, I almost never check them - as long as the VM's running, I basically assume all's well. It'd be very helpful if a computer could auto-email any time they're down for any length of time. Is the info available in some way? Could I write a little monitor at my end that asks every hour if my buildbots can be seen? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 7:50 AM, Steve Dower wrote: > Ray Donnelly wrote: >> What is it that you >> are afraid of if CPython can be compiled out of the box using >> mingw/MinGW-w64? Why are you fighting so hard against having option. > > I'm afraid of users having numpy crash because they're using an MSVC CPython > instead of a mingw CPython. I'm afraid of users not being able to use library > A and library B at the same time because A requires MSVC CPython and B > requires mingw CPython. (I can produce more examples if you like, but the > general concern is having a fragmented community, as I said in my previous > post.) > It might fragment the community to have multiple different binary distributions. But it ought to be possible for any person/organization to say "We're going to make our own build of Python, with these extension modules, built with this compiler, targeting this platform", and do everything from source. That might mean they can no longer take the short-cut of "download someone's MSVC-built extension and use it as-is", but they should be able to take anyone's extension and build it on their chosen compiler. Having MinGW as a formally supported platform would make life a lot easier for people who want to test CPython patches, for instance - my building and testing of PEP 463-enhanced Python was Linux-only, because I didn't want to try to set up an entire new buildchain just to try to get a Windows binary going. There's absolutely no need for that to be binary-compatible with anything else; as long as it'll run the standard library, it'll do. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou wrote: > And how do you know that it would have worked with MSVC if you only use > MinGW? > If you want to ensure compatibility with MSVC, you must build with MSVC. > There's no working around that. Precisely. If you build with MinGW, you can't ensure compatibility with MSVC. Reread my post: I gave two examples of situations where that isn't a problem. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou wrote: > How do you know this isn't a problem, since you haven't *tested* with > MSVC? > Why on Earth would you want to test your PEP work with an unsupported > Windows compiler and runtime, rather than with the officially supported > compiler and runtime? This discussion revolved around supporting MinGW in addition to MSVC. If it had been supported when I was doing that, I could have spun myself up a Windows build and tested it. Since it was (and so far still is) not, the hassle of hunting down a valid MSVC that could build for Win XP (as that's what my test box runs) was simply not worthwhile. My point is that there is no community fragmentation happening here; the only fragmentation is of binary distribution of extension modules, and there are several ways in which this needn't be a problem. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou wrote: > My point is that your "Windows build" would not have the same behaviour > as a MSVC-produced Windows build, and so testing it with it would not > certify that your code would actually be compatible with genuine > MSVC builds of CPython, which we will not stop supporting. > So you're saying it's impossible to support two compilers? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impact of Windows PowerShell OneGet ?
On Thu, Oct 30, 2014 at 6:34 AM, Glenn Linderman wrote: > (I'm actually not sure if *nix package managers allow multiple repositories > or not, but from the way people talk about them, it always sounds like a > "distribution" also provides "a repository" of additional packages). I'm fairly sure they all do. Certainly with apt (the Debian package manager), it's common to add additional repositories; for instance, PostgreSQL can be obtained either from the default Debian repos or from Postgres's own hosting (which usually has more versions available). A distribution will always provide a repository, and there are plenty of distros that provide only a small repo and chain to upstream for most packages - for instance AntiX has its own, and then pulls in debian.org and a few others. Adding a local-network repo is fairly straight-forward. Most likely, OneGet won't replace pip/PyPI, any more than apt or yum does; but it may be worth having Python itself available that way. That might simply mean having someone package up Python and put it on an appropriate server, or maybe python.org could end up hosting a repo. I've no idea what "trusted" will mean; in the case of apt, any sysadmin can deem any repo to be trusted (by importing its key), but this might be more along the lines of "only curated packages" or something. To what extent will Windows 10 users expect all their software to come via OneGet? That's the question. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] py34 makes it harder to read all of a pty
On Thu, Nov 13, 2014 at 6:34 PM, Charles-François Natali wrote: > 2014-11-12 22:16 GMT+00:00 Buck Golemon : >> This is due to the fix for issue21090, which aimed to un-silence errors >> which previously went unheard. The fix is for me, as a user, to write a loop >> that uses os.read and interpretes EIO as EOF. This is what I had hoped >> file.read() would do for me, however, and what it used to do in previous >> pythons. > > > There's no reason for read() to interpret EIO as EOF in the general > case: it was masked in previous versions because of a mere bug. The > behavior is now correct, although being able to retrieve the data read > so far in case of a buffered read could be useful. Every change breaks someone's workflow. http://xkcd.com/1172/ ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On Sun, Nov 16, 2014 at 2:40 AM, Paul Moore wrote: > On 15 November 2014 15:17, Benjamin Peterson wrote: >> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >>> On 15 Nov 2014 10:10, "Paul Moore" wrote: >>> > >>> > > Incidentally, it would be really useful if python.org provided stable >>> > > url's that always redirected to the latest .msi installers, for >>> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >>> > > scraping the web site) for this. >>> > >>> > https://www.python.org/ftp/python/$ver/python-$ver.msi >>> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >>> >>> Right, but what's the URL for "the latest 2.7.x release" or "the latest >>> 3.x.x release"? >> >> The website has an API you know. > > Um, no. Where can I find out about it? I didn't know either, and I've been pointing people to the .msi installers periodically - just by manually hunting down the latest link and recommending it. If there's an easy way to define the two tokens Nathaniel described, I'd like to hear it too. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 7:48 AM, MRAB wrote:
> The PEP says """any generator that depends on an implicitly-raised
> StopIteration to terminate it will have to be rewritten to either catch
> that exception or use a for-loop"""
>
> Shouldn't that be "... explicitly-raised ...", because returning raises
> StopIteration implicitly? ("raise StopIteration" is explicit)
The point here is primarily about some other function (maybe a
next(iter), or maybe something else entirely) raising StopIteration.
(If it explicitly raises StopIteration right there in the generator,
it can be trivially converted into a return statement, anyway.) The
return statement is an explicit indication that the generator should
now return; permitting a StopIteration to bubble up through and out is
the implicit option; but the 'plicitness' isn't necessarily obvious.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 3:45 AM, Nick Coghlan wrote:
> The part I found most compelling was when you pointed out that in the
> special method implementations, the normal return path was always spelled
> with "return", while the "value missing" result was indicated with a special
> kind of exception (StopIteration, AttributeError, IndexError or KeyError),
> and then any other exception was consider unexpected.
>
> Generators add the third notion of being able to suspend execution via
> "yield", which then left them with two different ways of spelling
> termination inside the frame: "return" OR "raise StopIteration". The second
> spelling ("raise StopIteration") is then inherently surprising, as it's
> entirely redundant, *except* in that it allows you to effectively have a
> "hidden return" in a generator frame that can't be done anywhere else.
(The above was said on -ideas, but discussion is now moving to -dev,
so I hope it's okay to send the response there.)
Added a paragraph to the PEP draft:
https://github.com/Rosuav/GenStopIter/commit/695961
Should today's date be added to the Post-History?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: > It would also be useful if we could extend the PEP with some examples of the > various categories of fixes that can be applied easily, e.g. a few examples > of "raise StopIteration" directly in a generator that can be replaced with > "return" (or omitted, if it's at the end); a few examples of situations > where "yield from" can supply an elegant fix (and an alternative for code > that needs to be backward compatible with Python 3.2 or 2.7); and finally > (to be honest) an example of code that will require being made more > complicated. > > Oh, and it would also be nice if the PEP included some suggested words that > 3rd party educators can use to explain the relationship between > StopIteration and generators in a healthier way (preferably a way that also > applies to older versions). > > Chris, are you up to drafting these additions? Sure, no problem. Will knock something up shortly. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 9:04 AM, Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > wrote: >> >> On 20.11.14 21:58, Antoine Pitrou wrote: >>> >>> To me "generator_return" sounds like the addition to generator syntax >>> allowing for return statements (which was done as part of the "yield >>> from" PEP). How about "generate_escape"? >> >> >> Or may be "generator_stop_iteration"? > > > Or just "generator_stop"? Unrelated to the GeneratorExit exception. I don't think that'll be a problem though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 10:34 AM, Chris Angelico wrote: > On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: >> It would also be useful if we could extend the PEP with some examples of the >> various categories of fixes that can be applied easily, e.g. a few examples >> of "raise StopIteration" directly in a generator that can be replaced with >> "return" (or omitted, if it's at the end); a few examples of situations >> where "yield from" can supply an elegant fix (and an alternative for code >> that needs to be backward compatible with Python 3.2 or 2.7); and finally >> (to be honest) an example of code that will require being made more >> complicated. >> >> Oh, and it would also be nice if the PEP included some suggested words that >> 3rd party educators can use to explain the relationship between >> StopIteration and generators in a healthier way (preferably a way that also >> applies to older versions). >> >> Chris, are you up to drafting these additions? > > Sure, no problem. Will knock something up shortly. Examples and explanatory text added. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger wrote: > Also, the proposal breaks a reasonably useful pattern of calling > next(subiterator) inside a generator and letting the generator terminate > when the data stream ends. Here is an example that I have taught for > years: > > def izip(iterable1, iterable2): > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 2:58 AM, Steven D'Aprano wrote: > Since zip() is documented as > halting on the shorter argument, it can't raise an exception. So what > other options are there apart from silently consuming the value? Sure, it's documented as doing that. But imagine something that isn't a well-known function - all you have is someone writing a generator that calls next() in multiple places. Is it obvious that it it'll silently terminate as soon as any one of those iterators is exhausted? In many cases, an exception (probably ValueError?) would be the most obvious response. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 3:37 AM, Donald Stufft wrote: > I don’t have an opinion on whether it’s enough of a big deal to actually > change > it, but I do find wrapping it with a try: except block and returning easier > to understand. If you showed me the current code unless I really thought about > it I wouldn't think about the fact that the next() calls can cause the > generator to terminate. And don't forget, by the way, that you can always request the current behaviour by explicitly wrapping the body of the function in try/except: def izip(iterable1, iterable2): try: it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 except StopIteration: pass That's exactly what current behaviour does, and if you think that that try block is too broad and should be narrowed, then you should support this proposal :) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam wrote: > > OPTION 1: > > Make comprehensions act more like generator expressions. > > It would mean a while loop in the object creation point is converted to a > for loop. (or something equivalent.) > > Then both a comprehension and a generator expressions can be viewed as > defining iterators, with the same behaviour, rather than comprehensions > defining the body of the loop, which has the different but valid behaviour > of StopIteration escaping. > > This would make explaining them a *lot* easier as they become the same thing > used in a different context, rather than two different things used in what > appears to be similar contexts. > > > I think this fits with what Guido wants, but does so in a narrower scope, > only effecting comprehensions. StopIteration is less likely to leak out. A list comp is usually compared to a statement-form loop. def stop(): raise StopIteration lst = [stop() for x in (1,2,3)] lst = [] for x in (1,2,3): lst.append(stop()) At the moment, these are identical (virtually; the pedantic will point out that 'x' doesn't leak out of the comprehension) - in each case, the exception raised by the body will bubble up and be printed to the console. But a generator expression is different: lst = list(stop() for x in (1,2,3)) In this form, lst is an empty list, and nothing is printed to the console. Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. This is backwards. Most of Python is about catching exceptions as narrowly as possible: you make your "try" blocks small, you use specific exception types rather than bare "except:" clauses, etc, etc, etc. You do your best to catch ONLY those exceptions that you truly understand, unless you're writing a "catch, log, and reraise" or "catch, log, and go back for more work" generic handler. A 'for' loop currently is written more-or-less like this: for var in expr: body it = iter(expr) while True: try: var = next(it) except StopIteration: break body But this suggestion of yours would make it become: it = iter(expr) while True: try: var = next(it) body except StopIteration: break I believe this to be the wrong direction to make the change. Instead, generator expressions should be the ones to change, because that's a narrowing of exception-catching scope. Currently, every generator function is implicitly guarded by "try: .. except StopIteration: pass". This is allowing errors to pass silently, to allow a hack involving non-local control flow (letting a chained function call terminate a generator) - violating the Zen of Python. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >> Making comprehensions work more like generator expressions >> would, IMO, imply making the same change to all for loops: having a >> StopIteration raised by the body of the loop quietly terminate the >> loop. > > > I'm not suggesting making any changes to generator expressions or for loops > at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 7:30 AM, Raymond Hettinger wrote: > Legitimate Use Cases for Raising StopIteration in a Generator > > > In a producer/consumer generator chain, the input generator signals > it is done by raising StopIteration and the output generator signals > that it is done by raising StopIteration (this isn't in dispute). > > That use case is currently implemented something like this: > > def middleware_generator(source_generator): > it = source_generator() > input_value = next(it) > output_value = do_something_interesting(input_value) > yield output_value Does your middleware_generator work with just a single element, yielding either one output value or none? Or is it more likely to be iterating over the source generator: def middleware_generator(source_generator): for input_value in source_generator: yield do_something_interesting(input_value) MUCH tidier code, plus it's safe against unexpected StopIterations. Even if you can't do it this way, all you need is to stick a 'break' at the end of the loop, if the try/except is so abhorrent. Wouldn't be the first time I've seen a loop with a hard break at the end of it. > This doesn't make the code better in any way. The new code > is wordy, slow, and unnecessarily convoluted: > > def middleware_generator(source_generator): > it = source_generator() > try: > input_value = next(it) > except StopIteration: > return # This causes StopIteration to be reraised > output_value = do_something_interesting(input_value) > yield output_value The raising of StopIteration here is an implementation detail; you might just as well have a comment saying "This causes the function to set an exception state and return NULL", which is what happens at the C level. What happens if do_something_interesting happens to raise StopIteration? Will you be surprised that this appears identical to the source generator yielding nothing? That's current behaviour. You don't have the option of narrowing the try/except scope as you've done above. > Is next() Surprising? > - > > If someone in this thread says that they were suprised that next() could > raise StopIteration, I don't buy it. Agreed, I don't think that's surprising to anyone. > Being able to consume a value from an iterator stream is a fundamental > skill and not hard to learn (when I teach iterators and generators, the > operation of next() has never been a stumbling block). In anything other than a generator, you're expected to cope with two possible results from next(): a returned value or a raised StopIteration. Suppose you want to read a file with a header - you'd need to do something like this: def process_file(f): f = iter(f) try: header=next(f) except StopIteration: cope_somehow_maybe_return for line in f: process_line_with_headers(line, header) Currently, *if and only if* you're writing a generator, you have an implicit "except StopIteration: return" there. Anywhere else, you need to catch that exception. Iterators are an implementation detail of generators. There is no particular reason for a generator author to be aware of iterator protocol, any more than this class needs to be: class X: def __iter__(self): return iter([1,2,3,4]) It's perfectly iterable, just as a generator is, and it knows nothing about StopIteration. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam wrote:
> Se we have these...
>
> Tuple Comprehension (...)
> List Comprehension [...]
> Dict Comprehension {...} Colon make's it different from sets.
> Set Comprehension {...}
>
> I don't think there is any other way to create them. And they don't actually
> expand to any python code that is visible to a programmer. They are self
> contained literal expressions for creating these few objects.
Hmmm, there's no such thing as tuple comprehensions. Are you confusing
comprehension syntax (which has a 'for' loop in it) with
tuple/list/etc display, which doesn't?
lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension
> Here is what I think(?) list comps do currently.
>
> lst = [expr for items in itr if cond]
>
> Is the same as.
>
> lst = []
> for x in itr: # StopIteration caught here.
> if cond: # StopIteration bubbles here.
> lst.append(expr) # StopIteration bubbles here.
Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.
> And it would be changed to this...
>
> def comp_expression():
> for x in itr: # StopIteration caught here.
> if cond:
>list.append(expr)
>
> # StopIteration from cond and expr caught here.
> lst = list(x for x in comp_expression())
That wouldn't quite work, but this would:
def ():
ret = []
try:
for x in itr:
if cond:
ret.append(x)
except StopIteration:
pass
return ret
lst = ()
(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam wrote: > > > On 11/22/2014 06:20 PM, Chris Angelico wrote: >> >> Hmmm, there's no such thing as tuple comprehensions. > > Just didn't think it through quite well enough. But you are correct, that > would be a generator expression. > > One less case to worry about. :-) Ah right, no probs. >>> >And it would be changed to this... >>> > >>> > def comp_expression(): >>> > for x in itr: # StopIteration caught here. >>> > if cond: >>> >list.append(expr) >>> > >>> > # StopIteration from cond and expr caught here. >>> > lst = list(x for x in comp_expression()) > > Right, the list.append() should be a yield(expr). In that case, your second generator expression is entirely redundant; all you want is list(comp_expression()). But the example doesn't say *why* this version should terminate on a StopIteration raised by expr, when the statement form would print an exception traceback. > The real question is how much breakage would such a change make? That will > probably need a patch in order to test it out. There's one attached here: http://bugs.python.org/issue22906 It doesn't create a __future__ directive, just applies the change globally. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 12:11 PM, Raymond Hettinger
wrote:
> The worry is that your proposal intentionally breaks that code which is
> currently
> bug free, clean, fast, stable, and relying on a part of the API that has
> been
> guaranteed and documented from day one.
(I'd just like to mention that this isn't "my proposal", beyond that
I'm doing the summarizing and PEP writing. The proposal itself is
primarily derived from one of Guido's posts on -ideas.)
> Here's one from Fredrick Lundh's ElementTree code in the standard library
> (there are several other examples besides this one in his code are well):
>
> def iterfind(elem, path, namespaces=None):
> # compile selector pattern
> cache_key = (path, None if namespaces is None
> else tuple(sorted(namespaces.items(
> if path[-1:] == "/":
> path = path + "*" # implicit all (FIXME: keep this?)
> try:
> selector = _cache[cache_key]
> except KeyError:
> if len(_cache) > 100:
> _cache.clear()
> if path[:1] == "/":
> raise SyntaxError("cannot use absolute path on element")
> next = iter(xpath_tokenizer(path, namespaces)).__next__
> token = next()
> selector = []
> while 1:
> try:
> selector.append(ops[token[0]](next, token))
> except StopIteration:
> raise SyntaxError("invalid path")
> try:
> token = next()
> if token[0] == "/":
> token = next()
> except StopIteration:
> break
> _cache[cache_key] = selector
> # execute selector pattern
> result = [elem]
> context = _SelectorContext(elem)
> for select in selector:
> result = select(context, result)
> return result
Most of the next() calls are already guarded with try/except; from
what I can see, only the first one isn't. So this wouldn't be much of
a change here.
> And here is an example from the pure python version of one of the itertools:
>
> def accumulate(iterable, func=operator.add):
> 'Return running totals'
> # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
> # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
> it = iter(iterable)
> total = next(it)
> yield total
> for element in it:
> total = func(total, element)
> yield total
Again, only the first one needs protection, and all that happens is
that there's clearly a control flow possibility here (that the first
"yield total" might not be reached). Currently, *any* function call
has the potential to be a silent control flow event.
> And here is an example from Django:
>
> def _generator():
> it = iter(text.split(' '))
> word = next(it)
> yield word
> pos = len(word) - word.rfind('\n') - 1
> for word in it:
> if "\n" in word:
> lines = word.split('\n')
> else:
> lines = (word,)
> pos += len(lines[0]) + 1
> if pos > width:
> yield '\n'
> pos = len(lines[-1])
> else:
> yield ' '
> if len(lines) > 1:
> pos = len(lines[-1])
> yield word
> return ''.join(_generator())
When you split a string, you're guaranteed at least one result, ergo
'it' is guaranteed to yield at least one word. So this one wouldn't
need to be changed - it can't possibly raise RuntimeError.
> I could scan for even more examples, but I think you get the gist.
> All I'm asking is that you consider that your proposal will do more
> harm than good. It doesn't add any new capability at all.
> It just kills some code that currently works.
I have considered it, and I'm not convinced that it will. I see lots
of people saying "code will have to be changed", but that's exactly
the same concern that people raise about switching from the sloppy Py2
merging of text and bytes to the strict Py3 separation - yes, code has
to be changed, but it's definitely much better to have immediate and
obvious failures when something's wrong than to have subtle behavioral
changes.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 4:59 PM, Nick Coghlan wrote: > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). The learning curve for source control as a concept is pretty steep. Once someone's learned one DVCS, learning another is much easier, and I don't know that it makes a lot of difference whether you learn git and then hg, or hg and then git. I learned git first, and mastered hg basics by keeping a Rosetta Stone chart handy; given that the document I was reading was intended for the reverse case, I expect it's not too hard to get the basics of git once you know hg. Just as git offers few advantages over hg, hg offers few advantages over git. If git and GitHub are where most people are, I would support using them for Python. I'm one of those PEP draft authors who starts with his own repo on GitHub and sends drafts in, and Guido had to remind me that I can simply test my edits in the peps repo before posting them (to make sure I haven't mucked up the markup); if the peps repo were itself hosted on GitHub, or at least in a git repo, I could have a workflow that directly incorporates that, instead of being "off to the side" with periodic manual imports. That said, it does make sense for CPython *itself* to use Mercurial, simply because it's written in Python. I don't know how strong that philosophical argument is with people, but I wouldn't argue against it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 5:03 PM, Wes Turner wrote: > hg imuutability is certainly a primarily attractive feature; > along with the keyring support. What exactly do you mean by immutability? Are you talking about how git allows a "force push" that can destroy data? That can be rejected in a repo's hook scripts; also, I'm fairly sure I remember reading somewhere about how to do that with hg, too. It's all bits of data inside computers; nothing's immutable. Both DVCSes pledge to make it obvious any time something is altered, not to make it impossible to alter. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 2:11 AM, Ethan Furman wrote: > On 11/22/2014 08:53 PM, Guido van Rossum wrote: >> >> In order to save everyone's breath, I am *accepting* the proposal of PEP >> 479. > > Excellent. > > Chris, thank you for your time, effort, and thoroughness in championing this > PEP. > Thank you, it's nice to have a successful one to counterbalance the "failure" of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I'm actually quite impressed with how on-topic the discussion threads went. They didn't ramble nearly as much as I thought they would. Thank you to all who participated, contributed suggestions, complained about the existing text, and generally worked hard to make sure I had more than enough material to draw on! Particular thanks to Guido, pushing changes through and being patient with my mistakes in markup, and adding content directly to the document. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam wrote:
> With the passage of the PEP, it will change what is different about them
> once it's in full effect. The stop hack won't work in both, and you may get
> a RuntimeError in generator expressions where you would get StopIteration in
> list-comps. (Is this correct?)
"them" being list comps and generator expressions?
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
>>> {1:2,3:4}[50]
Traceback (most recent call last):
File "", line 1, in
KeyError: 50
>>> [1,2,3,4][50]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon wrote:
> Hi,
>
> I have serious concerns about this PEP, and would ask you to reconsider it.
Hoping I'm not out of line in responding here, as PEP author. Some of
your concerns (eg "5 days is too short") are clearly for Guido, not
me, but perhaps I can respond to the rest of it.
> [ Very short summary:
> Generators are not the problem. It is the naive use of next() in an
> iterator that is the problem. (Note that all the examples involve calls to
> next()).
> Change next() rather than fiddling with generators.
> ]
>
> StopIteration is not a normal exception, indicating a problem, rather it
> exists to signal exhaustion of an iterator.
> However, next() raises StopIteration for an exhausted iterator, which really
> is an error.
> Any iterator code (generator or __next__ method) that calls next() treats
> the StopIteration as a normal exception and propogates it.
> The controlling loop then interprets StopIteration as a signal to stop and
> thus stops.
> *The problem is the implicit shift from signal to error and back to signal.*
The situation is this: Both __next__ and next() need the capability to
return literally any object at all. (I raised a hypothetical
possibility of some sort of sentinel object, but for such a sentinel
to be useful, it will need to have a name, which means that *by
definition* that object would have to come up when iterating over the
.values() of some namespace.) They both also need to be able to
indicate a lack of return value. This means that either they return a
(success, value) tuple, or they have some other means of signalling
exhaustion.
I'm not sure what you mean by your "However" above. In both __next__
and next(), this is a signal; it becomes an error as soon as you call
next() and don't cope adequately with the signal, just as KeyError is
an error.
> 2. The proposed solution does not address this issue at all, but rather
> legislates against generators raising StopIteration.
Because that's the place where a StopIteration will cause a silent
behavioral change, instead of cheerily bubbling up to top-level and
printing a traceback.
> 3. Generators and the iterator protocol were introduced in Python 2.2, 13
> years ago.
> For all of that time the iterator protocol has been defined by the
> __iter__(), next()/__next__() methods and the use of StopIteration to
> terminate iteration.
>
> Generators are a way to write iterators without the clunkiness of explicit
> __iter__() and next()/__next__() methods, but have always obeyed the same
> protocol as all other iterators. This has allowed code to rewritten from one
> form to the other whenever desired.
>
> Do not forget that despite the addition of the send() and throw() methods
> and their secondary role as coroutines, generators have primarily always
> been a clean and elegant way of writing iterators.
This question has been raised several times; there is a distinct
difference between __iter__() and __next__(), and it is only the
latter which is aware of StopIteration. Compare these three classes:
class X:
def __init__(self): self.state=0
def __iter__(self): return self
def __next__(self):
if self.state == 3: raise StopIteration
self.state += 1
return self.state
class Y:
def __iter__(self):
return iter([1,2,3])
class Z:
def __iter__(self):
yield 1
yield 2
yield 3
Note how just one of these classes uses StopIteration, and yet all
three are iterable, yielding the same results. Neither Y nor Z is
breaking iterator protocol - but neither of them is writing an
iterator, either.
> 4. Porting from Python 2 to Python 3 seems to be hard enough already.
Most of the code broken by this change can be fixed by a mechanical
replacement of "raise StopIteration" with "return"; the rest need to
be checked to see if they're buggy or unclear. There is an edge case
with "return some_value" vs "raise StopIteration(some_value)" (the
former's not compatible with 2.7), but apart from that, the
recommended form of code for 3.7 will work in all versions of Python
since 2.2.
> 5. I think I've already covered this in the other points, but to reiterate
> (excuse the pun):
> Calling next() on an exhausted iterator is, I would suggest, a logical
> error.
How do you know that it's exhausted, other than by calling next() on it?
> It also worth noting that calling next() is the only place a StopIteration
> exception is likely to occur outside of the iterator protocol.
This I agree with.
> An example
> --
>
> Consider a function to return the value from a set with a single member.
> def value_from_singleton(s):
> if len(s) < 2: #Intentional error here (should be len(s) == 1)
>return next(iter(s))
> raise ValueError("Not a singleton")
>
> Now suppose we pass an empty set to value_from_singleton(s), then we get a
> StopIteration exception, which is a bit weird, but not too bad.
Only a little we
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 10:18 AM, Ron Adam wrote:
>> The stop hack won't work in either (currently it does work in
>> genexps), but you'd get a different exception type if you attempt it.
>> This is correct. It's broadly similar to this distinction:
>>
> >>>{1:2,3:4}[50]
>>
>> Traceback (most recent call last):
>>File "", line 1, in
>> KeyError: 50
>
> >>>[1,2,3,4][50]
>>
>> Traceback (most recent call last):
>>File "", line 1, in
>> IndexError: list index out of range
>
>
> Comprehensions insert/append items, so you wouldn't get those unless you are
> reading from another object and would bubble out of generators too. Which is
> good because it's most likely an error that needs fixing.
My point is that doing the same errant operation on a list or a dict
will give different exceptions. In the same way, calling next() on an
empty iterator will raise StopIteration normally, but might raise
RuntimeError instead. It's still an exception, it still indicates a
place where code needs to be changed (unlike, say, a ValueError when
doing type conversions, which often means *data* needs to be changed),
so it doesn't hugely matter _which_ exception is raised. Also, I'm
hoping that someone can improve on my patch by having the full
traceback from the original StopException become visible - then, the
RuntimeError's __context__ will give all the info you would want.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Mon, Nov 24, 2014 at 2:42 PM, Guido van Rossum wrote: > Then there's this. http://git-man-page-generator.lokaltog.net/ Wow scarily accurate. http://git-man-page-generator.lokaltog.net/#2d1a13476a5f32c4db27fd7aa89a84f3 Anything to do with git submodules is virtually impossible to distinguish from an elaborate practical joke. I know this because I have tried to use them. But there is hope: the git maintainers *do* accept docs patches. https://www.kernel.org/pub/software/scm/git/docs/git-config.html#_configuration_file Formerly, this said "You will find a description of non-core porcelain configuration variables in the respective porcelain documentation in the appropriate manual page.". Did you know that that means it's acceptable for a third-party git hook to make use of 'git config' to allow end users to configure it? Neither did I, till I asked on the mailing list. However, there are enough git front-ends and tutorials that mean you can generally ignore the man pages until you need something more complicated. Basically, consider the git man pages at the same time you'd consider enabling a Mercurial extension. That's how it seems to be. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 24/11/2014 02:59, Barry Warsaw wrote: On Nov 23, 2014, at 08:55 AM, Guido van Rossum wrote: - Moving from Hg to Git is a fair amount of one-time work For anyone seriously interested in this, even experimentally, I would highly suggest looking at Eric Raymond's reposurgeon code. You can google it or find it covered in the vast discussions of his conversion of the Emacs repository from bazaar to git. I don't know for a fact that it would handle an hg to git conversion, but it's the most likely candidate for something so complex. There's a lot of similarity in the history of the Emacs and Python repositories, having gone through many previous iterations of vcs's - in Python's case, at least cvs, svn, and hg. For what it's worth, Mike Bayer successfully migrated SQLAlchemy from svn to hg and then to git. He now has an amazing system set up whereby commits can be made by contributors through either GitHub or BitBucket, I thought he still had Trac rigged up on sqlalchemy.org, but looks like the issue tracker has migrated to bitbucket... I believe his reasons for moving are similar to those I've read in Guido's posts. I believe he also has notes on what he did and how things are set up... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano wrote:
> I wouldn't interpret it like that.
>
> Calling next() on an empty iterator raises StopIteration. That's not a
> bug indicating a failure, it's the protocol working as expected. Your
> response to that may be to catch the StopIteration and ignore it, or to
> allow it to bubble up for something else to deal with it. Either way,
> next() raising StopIteration is not a bug, it is normal behaviour.
>
> (Failure to deal with any such StopIteration may be a bug.)
>
> However, if next() raises RuntimeError, that's not part of the protocol
> for iterators, so it is almost certainly a bug to be fixed. (Probably
> coming from an explicit "raise StopIteration" inside a generator
> function.) Your fix for the bug may be to refuse to fix it and just
> catch the exception and ignore it, but that's kind of nasty and hackish
> and shouldn't be considered good code.
>
> Do you agree this is a reasonable way to look at it?
Yes. Specifically, your parenthesis in the middle is the important
bit. If you have a csv.DictReader, KeyError might be an important part
of your protocol (maybe you have an optional column in the CSV file),
but it should be caught before it crosses the boundary of "part of
your protocol". At some point, it needs to be converted into
ValueError, perhaps, or replaced with a default value, or some other
coping mechanism is used. Failure to deal with StopIteration when
calling next() is failure to cope with all of that function's
protocol, and that is most likely to be a bug. (There are times, and
some of them have been mentioned in these discussion threads, where
calling next() can never raise StopIteration, so there need be no
try/except - eg it=iter(string.split(" ")) - but that just means that
a StopIteration from that call is an error somewhere else. I'm
definitely happy for that kind of "shouldn't happen" to turn into a
RuntimeError rather than being left as an unexpectedly-short
generator.)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher wrote: > Yield can also raise StopIteration, if it's thrown in. The current > interaction of generator.throw(StopIteration) with yield from can't be > emulated under the PEP's behavior, though it's not clear that that's a > problem. > Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 11:58 AM, Greg wrote: > The Abstract claims that the proposal will "unify the behaviour of > list comprehensions and generator expressions", but it doesn't do > that. I don't know that it completely unifies the behaviours, but it does merge them on the specific situation of a leaking StopIteration. With the original code examples that sparked this all off (see the first footnote in the PEP), current CPython has the list-comp form terminate cleanly, while the genexp form infinitely loops. With the POC patch on the issue tracker, both forms cause RuntimeError. Is there a better word than "unify" for that? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 10:24 PM, Nick Coghlan wrote: > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". If I understand you correctly, you agree that this is a benefit, correct? > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... >try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. I'm not sure how often the ease-of-reading concern will come up, but I can at least benchmark the performance of it. I have two virtually-identical builds of CPython 3.5, one with and one without the POC patch. There's no guarantee that this will properly match the performance of the final product, as there'll likely be some additional checks (especially when there's a __future__ directive to concern ourselves with), but it's a start. yield from: https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py explicit loop: https://github.com/Rosuav/GenStopIter/blob/c071d/perftest.py The numbers are pretty noisy, but I'm seeing about a 5% slowdown in the 'yield from' version, with a recursion depth of 100 generators. (Obviously less percentage slowdown with less depth, as other factors have more impact.) Rewriting the loop to use an explicit try/except and while loop roughly doubles the time taken (so clearly 'yield from' is implemented very efficiently), and also destroys any meaning in the numbers - the two interpreters come out effectively equal. My preliminary conclusion is that there is some impact, but it's unlikely to be significant in the real world. Do you have a more real-world code example to try? > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def my_generator(): > ... > yield next(it) > ... > > Which would be semantically equivalent to: > > def my_generator(): >try: >... > yield next(it) > ... > except StopIteration > return > > but *much* faster (especially if used in a producer/consumer pipeline) > since it would allow a single StopIteration instance to propagate > through the entire pipeline, rather than creating and destroying new > ones at each stage. If the issue is performance, I would prefer to see something done with a peephole optimizer instead: if there's a "try... except StopIteration: return" construct, it's optimized away down to a magic flag. That way, the code is guaranteed to be correct in all cases, with no hidden behavioral changes, and this is _just_ a performance optimization. I'd still rather see the exception-catching scope narrowed as much as possible, though, which means not having something that's semantically equivalent to wrapping the whole generator in "try... except StopIteration: pass". > P.S. While I'm less convinced this part is a good idea, if > "allow_implicit_stop" accepted both generator functions *and* > generator objects, then folks could even still explicitly opt in to > the "or stop()" trick - and anyone reading the code would have a name > to look up to see what was going on. That would look somewhat thus, then: use_an_iterable(allow_implicit_stop(x+1 for x in itr if x<10 or stop())) I'm definitely not convinced that this would improve matters. However, I don't currently have any recommendation for an "or stop()" replacement, other than "refactor it into an explicitly-written generator function and call it, then you can use return statements". Suggestions welcomed. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 2:55 AM, Hrvoje Niksic wrote: > To retrieve a single value from an iterator, one can use the for/break/else > idiom: > > def my_generator(): > ... > for val in it: > yield val > break > else: > return > ... While that does work, it's not really much more "normal" than a try/except. A for loop implies iteration; having a loop with a hard "break" at the end of it messes with expectations. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > The implicit stop decorator would then check the flags on the code object > attached to the passed in function. If GENERATOR wasn't set, that would be > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > function would be passed through unmodified. However, if EXPLICIT_STOP *was* > set, the generator function would be replaced by a *new* generator function > with a *new* code object, where the only change was to clear the > EXPLICIT_STOP flag. Is it possible to replace the code object without replacing the function? Imagine if you have multiple decorators, one of which retains a reference to the function and then this one which replaces it - the order of decoration would be critical. OTOH, I don't know that anyone would retain references to __code__. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum wrote: > The design just copies the code object with one flag set differently. Code > objects are immutable but they can be copied (though the interface to do > that is kind of hidden). Yes, but the proposal as written spoke of replacing the generator *function*, which has broader consequences. If it's simply replacing the __code__ attribute of that function, it ought to be safe, I think? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum wrote: > No, that was a figure of speech. The proposed decorator returns a new > function object that references a new code object. The original function and > code object are unchanged. Then it has a potentially-confusing interaction with decorators like Flask's app.route(), which return the original function unchanged, but also save a reference to it elsewhere. The order of decoration determines the effect of the @hettinger decorator; there will be two functions around which are almost, but not entirely, identical, and it'd be very easy to not notice that you decorated in the wrong order. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum wrote: > Well, that's just a general problem with decorator ordering. Indeed. I was hoping it could be avoided in this instance by just altering __code__ on an existing function, but if that's not possible, we fall back to what is, after all, a known and documented concern. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:11 PM, Guido van Rossum wrote: > A decorator with a side effect *elsewhere* (like the route registrations) is > acceptable; one with a side effect *on the decorated function* is > questionable, and instead the decorator should behave "functionally", i.e. > return a new object instead. Okay! I learn something new every day. :) Did not know that... so it's a good thing I don't write decorators. I withdraw the suggestion. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
