[issue47257] add methods to get first and last elements of a range

[paul rubin]

paul rubin  added the comment:

Oh nice, I didn't realize you could do that.  len(range) and bool(range) (to 
test for empty) also work.  Ok I guess this enhancement is not needed.  I will 
close ticket, hope that is procedurally correct, otherwise feel free to fix.  
Thanks.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue47257>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47257] add methods to get first and last elements of a range

[paul rubin]

New submission from paul rubin :

Inspired by a question on comp.lang.python about how to deal with an int set 
composed of integers and ranges.  Range objects like range(1,5,2) contain 
start, stop, and step values, but it's messy and potentially tricky to get the 
actual first and last values of the range.  Examples:

range(1,5,2) - first = 1, last = 3

range (5, 1, 2) - range is empty, first = last = None

range(5, 1, -1) - first is 5, last is 2

Note in the case where the range is not empty, you can get the "last" by a 
messy calculation but it's easier to pick the first element from the reverse 
iterator.  But then you might forget to catch the stopiteration exception in 
the case that the list is empty.  The same goes for the first element, roughly. 
 And constructing the iterators just to pick one element seems like unnecessary 
overhead.

So it is better to have actual methods for these, with type Optional[int].  
Then mypy should remind you to check for the empty case if you forget.

--
messages: 416962
nosy: phr
priority: normal
severity: normal
status: open
title: add methods to get first and last elements of a range
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue47257>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44571] itertools: takedowhile()

[paul rubin]

paul rubin  added the comment:

Bah, the above doesn't work in the cases where the iterator is empty or 
(different symptom) where the tail part is empty.  Rather than posting a 
corrected version (unless someone wants it) I'll just say not to use that code 
fragment, but that the intended API still looks reasonable.  I do support 
having some kind of solution but don't want to keep stretching out an already 
closed discussion unless people think there is more to say.

--

___
Python tracker 
<https://bugs.python.org/issue44571>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44571] itertools: takedowhile()

[paul rubin]

paul rubin  added the comment:

Oh wow, before_and_after will go into the itertools module per that patch?  I 
found this issue while looking for a way to this, but had written the following 
implementation:

def span(pred, xs):
# split xs into two iterators a,b where a() is the prefix of xs 
# that satisfies the predicate, and b() is the rest of xs.  
# Similar to Data.List.Span in Haskell. 

ixs = iter(xs)
t = None
def a():
nonlocal t
for x in ixs:
if pred(x): yield x
else: break
t = x
def b():
return itertools.chain([t], ixs)
return a, b

def tspan():  # test
xs = [1,3,5,2,4,6,8]
def odd(x): return x%2==1
# This should print [1,3,5] then [2,4,6,8]  
for p in span(odd, xs):
print(list(p()))

--
nosy: +phr

___
Python tracker 
<https://bugs.python.org/issue44571>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40734] /usr/bin surprisingly in sys.path under IDLE

[paul rubin]

paul rubin  added the comment:

Yes as mentioned I'm running Debian GNU/Linux, not Windows.  By "idle is 
installed in /usr/bin" I mean that it is an executable shell script stored at 
/usr/bin/idle .  Yes, shell prompt is the $ prompt to bash.  When I run 
"python3 -m idlelib", /usr/bin does not appear in sys.path.  "python -m 
idlelib" attempts to run python2 and I don't have python2 idle installed.  I'll 
see if I can figure out what's going on with sys.path in the user process.  The 
explanation about the two processes was helpful.  Thanks.

--

___
Python tracker 
<https://bugs.python.org/issue40734>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40734] /usr/bin surprisingly in sys.path under IDLE

[paul rubin]

paul rubin  added the comment:

I'm using Debian 10 MATE live install and have been running IDLE by clicking an 
icon on the top panel, but I just tried running IDLE from the shell prompt in a 
terminal window, and also see /usr/bin in the path.  In both cases, the output 
of os.system('pwd').read() is my home directory.  IDLE itself is installed in 
/usr/bin .

--

___
Python tracker 
<https://bugs.python.org/issue40734>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40734] /usr/bin surprisingly in sys.path under IDLE

[paul rubin]

paul rubin  added the comment:

Matthias, I get the same result you do when I run python from the shell command 
line.  I see /usr/bin in the path when I import sys and print sys.path from 
inside IDLE.  In other words this is an IDLE configuration oddity.  Again I 
don't know if it's a bug.  It's perplexing though.

--

___
Python tracker 
<https://bugs.python.org/issue40734>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40733] Make IDLE doc more visible, mention in main python docs page

[paul rubin]

paul rubin  added the comment:

I think I used duckduckgo to find the docs.  They don't change much between 
versions and I was trying to find how to do a specific thing.  My installation 
has the docs included but it didn't explain how to do what I wanted, so I had 
hoped there was a more complete doc on docs.python.org.  Unfortunately it 
turned out to be the same doc.  The functionality I wanted (to put a startup 
script in .idlerc) seems to not exist though.  I'll research workarounds a bit 
further and possibly open an RFE if people think that sounds right.

--

___
Python tracker 
<https://bugs.python.org/issue40733>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40733] mention IDLE in main python docs page

[paul rubin]

Change by paul rubin :


--
title: mention IDLE in main python ocs page -> mention IDLE in main python docs 
page

___
Python tracker 
<https://bugs.python.org/issue40733>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40734] /usr/bin surprisingly in sys.path under IDLE

[paul rubin]

New submission from paul rubin :

This is in the standard python 3.7.3 install under Debian 10.  It's possible 
that this is on purpose, and it's (separately) possible that the Debian 
packagers did this for some reason.  I'm not sure it's a bug but am reporting 
it as it's an oddity that might warrant looking into.

When I look at sys.path in the IDLE shell, the path includes /usr/bin, which is 
not there under the normal python prompt.  Since /usr/bin is not a place where 
python modules usually live, it's a bit strange to find it on the path.  It 
doesn't seem healthy since it could lead to surprises.  But maybe I'm missing 
something.

Feel free to close this if the inclusion is intentional.

--
assignee: terry.reedy
components: IDLE
messages: 369643
nosy: phr, terry.reedy
priority: normal
severity: normal
status: open
title: /usr/bin surprisingly in sys.path under IDLE
type: behavior
versions: Python 3.7

___
Python tracker 
<https://bugs.python.org/issue40734>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40733] mention IDLE in main python ocs page

[paul rubin]

New submission from paul rubin :

The IDLE documentation is in https://docs.python.org/3/library/idle.html which 
is not where I'd have thought to look for it, since I think of IDLE as an 
application rather than a library.  So I looked for it on the main docs page, 
docs.python.org, and didn't find it there.  I ended up finding it by web 
search.  

I guess its current location is reasonable, but maybe a link to it could be 
included in docs.python.org's main page left side navigation panel, or IDLE 
could simply be mentioned in the entry for library documentation.

--
assignee: docs@python
components: Documentation
messages: 369640
nosy: docs@python, phr
priority: normal
severity: normal
status: open
title: mention IDLE in main python ocs page
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue40733>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Note: PEP 603 may essentially take care of this, if it is accepted.

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40028] Math module method to find prime factors for non-negative int n

[paul rubin]

paul rubin  added the comment:

I don't think the interface needs much bikeshedding, as long as the implementer 
chooses something reasonable.  E.g. factor(30) gives the list [2,3,5].  
Implementation is harder if you want to handle numbers of non-trivial size.  
Neal Koblitz's book "A Course in Number Theory and Cryptogoraphy" has good 
coverage of factoring algorithms.  To factor numbers up to 2**64, Pollard's rho 
method is simple to code and has always worked for me, but I don't know if 
there are specific numbers in that range that could give it trouble.  For 
bigger numbers you need fancier algorithms and eventually fancy hardware and 
long computing runs.  Part of a design discussion would include trying to 
decide the scope of such a module.

--

___
Python tracker 
<https://bugs.python.org/issue40028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40623] JSON streaming

[paul rubin]

paul rubin  added the comment:

Also I didn't know about ndjson (I just looked at it, ndjson.org) but its 
existence and formalization is even more evidence that this is useful.  I'll 
check what the two different python modules linked from that site do that's 
different from your example of iterating through the file by lines.

--

___
Python tracker 
<https://bugs.python.org/issue40623>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40623] JSON streaming

[paul rubin]

paul rubin  added the comment:

It's coming back to me, I think I used the no-separator format because I made 
the multi-document input files by using json.dump after opening the file in 
append mode.  That seems pretty natural.  I figured the wikipedia article and 
the json.tool patch just released were evidence that there is interest in this. 
 The approach of writing newlines between the docs and iterating through lines 
is probably workable though.  I don't know why I didn't do that before.  I 
might not have been sure that json docs never contain newlines.

Really it would be nice if json.load could read in anything that json.dump 
could write out (including with the indent parameter), but that's potentially 
more complicated and might conflict with the json spec.

--

___
Python tracker 
<https://bugs.python.org/issue40623>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40623] JSON streaming

[paul rubin]

paul rubin  added the comment:

Note: the function in my attached file wants no separation at all between the 
json docs (rather than a newline between them), but that was ok for the 
application I wrote it for some time back.  I forgot about that when first 
writing this rfe so thought I better clarify.

--

___
Python tracker 
<https://bugs.python.org/issue40623>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40623] JSON streaming

[paul rubin]

Change by paul rubin :


Added file: https://bugs.python.org/file49154/jsonstream.py

___
Python tracker 
<https://bugs.python.org/issue40623>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40623] JSON streaming

[paul rubin]

New submission from paul rubin :

This is a well-explored issue in other contexts: 
https://en.wikipedia.org/wiki/JSON_streaming

There is also a patch for it in json.tool, for release in 3.9: 
https://bugs.python.org/issue31553

Basically it's often convenient to have a file containing a list of json docs, 
one per line.  However, there is no convenient way to read them back in one by 
one, since json.load(filehandle) barfs when it sees the unexpected newline at 
the end of the first doc.

It would be great if the json module itself had a function to handle this.  I 
have an awful hack that I use myself, that is not suitable for a production 
library, but I'll attach it to show what functionality I'm suggesting.  I hope 
this is simple enough to not need a PEP.  Thanks!

--
components: Library (Lib)
files: jsonstream.py
messages: 368823
nosy: phr
priority: normal
severity: normal
status: open
title: JSON streaming
type: enhancement
Added file: https://bugs.python.org/file49153/jsonstream.py

___
Python tracker 
<https://bugs.python.org/issue40623>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36027] Support negative exponents in pow() where a modulus is specified.

[paul rubin]

paul rubin  added the comment:

https://bugs.python.org/issue457066  The old is new again ;-).

--
nosy: +phr

___
Python tracker 
<https://bugs.python.org/issue36027>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40028] Math module method to find prime factors for non-negative int n

[paul rubin]

paul rubin  added the comment:

I'm the one always asking for more stuff in the stdlib, but above some 
simplistic approaches this seems out of scope.  Doing it usefully above say 
2**32 requires fancy algorithms.  Better to use some external package that 
implements that stuff.

--
nosy: +phr

___
Python tracker 
<https://bugs.python.org/issue40028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Totally tangential: Veky, your ordinal example would work ok in Haskell and 
you'd have omega work the same way as epsilon0.  Take a look at Herman Ruge 
Jervell's book "Proof Theory" for a really nice tree-based ordinal notation 
that goes much higher than epsilon0.  It would be cool if your student 
implemented it.  Raymond, I agree that ordinals are a very esoteric use of 
multisets ;).

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Yeah I think the basic answer to this ticket is "Python doesn't really have 
multisets and a proposal to add them should go somewhere else".  Fair enough-- 
consider the request withdrawn from here.

Regarding minimalism vs completeness, regarding some feature X (say X is 
multisets), it's reasonable per minimalism to decide not to have X.  But if on 
weighing the use cases you decide to have X after all, I think it's best to 
implement X properly and completely with all the edge cases handled, rather 
than implement a half-baked subset.  That was something Java was historically 
very good at and Python wasn't.  I guess that is one for the theorists though.  
Thanks everyone.

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Oops I meant "*without* 100s of third party modules" in the case of ruby gems 
or npm.  There are just a few pip modules that I really use all the time, most 
notably bs4.  I continue to use urllib/urllib2 instead of requests because I'm 
used to them and because they eliminate an unnecessary dependency.

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Kyle, thanks, I saw your comment after posting my own, and I looked at 
Raymond's mailing list post that you linked.  I do think that "completing the 
grid" is a good thing in the cases where it's obvious how to do it (if there's 
one and one obvious way, then doing it that way should work), and having 
missing combinations in the grid is also a form of interface complexity (expect 
subset to work, find that it doesn't, and implement some hack).  Also it seems 
to me that sets, dicts, bags are basic data objects in many languages these 
days, without a lot of design space to move around in.  So we don't have to 
worry much about constraining future choices.  We could just look at Smalltalk 
or Ruby or whatever and just do what they did.  

However, that is philosophical.  I did learn from Raymond's message that 
Counter doesn't really attempt to be a multiset implementation.  In that case, 
it works for what it is made for, so it really then is a separate question of 
whether implementing multisets properly is worthwhile.  (That's as opposed to 
saying we already have a multiset implementation but some functionality is 
missing from it).

I dislike the concept of shovelling off basic functionality to 3rd party 
libraries.  The rejection of that philosophy (Python's old motto "Batteries 
included") is one of the things that attracted me to Python in the first place. 
 Though that value is mostly historical now, I'm sad about the loss.  It's 
impossible to write a large Ruby or Javascript (npm) application with 100s of 
third party modules, every one of which is an attack vector ("supply chain 
attack" is the current buzzword), and  whose implementations vary widely in 
quality and usability.  I looked at the docs for Ruby's multiset gem 
(https://maraigue.hhiro.net/multiset/) and they are partly in Japanese.  Python 
has until the past few years managed to keep away from that kind of thing.

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

paul rubin  added the comment:

Yes, the idea was for them to be hashable, to be used as dict keys.  E.g. if 
you use frozensets to model graphs, you'd use frozen multisets for hypergraphs. 
 My immediate use case involved word puzzles, e.g. treating words as bags of 
scrabble tiles with letters on them, and finding out what other words you could 
form from the letters.  It is not a pressing need.  

I'm trying to remember what the other missing operation I ran into was, a few 
days ago.  It may have been subset.  I found a workaround but it seemed a bit 
ugly.  It would be nice if Counters could also be thought of as multisets which 
can do the same things sets can do with unneeded head scratching.  It seems 
like counters and multisets/bags really are different things conceptually.  
Their operations and implementations overlap, but they are different.  E.g. it 
can make sense for a counter to have a negative count of something, but not for 
a bag.

--

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1726707] add itertools.ichain function and count.getvalue

[paul rubin]

paul rubin  added the comment:

Note, nowadays this is implement as itertools.chain.from_iterable .

--

___
Python tracker 
<https://bugs.python.org/issue1726707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40334] PEP 617: new PEG-based parser

[paul rubin]

Change by paul rubin :


--
nosy:  -phr

___
Python tracker 
<https://bugs.python.org/issue40334>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40411] frozen collection.Counter

[paul rubin]

New submission from paul rubin :

It would be nice to have frozen Counters analogous to frozensets, so they are 
usable as dictionary keys.  One can of course create frozenset(counter.items()) 
but that means the set items are tuples rather than the original set elements, 
so it's no longer quick to check membership.  I can work around this in my 
immediate application but it seems like a shortcoming.  There are some other 
set operations that aren't supported by counters either, that would be nice if 
they are conceptually multisets.

--
components: Library (Lib)
messages: 367472
nosy: phr
priority: normal
severity: normal
status: open
title: frozen collection.Counter
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue40411>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40334] PEP 617: new PEG-based parser

[paul rubin]

paul rubin  added the comment:

I just saw this.  Interesting.  Sometimes I use ast.literal_eval to read big, 
deeply nested data objects.  I can probably convert to JSON if necessary but 
it's another thing to watch out for.  I might try to benchmark some of these.

--
nosy: +phr

___
Python tracker 
<https://bugs.python.org/issue40334>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38333] add type signatures to library function docs

[paul rubin]

paul rubin  added the comment:

I don't think we're going to accomplish anything continuing the eternal 
static-vs-dynamic debate, which in Python's case has already been resolved by 
adding optional static typing.  It's a done deal, and the issue here is just 
how to document it.  Erlang (via the Erlang Dialyzer), Clojure, and Racket have 
all been down a similar road and gained some value from it.

Haskell's Maybe type (its version of Option) works fine.  In Python the 
convention of returning None for a missing value is not great, but we are stuck 
with it and Option[whatever] is helpful.

--

___
Python tracker 
<https://bugs.python.org/issue38333>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38333] add type signatures to library function docs

[paul rubin]

paul rubin  added the comment:

At first glance, having a typeclass for each protocol (at least the widely used 
ones) seems fine.  It's inherent in Haskell and a lot of libraries are 
organized around a common set of typeclasses--look up "Typeclassopedia" for 
descriptions of them.  Certainly the case of abs (which you asked about by 
name), the typeclass is already there in the typing module.

You're right about abs(complex) returning float.  Haskell's type signature for 
abs is "Num a => a -> a" which means the input type is the same as the output.  
That is a little bit peculiar since the abs of a complex number is complex, but 
we usually think of abs as a mathematical norm, which is conventionally a real.

Anyway, "abs(x: Abs[T]) -> Any"  is a better-than-nothing signature for abs, 
and the docs can comment a little further, and maybe someday it could even do a 
type-level lookup at typechecking time, i.e. have something like Haskell's 
"type families".

I like to think it's possible to supply reasonable signatures for most 
functions.  I just fixed a bug in something today because bs4 (beautiful soup 
4) has no typeshed stub so mypy uses Any for functions like soup.find, instead 
of Optional[tag].  So the program worked fine as long as find kept returning a 
tag, but then crashed because it hit a document without a tag and my code 
didn't check for None.  That's something more precise types would have caught 
at mypy time.

--

___
Python tracker 
<https://bugs.python.org/issue38333>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38333] add type signatures to library function docs

[paul rubin]

paul rubin  added the comment:

abs takes any value that understands the __abs__ method and returns something 
of the same type.  In fact there is already a type protocol for it:

https://mypy.readthedocs.io/en/stable/protocols.html#supportsabs-t

So abs's signature would be (x : Abs[T]) -> T where T is a type parameter.  

I'm sure there are some examples where no good signature is possible, but lots 
of others are fine.  Someone did a Smalltalk study long ago and found that most 
functions were monomorphic in practice even though Smalltalk is dynamically 
typed like Python.  As a matter of style, Python code tends to be typed even 
when it doesn't have to be.  Not all the time of course.

I'm still getting used to types and mypy (I was a py2 holdout til quite 
recently, and mypy has been a more attractive reason to change than any of the 
other stuff) and I do keep noticing cases that don't work as I hoped, but it's 
still a good move in general.

--

___
Python tracker 
<https://bugs.python.org/issue38333>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38333] add type signatures to library function docs

[paul rubin]

paul rubin  added the comment:

Yes, the suggestion was just for the docs, and since those are intended for 
human rather than machine consumption, it's fine if there are some blurry cases 
where there is no signature.  Ideally in those cases, the issue should be 
explained in the doc text.

I actually don't see what's wrong with including signatures in the source code 
as well, as long as doing so doesn't break anyone's existing code.  I agree 
with Veky that one should be very hesitant about breaking existing working 
code, even if that code relies on undocumented behavior.

--

___
Python tracker 
<https://bugs.python.org/issue38333>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38333] add type signatures to library function docs

[paul rubin]

New submission from paul rubin :

It would be nice if the library reference manual had type signatures for all 
the stdlib functions at some point.  It might be possible to extract a lot of 
them automatically from typeshed and semi-automatically paste them into the doc 
files.  It might also be ok to do this gradually.  I can help with this but 
wouldn't want to take on the entire task.

--
assignee: docs@python
components: Documentation
messages: 353634
nosy: docs@python, phr
priority: normal
severity: normal
status: open
title: add type signatures to library function docs
type: enhancement
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38333>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: A use-case for for...else with no break

[Paul Rubin]

Steve D'Aprano  writes:
> for x in something():
> print(x, end='')

print(''.join(something()))
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Let's talk about debuggers!

[Paul Rubin]

Terry Reedy  writes:
> On Windows, [IDLE] uses native widgets when possible...
> In summary, I think debugger should rate at least 'good' rather than
> fail' when it comes to showing you the next line.

I actually like how the Tk widgets look.  I've done some semi-industrial
applications with tkinter and they have a nice factory-floor vibe.

I generally just use pdb for debugging.  The feature I miss most is the
ability to trap to the debugger if the program throws an unhandled
exception.  I think some other Python debuggers do support that.  I've
found it invaluable in other languages.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I'd like to use "semantic indentation"

[Paul Rubin]

Chris Angelico  writes:
> USA:
> Alabama:
> Abbeville
> Addison
> ...
> and then, as Paul suggested, write a simple parser to read it.

That looks like YAML, which there's already a library for.  I'm not
crazy about it but it might be an ok choice for this.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I'd like to use "semantic indentation"

[Paul Rubin]

r...@zedat.fu-berlin.de (Stefan Ram) writes:
>   I would like to write source code similar to:
> country( 'USA' )
>   state( 'Alabama' )

Aside from the workaround that I mentioned, this looks more like data
than code.  Maybe you really want to create a nested structure
(dictionaries, JSON, XML or whatever) and traverse it.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I'd like to use "semantic indentation"

[Paul Rubin]

r...@zedat.fu-berlin.de (Stefan Ram) writes:
>   I would like to write source code similar to:
> country( 'USA' )
>   state( 'Alabama' ) ...

> It seems I can't do this with Python.  Is there any workaround?

  _= country( 'USA' )
  _=   state( 'Alabama' )
  _= town( 'Abbeville' )
  _= town( 'Addison' )
  _=   state( 'Arizona' )
  _= town( 'Apache Junction' )
  _= town( 'Avondale )
  _= town( 'Benson' )
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project Euler 20.

[Paul Rubin]

Ian Kelly  writes:
 sum(int(c) for c in str(math.factorial(100)))

Doh!  Using int(c) didn't occur to me and I didn't know about
math.factorial.  Notice also that WJ hasn't yet dared posting his crap
on comp.lang.haskell.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project Euler 20.

[Paul Rubin]

"Robert L."  writes:
>> Find the sum of the digits in the number 100!
> In Python?

So you have come to plague us here too.

>>> sum(ord(c)-ord('0') for c in str(reduce(lambda a,b: a*b, range(1,101),1)))
648
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Old Man Yells At Cloud

[Paul Rubin]

Steve D'Aprano  writes:
> Having to spend a few hours being paid to migrate code using "print x"
> to "print(x)", or even a few months, is not a life-changing experience.

Didn't someone further up the thread mention some company that had spent
1.5 years porting a py2 codebase to py3?

The issue of breaking the print statement isn't the difficulty of
converting old programs, or that the print statement is superior to the
print function or vice versa.  Reasonable people might believe that one
is slightly better than the other, but it would be hard to argue that
one is overwhelmingly better than the other.  So there's not a
convincing reason to change.

That calls the whole py3 movement into question, since its advocates so
vigorously defend unnecessary changes.  It's fine to fix broken stuff
(Unicode was broken, indexes escaping list comprehensions was broken)
but fixing highly visible stuff that wasn't broken makes the more subtle
changes easier to ignore.

Py3 imo would have been more successful if it introduced even more
breakage, but produced dramatic benefits (for example a 10x speedup) as
a result.  That would have been doable.  Instead we got minor benefits
and useless breakage.  Py4 as a result of learning the wrong lesson
won't break anything, so it won't be able to introduce dramatic benefits
either.  Will the 5th time (Py5) be the charm?  (I don't mean Pycharm).

Python is gaining ground in numerics and data science, which is great.
Micropython for embedded MCUs is also poised for popularity.  I don't
know how Python is doing at general systems stuff which is what I mostly
use it for.  I think Ruby is losing ground, and some of the ground it
has lost has been to Elixir.  An Elixir-like reimplementation of Python
might be an interesting avenue to pursue.  So would a dialect with less
pervasive dynamism than Python, but that could be compiled to fast
machine code with traditional Lisp techniques.  The dynamism would still
be available "opt-in" so you could turn it on when you wanted it, and
only those parts of your program would slow down.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [RELEASE] Python 3.6.3rc1 and 3.7.0a1 are now available for testing and more

[Paul Rubin]

Ned Deily  writes:
> You can find Python 3.7.0a1 and more information here:
> https://www.python.org/downloads/release/python-370a1/

This says:

The next pre-release of Python 3.7 will be 3.6.0a2, currently
scheduled for 2016-10-16.

:)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The Incredible Growth of Python (stackoverflow.blog)

[Paul Rubin]

Ben Finney  writes:
>> I've never seen one.
> who has told you... they are working on a Python 3 code base.

Just because they've told me about it doesn't mean I saw it personally.
The ones I've seen, including new ones, are Python 2.

Some people here use Py3 but I haven't heard (or don't remember) enough
about what they're working on, to know if those are py3 codebases of any
size.  If they say yes, I'll take their word for it, but this is a
self-selected group of course.

> That simply isn't true, unless you think it more likely everyone who
> discusses their Python 3 code base is lying.

People discuss Python language issues here a lot, but don't discuss as
much about code bases.

I know when I install a new OS (currently Debian 9 which was released
a month or so ago) and type "python" on the command line, I get Py2.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The Incredible Growth of Python (stackoverflow.blog)

[Paul Rubin]

Chris Angelico  writes:
> students learning Python *today* ... they're learning Python 3.

I'm not so sure of that.  I do know a few people currently learning
Python, and they're using Python 2.

>>  * static type annotation

Seems like a big win if you ask me.

>>  * asyncio with its a-dialect
> Actually, I think this one is a huge enough feature that it's going to
> be a big thing to *drive* the uptake of Python.

God I hope not, it's perverse.  I'd like to see a Python that's more
like Erlang in its approach to concurrency.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup doesn't work with a threaded input queue?

[Paul Rubin]

Christopher Reimer  writes:
> I have 20 read_threads requesting and putting pages into the output
> queue that is the input_queue for the parser. 

Given how slow parsing is, you probably want to scrap the pages into
disk files, and then run the parser in parallel processes that read from
the disk.  You could also use something like Redis (redis.io) as a queue.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Reading the documentation

[Paul Rubin]

Chris Angelico  writes:
>> And there are numbers which repeat in decimal but not binary, 
> Which ones repeat in decimal but not binary? An example, please.

That should really have said binary but not decimal, since 2 divides 10.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> Did __next__ cache the most recently generated value?

No but if they're going to change stuff, they might as well actually
improve it instead of just renaming it to break code gratutiously.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> the public API is to call the next() built-in function, and the
> implementation is the __next__ dunder.

In that case it would have been nice to make next() cache the most
recently generated value from the iterator.  That would make lots of
code simpler.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Peter Otten <__pete...@web.de> writes:
> Python 3 where the next() method has been renamed to __next__().

Oh cripes, you're right, it never occurred to me that py3 had broken
.next().  I thought it was called .next() instead of .__next()
so that it wouldn't be a dunder method.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> I've read a few people claim that disallowing multiplication from
> standard arithmetic renders it weak enough that you can prove it
> complete and correct, but since they give no proof or even evidence I
> have my doubts.

That system is called Presburger arithmetic (see Wikipedia).  It can be
proved consistent by quantifier elimination but is not powerful enough
to prove (or maybe even state) its own consistency.

Surprisingly, there are also theories that can state and prove their own
consistency:

  https://en.wikipedia.org/wiki/Self-verifying_theories
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Ben Finney  writes:
> generate_id = functools.partial(next, itertools.count())

Is something wrong with:

>>> g = itertools.count().next
>>> g()
0
>>> g()
1
>>> g()
2
>>> ...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Rustom Mody  writes:
> Do you mean Frege or Cantor?

Frege.  Cantor was concerned with set theory, while Frege was concerned
with logic in general.  Frege's notation was different from what we use
now but the issue was about the same: unrestricted comprehension led to
contradiction.  As you mention, Russell was the one who recognized the
inconsistency in Frege's system, though I don't know how Frege took it
at a personal level.

Many of the original writings from that era are reproduced in the book
"From Frege to Gӧdel: A Source Book in Mathematical Logic".  It has good
introductions to the papers and is pretty interesting.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Rustom Mody  writes:
> Specifically the term 'comprehension' used today as a programming construct
> traces somewhat tenuously to an axiom that Zermelo/Fraenkel formulated
> in the 1920s

I thought went back to Frege.  Also, it appears in Zermelo set theory Z.
ZF is Z with the Axiom of Replacement added, but Z was somewhat earlier
than ZF.  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Marko Rauhamaa  writes:
> The question is, is it bad style—or even an error—to rely on the
> execution order of the comprehension loops? 

Bad style: IMO, yes, most of the time.  I've made use of it at
particular times.  If it's done in an obscure way it at least
deserves a code commment.

Error: no, I think the language spec is clear.  It's not like the
situation of people depending on reference counting to free resources
predictably.  Use the 'with' statement for that.

> Is a Python implementation allowed to parallelize or otherwise reorder
> the evaluation loop?

No.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What extended ASCII character set uses 0x9D?

[Paul Rubin]

John Nagle  writes:
> Since, as someone pointed out, there was UTF-8 which had been
> run through an ASCII-type lower casing algorithm

I spent a few minutes figuring out if some of the mysterious 0x81's
could be from ASCII-lower-casing some Unicode combining characters, but
the numbers didn't seem to work out.  Might still be worth looking for
in some other cases.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> For loops and comprehensions (in Python) are inherently procedural,

Sure, and floating point arithmetic is inherently imprecise and doesn't
follow the associative laws for either addition or multiplication.
There are times when we have to be aware of those details.  Usually,
though, we want to act as if they represent the mathematical reals, and
we read Python statements involving floats as if they were mathematical
statements involving reals.  When possible, we write in a style where
this doesn't cause problems, use double precision to decrease rounding
errors in long calculations, etc.

Similarly we occasionally have to be aware of the procedural nature
of Python list comprehensions, but most of the time we think of them
in terms of the mathematical abstraction they are designed to resemble.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cross-language comparison: function map and similar

[Paul Rubin]

Steve D'Aprano  writes:
> Are there language implementations which evaluate the result of map()
> (or its equivalent) in some order other than the obvious left-to-right
> first-to-last sequential order? Is that order guaranteed by the
> language, or is it an implementation detail?

Haskell just gives back an unevaluated thunk.  The elements are
evaluated in whatever order you happen to use them in.  Since the
evaluation isn't supposed to have observable side effects, there's no
way to tell the order.

There are also parallel versions (Control.Parallel.Strategies.parMap
etc.) and an extension that recognizes "racing stripes" on the square
brackets to make list comprehensions run faster:

   foo = [| sqrt(x) | x <- [1.0 .. 1000.0] |]

parallelizes the calculation and offloads it to a GPU or other vector
processor.  You might also like this old post

 
https://donsbot.wordpress.com/2007/11/29/use-those-extra-cores-and-beat-c-today-parallel-haskell-redux/

"map" is conceptually the lifting of a function from a given type, to
the type under the action of some functor.  For historical reasons map
only works on lists while fmap works on arbitrary functors, but they are
in principle the same thing.  Depending on what the functor does, the
whole concept of left-to-right order might be meaningless.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Paul Rubin <no.email@nospam.invalid> writes:
>   FORALL P. [ P(0) and P(n) -> P(n+1) ]

Sorry, that was supposed to say

  FORALL P. [ (P(0) and P(n) -> P(n+1)) -> forall n. P(n) ]

FORALL quantifies over formulas and forall quantifies over numbers.

Maybe something is still missing from the above ;-).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Marko Rauhamaa  writes:
> For the non-logicians in the audience, an "axiom schema" is a generator
> pattern that produces an infinity of actual axioms. 

For non-logicians this is not worth worrying about: schemas are
basically a technical hack to get around the inability of first-order
logic to quantify over formulas.  Naively most of us think of single
axioms with second-order quantifiers, instead of schemas.

For example, arithmetic induction goes something like:

  FORALL P. [ P(0) and P(n) -> P(n+1) ]

where "FORALL P" means quantifying over all formulas P.  Since in FOL we
can only quantify over numbers and not formulas, we write down a
separate axiom for each possible formula.  Since there are infinitely
many formulas, this is an infinite axiom schema.

Historically (in "naive set theory") we didn't bother with any of this.
We could write { S : S \not\in S } for the set of all sets that are not
members of themselves.  Is S a member of itself ("Russell's paradox")?
Either way leads to contradiction.  So the comprehension axiom schemas
for set theory had to be designed to not allow formulas like that.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> It's quite clever, actually, in that it gives *pseudo* random-access
> with lazy evaluation. You can evaluate the Nth element without
> evaluating the earlier ones. But you can't do so without *storing* the
> previous ones, they have to be allocated, with only the actual
> evaluation being delayed.

Look at any of the memoization packages for ways around that.  Or of
course you could just write a function instead of a list...

> In Haskell, you cannot get the last N elements of a list without
> allocating memory for the previous ones.

lastn n xxs@(x:xs)
  | length (take n xs) == n-1 = xxs
  | otherwise = lastn n xs

main = print . lastn 5 $ [1..1000]

*Main> main
[996,997,998,999,1000]

works for me.  The 1000 list nodes all get allocated, but are
immediately freed, so only 5 cells have to be in memory at a time.  In
principle a fancy enough compiler optimization could get rid of all the
allocation completely.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> And very possibly the first language with comprehensions, SETL (from 1969),
> used "forall" ... 

It goes back much further, probably to Principia Mathematica, or (with
different notation) maybe Frege's Foundations of Arithmetic:

  https://en.wikipedia.org/wiki/Set-builder_notation

I don't think Python should try to turn it into something different.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Marko Rauhamaa  writes:
> Jussi Piitulainen :
>> But what is "set comprehension" in French, German, or Finnish?

The idea comes from set theory: for some reason English Wikipedia
doesn't have Finnish cross-wiki links for most of the relevant
terms, but Google translates "axiom of specification" as
"Määritelmän axiom".Maybe you can find something from there.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proposed new syntax

[Paul Rubin]

Steve D'Aprano  writes:
> What would you expect this syntax to return?
> [x + 1 for x in (0, 1, 2, 999, 3, 4) while x < 5]

[1,2,3] though the later example is more confusing.

I don't think we need this since we have itertools.takewhile:

  from operator import gt
  from functools import partial
  from itertools import takewhile

  [x + 1 for x in takewhile(partial(gt,5), (0,1,2,999,3,4))]

In the eye of the beholder maybe, but I think it's less ugly in Haskell:

  [x + 1 | x <- takeWhile (< 5) [0,1,2,999,3,4]]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Challenge: find the first value where two functions differ

[Paul Rubin]

Chris Angelico  writes:
> 4503599761588224

I get the same result from searching a wider interval (+/- 50) around
each perfect square in the relevant range.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: YAML in std lib?

[Paul Rubin]

Ben Finney  writes:
> I don't know of any PEP yet which specifies exactly what to add to the
> standard library for YAML

YAML is more of a Ruby thing, so there might not be much constituency
for putting it in Python.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Falsey Enums

[Paul Rubin]

Dan Sommers  writes:
> def __bool__(self):
> return False if self == X.Falsey else True

return self != X.Falsey
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: OT was Re: Python 3 removes name binding from outer scope

[Paul Rubin]

Chris Angelico  writes:
>>> Bipartisan-US-Bill-Moves-to-Criminalize-BDS-Support-20170720-0001.html
>> Heh, at first I read that as a bill to criminalise BSD support :-)
> I spluttered my drink on reading that. Good job Steven!

https://en.wikipedia.org/wiki/BDS_C
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3 removes name binding from outer scope

[Paul Rubin]

Ben Finney  writes:
> How can I stop Python from deleting a name binding, when that name is
> used for binding the exception that is caught?

Use sys.exc_info()
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Recent Spam problem

[Paul Rubin]

Rustom Mody  writes:
> Since spammers are unlikely to be choosy about whom they spam:
> Tentative conclusion: Something about the USENET-ML gateway is more leaky
> out here than elsewhere

It could be a sort-of DOS attack by some disgruntled idiot.  I wonder if
the email address in those spam posts actually works.  Then there's the
weird Italian rants.  No idea about those.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Paul Rubin]

Rustom Mody  writes:
> Yeah I know append method is supposedly O(1).

It's amortized O(1).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Paul Rubin]

Chris Angelico  writes:
> Maybe I'm completely on the wrong track here, but the last time I
> implemented a self-balancing tree, it usually involved a fair amount
> of mutation.

AVL trees are fairly simple to implement without mutation.  Red-black
trees are traditionally implemented with mutation, inserting by making
nodes mis-colored, then going and re-coloring them.  But they can be
done mutation-free as well.  Here's an amazing Haskell implementation
where the tree invariants are encoded in the datatype:

https://gist.github.com/rampion/2659812

Reddit discussion of above:  https://redd.it/ti5il

More recent versions of GHC make the type signatures even nicer, since
you can put numbers directly into types without that nested type encoding.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Paul Rubin]

Chris Angelico  writes:
> some point it'll need to be rebalanced, which could at worst case
> be O(n). 

No, you use a structure like an AVL tree or red-black tree, so it's
within a constant factor of balanced after each insertion.  You rewrite
O(log n) of the nodes, and juggle around a constant number of them at
the top of the tree.  The Wikipedia articles about those data structures
are pretty good.  C++ std::map is also implemented that way, I think.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Paul Rubin]

Steven D'Aprano  writes:
> for parrot in parrots:
> accumulator[parrot.colour].append(parrot)
>
> That's pretty compact and understandable, but it require mutating a bunch 
> of pre-allocated lists inside an accumulator. Can we re-write this in a 
> functional style?

Not so easily in Python since the built-in list and dict types are
designed for mutation update.  In Haskell, the list type is a linked
list and the dictionary type is a balanced tree.  So, you can make a new
list consisting of a new item consed to the front of the old list, and
you can make a new ("updated") dictionary by building O(log n) new
nodes.

You might like Chris Okasaki's wonderful book "Purely Functional Data
Structures" that explains all this and more.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's with all of the Case Solution and Test Bank nonsense posts?

[Paul Rubin]

Michael Torrie  writes:
>> can you get a newsreader to work with a https news service?
> No.  A newsreader works with NNTP protocol.

Traditionally NNTP over SSL was done on port 563.  Some feeds now also
provide it on 443 to get around client-side firewall hassles.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's with all of the Case Solution and Test Bank nonsense posts?

[Paul Rubin]

timetowal...@gmail.com writes:
> What's with all of the Case Solution and Test Bank nonsense posts?
> Is is possible to have these posts filtered out?

As people have said, you can block them with a good news reader.  But
like you, I wonder why the heck they have camped out on this particular
newsgroup.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Checking for an exception

[Paul Rubin]

Steve D'Aprano  writes:
> What's the right/best way to test whether an object is an exception
> ahead of time? (That is, without trying to raise from it.)

Maybe I'm missing something but 
   isinstance(obj, Exception)
seems to work.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy (Posting On Python-List Prohibited)

[Paul Rubin]

Lawrence D’Oliveiro  writes:
> while “memory footprint” depends on how much memory is actually being
> retained in accessible objects.

If the object won't be re-accessed but is still retained by gc, then
refcounting won't free it either.

> Once again: The trouble with GC is, it doesn’t know when to kick in:
> it just keeps on allocating memory until it runs out.

When was the last time you encountered a problem like that in practice?
It's almost never an issue.  "Runs out" means reached an allocation
threshold that's usually much smaller than the program's memory region.
And as you say, you can always manually trigger a gc if the need arises.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Paul Rubin]

Lawrence D’Oliveiro  writes:
> The trouble with GC is, it doesn’t know when to kick in: it just keeps
> on allocating memory until it runs out.

That's not how GC works, geez.  Typically it would run after every N
bytes of memory allocated, for N chosen to balance memory footprint
with cpu overhead.  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Gregory Ewing  writes:
> A JIT compiler works by observing the actual values

To be pedantic, that's called a "tracing JIT".  Other runtime code
generation is also frequently called JIT compilation even when it's
fairly stupid combining of assembly code templates, or the like.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Chris Angelico  writes:
> while True:
> c = sys.stdin.read(1)
> if not c: break
> if c.isprintable(): text += c
> elif c == "\x08": text = text[:-1]
> # etc
> Can you write _that_ as a do-while?

I prefer to write that sort of thing with iterators:

 for c in iter(lambda: sys.stdin.read(1), ''):
 if c.isprintable(): text.append(c)
 elif c == '\x08': text.pop()
 ...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Paul Rubin]

Cem Karan  writes:
> I'm not too sure how much of performance impact that will have.  My
> code generates a very large number of tiny, short-lived objects at a
> fairly high rate of speed throughout its lifetime.  At least in the
> last iteration of the code, garbage collection consumed less than 1%
> of the total runtime.  Maybe this is something that needs to be done
> and profiled to see how well it works?

If the gc uses that little runtime and your app isn't suffering from the
added memory fragmentation, then it sounds like you're doing fine.

> I **still** can't figure out how they managed to do it,

How it works (i.e. what the implementation does) is quite simple and
understandable.  The amazing thing is that it doesn't leak memory
catastrophically.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Instagram: 40% Py3 to 99% Py3 in 10 months (Posting On Python-List Prohibited)

[Paul Rubin]

Steven D'Aprano  writes:
> genuinely good reason...  (Which might include "the customer insists",
> or "yeah, I know it sucks, but politics".)

I think the current LTS versions of Ubuntu and Debian both come with
Python 2.  Not sure about Centos/RHEL.  Those seem like ok reasons to
me.

> if you want to use 1.5 go right ahead

I tested all my Python 2 code for compatibility with 1.5 for quite a
long time into the Python 2 series, but eventually got too addicted to
stuff like using iterators pervasively.  I don't think Python 2 really
broke any 1.5 code though.  At least the print statement still worked.

I'm comfortable enough with Python that it's still what I do most of my
personal stuff with, but both py2 and py3 have enough deficiencies that
I see them being relegated to the numerical computation niche, where
they really do have strong library support and a dedicated user base.

Unfortunately, everything I know of that fixes Python's deficiencies
also introduces deficiencies of its own, so at best it's a wash.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Paul Rubin]

Cem Karan  writes:
> Can you give examples of how it's not reliable?

Basically there's a chance of it leaking memory by mistaking a data word
for a pointer.  This is unlikely to happen by accident and usually
inconsequential if it does happen, but maybe there could be malicious
data that makes it happen

Also, it's a non-compacting gc that has to touch all the garbage as it
sweeps, not a reliability issue per se, but not great for performance
especially in large, long-running systems.

It's brilliant though.  It's one of those things that seemingly can't
possibly work, but it turns out to be quite effective.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Paul Rubin]

Chris Angelico  writes:
> Or let's look at it a different way. Instead of using a PyObject* in C
> code, you could write C++ code that uses a trivial wrapper class that
> holds the pointer, increments its refcount on construction, and
> decrements that refcount on destruction.

That's the C++ STL shared_ptr template.  Unfortunately it has the same
problem as Python refcounts, i.e. it has to use locks to maintain thread
safety, which slows it down significantly.

The simplest way to start experimenting with GC in Python might be to
redefine the refcount macros to do nothing, connect the allocator to the
Boehm GC, and stop all the threads when GC time comes.  I don't know if
Guile has threads at all, but I know it uses the Boehm GC and it's quite
effective.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Instagram: 40% Py3 to 99% Py3 in 10 months (Posting On Python-List Prohibited)

[Paul Rubin]

Lawrence D’Oliveiro  writes:
> Is Python 2 the Windows XP of the programming world?

That's a good way to put it.  It's nice to hear about Instagram but so
far I don't personally know anyone who uses Python 3.  Meanwhile there's
still lots of new Py2 projects being started.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Instagram: 40% Py3 to 99% Py3 in 10 months

[Paul Rubin]

Paul Barry  writes:
> The process they followed is discussed in their recent Keynote at PyCon
> 2017: https://youtu.be/66XoCk79kjM
> Well worth the 40 minutes it takes to watch  :-)

If it takes 40 minutes to describe how they did it, that sounds like
more hassle than most users of working py2 code probably want to deal
with.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Progress on the Gilectomy

[Paul Rubin]

I always thought the GIL removal obstacle was the need to put locks
around every refcount adjustment, and the only real cure for that is to
use a tracing GC.  That is a good idea in many ways, but it would break
the existing C API quite seriously.  Reworking the C modules in the
stdlib would be a large but not impossible undertaking.  The many
external C modules out there would be more of an issue.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Python - Career question

[Paul Rubin]

Larry Martell  writes:
> I can tell they think I am old and they dismiss me right away.

http://oldgeekjobs.com ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [OT] How to improve my programming skills?

[Paul Rubin]

Mirko  writes:
> TLDR: Sorry for OT. Long-time Linux geek and hobby programmer wants to
> improve his coding skills.  What's most important: project planing,
> algorithms and data structures, contributing to FOSS, web development,
> learning other languages or something else?

If it's specifically about coding skills, the most important thing is
reading and writing lots of code.  That said, coding skills are just one
facet of a good developer.  The most important developers in the group I
work with aren't necessarily the strongest coders.  They're the ones
with the best knowledge of the (large) program's organization, how
to use the testing frameworks (CI system), and above all Git ;-).

One coding exercise I like is to write some simple program and then
tweak the code until you think it is perfect.  That's similar to
improving your writing by editing something you've written til you get
all the phrasing just right.

> - I never sit down and plan anything with pencil and paper,
> flowcharts, UML or anything else.

That's not worthwhile unless you're doing some very complicated,
and even then it's usually just some diagrams and some written docs.
Flowcharts were a 1950s thing and UML is a corporate Java thing.

> - I never learned algorithms and data structures. I know *what*
> (linked) lists, dicts, arrays, structs, and trees are; what binary
> search or bubble-sort is, but I never really studied them, let alone
> implemented them for educative purposes.

I'd say Python saves you from having to understand how stuff like linked
lists and hash tables work, at least at first.  It will become important
but not for now.  If you were programming in C it would all be important
and fundamental immediately.

But, read this: http://antirez.com/news/112

> - When it comes to coding, I'm heavily shy and unsure.

Do you play a musical instrument?  Think of how much more confident
you became with practice, even if you never got very good at it.

> But web programming always feel like being trapped in a
> mangrove jungle,

Yeah, I feel the same way and avoid that stuff.

> - I'm very hard to motivate, when the issue or topic doesn't interest
> me much. I know some tricks to increase my motivation in such cases,
> but don't use them enough.

For doing stuff on your own, it's fine to narrow them down to stuff
that excites you and that you want to keep doing.  It does help to
cultivate persistence for things like debugging.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: noCaptcha cracking handler

[Paul Rubin]

skyteacherus...@gmail.com writes:
> Please check out my latest repository
> https://github.com/gubrul/noCaptcha

Thanks.  Captchas have gotten to be a real PITA lately, even as someone
who doesn't want to run automated clients on the infected sites.  I
remember getting sick of trying to identify pictures of buildings with
store fronts, and thinking maybe there should be a bot to do them... oh
wait...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python package to accept payments in Internet

[Paul Rubin]

Chris Warrick  writes:
> there might be other users that will avoid your package for licensing
> reasons.

And there also might be other users who embrace the package for
those same reasons.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Ben Bacarisse  writes:
> ?  I get "AttributeError: 'itertools.dropwhile' object has no attribute
> 'next'" from your example.

Hmm, .next() worked ok for me in Python 2.7.5.  Not sure what happened.
Maybe something went wrong with my paste.  Oh well.

> Coming from the lazy language Haskell, I find your example natural...

Yep ;)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Looping [was Re: Python and the need for speed]

[Paul Rubin]

Ben Bacarisse  writes:
>   c = sys.stdin.read(1)
>   while c == ' ':
>   c = sys.stdin.read(1)

c = itertools.dropwhile(
 lambda c: c==' ',
 iter(lambda: sys.stdin.read(1),None)
 ).next()
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: GPUs with various tools

[Paul Rubin]

Jeffrey Layton  writes:
> I'm working with Continuum's Accelerate and I'm slowly learning PyCUDA
> but I'm looking for something that's already coded.

TensorFlow has good Python bindings.  That's probably the best way to
get started.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Steve D'Aprano  writes:
> On the other hand, there's Cython. Cython claims *not* to be a JIT compiler,

One of the uses of "JIT compiler" these days is what's sometimes called
a tracing JIT, like PyPy or LuaJIT or the Javascript flavor-of-the-week.
That means it interprets the program while collecting execution traces
to figure out the actual types the program uses at runtime, then
generates machine code for those specific code paths, with some guards
in case an unexpectedly typed value shows up.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Bigotry and hate speech on the python mailing list

[Paul Rubin]

Rurpy  writes:
> A couple weeks ago a frequent poster here (Steve D'Aprano
> ) called another participant an "ugly
> american"

Oh stop trolling.  Ugly American was a 1950s thing, now just amusing.
And context matters.  Steven, please try to be a little more attentive
to people's delicate feefees in the future.  Rurpy, should that take
care of it?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Static typing [was Re: Python and the need for speed]

[Paul Rubin]

Steve D'Aprano  writes:
> Since it is *optional*, it is only a hint, not a fact. You can tell the
> compiler that you believe that n will be an int, but it's not guaranteed.

The runtime could check at the entry to the function that n is an int,
and then it wouldn't have to keep re-checking on the inside of a loop.
That's what a JIT compiler does in the absence of annotations, more or
less; but the annotations make life easier for ahead-of-time compilers.
Again this type of thing has been standard in Lisp since the 1960's.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and the need for speed

[Paul Rubin]

bartc  writes:
> I do know that if I want to port some program (be it in Python or
> C++), or simply try and understand it, if I see it's full of class
> definitions or whatever, then I won't bother.

There was a time in the evolution of OOP when inheritance was thought of
as a cool and enabling thing for code re-use, and lots of architecture
astronautics involved designing deeply nested and complex inheritance
hierarchies in programs.  These days I think there's more accepted that
inheritance is confusing and obscures the control flow in programs, that
it was often mis-used, and that while there are still legitimate uses
cases for it, it should be used sparingly.  That helps quite a lot.

C++ templates let you write generics that are often cleaner than using
subclasses, though the design of templates can lead to awful code and
notoriously bloated and useless error messages.  The long-awaited
Concepts extension should help some with that.  Again though, like
anything else, templates work best when used tastefully rather than
willy-nilly.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Merging multiple sorted sequences.

[Paul Rubin]

Erik  writes:
> I need to be able to lazily merge a variable number of
> already-sorted(*) variable-length sequences 

If the number of sequences is large, the traditional way is with the
heapq module.
-- 
https://mail.python.org/mailman/listinfo/python-list