Re: [Python-Dev] Time for a change of random number generator?

[Robert Kern]

On 2016-02-11 00:08, Greg Ewing wrote:

The Mersenne Twister is no longer regarded as quite state-of-the art
because it can get into states that produce long sequences that are
not very random.

There is a variation on MT called WELL that has better properties
in this regard. Does anyone think it would be a good idea to replace
MT with WELL as Python's default rng?

https://en.wikipedia.org/wiki/Well_equidistributed_long-period_linear


There was a side-discussion about this during the secrets module proposal 
discussion.


WELL would not be my first choice. It escapes the excess-0 islands faster than 
MT, but still suffers from them. More troubling to me is that it is a linear 
feedback shift register, like MT, and all LFSRs quickly fail the linear 
complexity test in BigCrush.


xorshift* shares some of these flaws, but is significantly stronger and 
dominates WELL in most (all?) relevant dimensions.


  http://xorshift.di.unimi.it/

I'm favorable to the PCG family these days, though xorshift* and Random123 are 
reasonable alternatives.


  http://www.pcg-random.org/
  https://www.deshawresearch.com/resources_random123.html

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Time for a change of random number generator?

[Robert Kern]

On 2016-02-12 04:15, Tim Peters wrote:

[Greg Ewing ]

The Mersenne Twister is no longer regarded as quite state-of-the art
because it can get into states that produce long sequences that are
not very random.

There is a variation on MT called WELL that has better properties
in this regard. Does anyone think it would be a good idea to replace
MT with WELL as Python's default rng?


I don't think so, because I've seen no groundswell of discontent about
the Twister among Python users.  Perhaps I'm missing some?


Well me, but I'm mostly focused on numpy's PRNG, which is proceeding apace.

  https://github.com/bashtage/ng-numpy-randomstate

While I am concerned about MT's BigCrush failures, what makes me most 
discontented is not having multiple guaranteed-independent streams.



It's prudent to wait for someone else to find the early surprises in
PCG and Random123 too ;-)


Quite so!

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] potential argparse problem: bad mix of parse_known_args and prefix matching

[Robert Kern]

On 2013-11-26 18:16, Eli Bendersky wrote:


FWIW I'm not advocating a breaking behavior change here - I fully realize the
ship has sailed. I'm interested in mitigation actions, though. Making the
documentation explain this explicitly + adding an option to disable prefix
matching (in 3.5 since we're past the 3.4 features point) will go a long way for
alleviating this gotcha.


There is already the One Obvious Way to handle situations like yours: the user 
uses "--" to mark that the remaining arguments are pure arguments and not --options.


  parent-script --verbose -- ./child_script.py --no-verbose --etc

This is a common idiom across many argument processors. parse_known_args() is 
not a good solution for this situation even if you mitigate the prefix issue. 
Exact matches of --options are still possible.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python 3 niggle: None < 1 raises TypeError

[Robert Kern]

On 2014-02-18 14:11, MRAB wrote:

On 2014-02-18 13:48, Serhiy Storchaka wrote:

18.02.14 10:10, Paul Moore написав(ла):

Or alternatively, a "default on None" function - Oracle SQL calls this
nvl, so I will too:

def nvl(x, dflt):
 return dflt if x is None else x

results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1))


Or, as was proposed above:

results = sorted(invoices,
   key=lambda x: (x.duedate is not None, x.duedate))


That makes me wonder.

Why is:

 None < None

unorderable and not False but:

  (None, ) < (None, )

orderable?


tuple's rich comparison uses PyObject_RichCompareBool(x, y, Py_EQ) to find the 
first pair of items that is unequal. Then it will test the order of any 
remaining elements.


  http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/tupleobject.c#l591

PyObject_RichCompareBool(x, y, Py_EQ) treats identical objects as equal.

  http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/object.c#l716

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication

[Robert Kern]

On 2014-04-07 19:54, francis wrote:




So, I guess as far as I'm concerned, this is ready to go. Feedback

welcome:

   http://legacy.python.org/dev/peps/pep-0465/



Hi,
just curiosity: why is the second parameter 'o2' in:

PyObject* PyObject_MatrixMultiply(PyObject *o1, PyObject o2)

not a pointer to PyObject?


Typo, I'm fairly certain.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication

[Robert Kern]

On 2014-04-09 12:12, Nick Coghlan wrote:

On 8 April 2014 18:32, cjw  wrote:

Guido,

I am sorry to read this.

I shall be responding more completely in a day or two.

In my view, @ and @@ are completely redundant.  Both operations are  already
provided, * and **, in numpy.matrix.

PEP 465 provides no clear indication as to how the standard operators fail.


Note that numpy.matrix is specifically discussed in
http://www.python.org/dev/peps/pep-0465/#rejected-alternatives-to-adding-a-new-operator
(it's the first rejected alternative listed).


To be fair to Colin, the PEP asserts that the community at large would prefer an 
operator to the status quo but only alludes to the reason why it does so rather 
than explaining it fully. Personally, I think that's a reasonable allocation of 
Nathaniel's time, but then I happen to have agreed with the PEP's position 
before it was written, and I personally witnessed all of the history myself so I 
don't need it repeated back to me.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pep8 reasoning

[Robert Kern]

On 2014-04-24 14:59, Barry Warsaw wrote:


I will say this: the original preference for underscore_names in PEP 8 was
spurred by user studies some of our early non-native English speaking users
conducted many years ago.  We learned that it was more difficult for many of
them to parse mixedCase names than underscore_names.  I'm afraid I probably no
longer have references to those studies, but the difference was pronounced,
IIRC, and I think it's easy to see why.  Underscores can be scanned by the eye
as spaces, while I'd hypothesize that the brain has to do more work to read
mixedCase names.


A more recent set of studies show some mixedResults (ha ha). On a low-level 
reading task, the studies agree with yours in that mixedCase takes more time and 
effort; however, it appears to improve accuracy as well. On a higher-level 
comprehension task, mixedCase took less or the same time and still improved 
accuracy. Experienced programmers don't see too much of a difference either way, 
but inexperienced programmers see a more marked benefit to mixedCase.


  http://www.cs.loyola.edu/~binkley/papers/tr-loy110720.pdf‎

That said, I can't vouch for the experiments or the analysis, and it isn't 
really germane to Chris' historical question. I mention it only because I had 
just run across this paper last night, so it was fresh in my mind when you 
mentioned studies on the subject.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VC++ 2008 Express Edition now locked away?

[Robert Kern]

On 2013-03-06 16:55, Chris Angelico wrote:

On Thu, Mar 7, 2013 at 3:46 AM, Stefan Behnel  wrote:

Chris Angelico, 06.03.2013 17:30:

On Thu, Mar 7, 2013 at 1:40 AM, Ezio Melotti wrote:

I did try a few weeks ago, when I had to download a copy of Windows
for a project.  Long story short, after 30+ minutes and a number of
confirmation emails I reached a point where I had a couple of new
accounts on MSDN/Dreamspark, a "purchased" free copy of Windows in my
e-cart, and some .exe I had to download in order to download and
verify the purchased copy.  That's where I gave up.


That's the point where I'd start looking at peer-to-peer downloads.
These sorts of things are often available on torrent sites; once the
original publisher starts making life harder, third-party sources
become more attractive.


May I express my doubts that the license allows a redistribution of the
software in this form?


Someone would have to check, but in most cases, software licenses
govern the use, more than the distribution. If you're allowed to
download it free of charge from microsoft.com, you should be able to
get hold of it in some other way and it be exactly the same.


Sorry, but that's not how copyright works. The owner of the copyright on a work 
has to give you permission to allow you to distribute their work (modulo certain 
statutorily-defined exceptions that don't apply here). Just because you got the 
work from them free of charge doesn't mean that they have given you permission 
to redistribute it. If the agreements that you have with the copyright owner do 
not mention redistribution, you do not have permission to redistribute it.


IANAL, TINLA.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Semantics of __int__(), __index__()

[Robert Kern]

On 2013-04-03 13:47, Hrvoje Niksic wrote:

On 04/03/2013 01:17 PM, Nick Coghlan wrote:

 > > >
 > > I like Nick's answer to that: int *should* always return something of
 > > exact type int.  Otherwise you're always left wondering whether you
 > > have to do "int(int(x))", or perhaps even "int(int(int(x)))", to be
 > > absolutely sure of getting an int.
 >
 > Agreed.

Perhaps we should start emitting a DeprecationWarning for int subclasses
returned from __int__ and __index__ in 3.4?


Why would one want to be absolutely sure of getting an int?

It seems like a good feature that an __int__ implementation can choose to return
an int subclass with additional (and optional) information. After all, int
subclass instances should be usable everywhere where ints are, including in C
code.  I can imagine numpy and similar projects would be making use of this
ability already -- just think of uses for numpy's subclasses of "float".


We don't.

[~]
|1> type(float(np.float64(1.0)))
float

[~]
|2> type(int(np.int32(1)))
int

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __subclasses__() return order

[Robert Kern]

On 2013-05-25 09:18, Antoine Pitrou wrote:


Hello,

In http://bugs.python.org/issue17936, I proposed making tp_subclasses
(the internal container implementing object.__subclasses__) a dict.
This would make the return order of __subclasses__ completely
undefined, while it is right now slightly predictable. I have never seen
__subclasses__ actually used in production code, so I'm wondering
whether someone might be affected by such a change.


I do use a package that does use __subclasses__ in production code, but the 
order is unimportant.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

[Robert Kern]

On 2013-06-14 21:03, Brett Cannon wrote:


On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz mailto:masch...@gmail.com>> wrote:



  - add a `size` attribute to all objects (I wouldn't mind if this is None
in case you don't really know how to define the size of something, but it
would be good to have it, so that `anything.size` would never throw an error

This is what len() is for. I don't know why numpy doesn't define the __len__
method on their array types for that.


It does. It gives the size of the first axis, i.e. the one accessed by simple 
indexing with an integer: some_array[i]. The `size` attribute givens the total 
number of items in the possibly-multidimensional array. However, one of the 
other axes can be 0-length, so the array will have no elements but the length 
will be nonzero.


[~]
|4> np.empty([3,4,0])
array([], shape=(3, 4, 0), dtype=float64)

[~]
|5> np.empty([3,4,0])[1]
array([], shape=(4, 0), dtype=float64)

[~]
|6> len(np.empty([3,4,0]))
3

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

[Robert Kern]

On 2013-06-14 21:55, R. David Murray wrote:

On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz  wrote:

2. Testing for empty lists or empty ndarrays:

  In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
- `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
- `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
- `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

  Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.


In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

>>> bool(numpy.array([[]])
False
>>> bool(numpy.array([[1]])
True


numpy does not do so. Empty arrays are extremely rare and testing for them rarer 
(rarer still is testing for emptiness not knowing if it is an array or some 
other sequence). What people usually want from bool(some_array) is either 
some_array.all() or some_array.any(). In the face of this ambiguity, numpy 
refuses the temptation to guess and raises an exception explaining matters.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python

[Robert Kern]

On 2013-06-14 23:31, Robert Kern wrote:

On 2013-06-14 21:55, R. David Murray wrote:

On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz  wrote:

2. Testing for empty lists or empty ndarrays:

  In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
- `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
- `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
- `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

  Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.


In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

>>> bool(numpy.array([[]])
False
>>> bool(numpy.array([[1]])
True


numpy does not do so. Empty arrays are extremely rare and testing for them rarer
(rarer still is testing for emptiness not knowing if it is an array or some
other sequence). What people usually want from bool(some_array) is either
some_array.all() or some_array.any(). In the face of this ambiguity, numpy
refuses the temptation to guess and raises an exception explaining matters.


Actually, that's a bit of a lie. In the empty case and the one-element case, we 
do return a bool, False for empty and bool(element) for whatever that one 
element is. Anything else raises the exception since we don't know whether it is 
all() or any() that was desired.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bdb.py trace C implementation?

[Robert Kern]

On 2009-04-01 17:53, Benjamin Peterson wrote:

2009/4/1 Guido van Rossum:

Tracing has other uses besides debugging though.


The OP said he wished to implement a C trace function for bdb.
Wouldn't that make it only applicable to debugging?


Once you are at the breakpoint and stepping through the code manually, the 
performance is not all that important. However, up until that breakpoint, you 
are running a lot of code "in bulk". It would be useful to have a performant 
trace function that interferes with that code the least.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue5434: datetime.monthdelta

[Robert Kern]

On 2009-04-16 13:42, Paul Moore wrote:

2009/4/16 Ned Deily:

In article
,
  Jess Austin  wrote:

I'm new to python core development, and I've been advised to write to
python-dev concerning a feature/patch I've placed at
http://bugs.python.org/issue5434, with Rietveld at
http://codereview.appspot.com/25079.

Without having looked at the code, I wonder whether you've looked at
python-dateutil.   I believe its relativedelta type does what you
propose, plus much more, and it has the advantage of being widely used
and tested.


The key thing missing (I believe) from dateutil is any equivalent of monthmod.

Hmm, it might be possible via relativedelta(d1,d2), but it's not clear
to me from the documentation precisely what attributes/methods of a
relativedelta object are valid for getting data *out* of it.


I thought the examples were quite clear. relativedelta() has an alternate 
constructor precisely suited to these calculations but is general and handles 
more than just months.


>>> from dateutil.relativedelta import *
>>> dt = relativedelta(months=1)
>>> dt
relativedelta(months=+1)
>>> from datetime import datetime
>>> datetime(2009, 1, 15) + dt
datetime.datetime(2009, 2, 15, 0, 0)
>>> datetime(2009, 1, 31) + dt
datetime.datetime(2009, 2, 28, 0, 0)
>>> dt.months
1
>>> datetime(2009, 1, 31) + relativedelta(years=-1)
datetime.datetime(2008, 1, 31, 0, 0)

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue5434: datetime.monthdelta

[Robert Kern]

On 2009-04-16 17:17, Paul Moore wrote:

2009/4/16 Robert Kern:



from dateutil.relativedelta import *
dt = relativedelta(months=1)
dt

relativedelta(months=+1)

from datetime import datetime
datetime(2009, 1, 15) + dt

datetime.datetime(2009, 2, 15, 0, 0)

datetime(2009, 1, 31) + dt

datetime.datetime(2009, 2, 28, 0, 0)

dt.months

1

datetime(2009, 1, 31) + relativedelta(years=-1)

datetime.datetime(2008, 1, 31, 0, 0)


Yes, but given

r = relativedelta(d1, d2)

how do I determine the number of months between d1 and d2, and the
"remainder" - what monthmod gives me.


Oops! Sorry, I read too quickly and misread "monthmod" as "monthdelta".


From the code, r.months looks
like it works, but it's not documented, and I'm not 100% sure if it's
always computed.


The result of relativedelta(d1, d2) is the same thing as if it were explicitly 
constructed from the years=, months=, etc. keyword arguments. From this example, 
I think this is something that can be relied upon:


"""
It works with dates too.

>>> relativedelta(TODAY, johnbirthday)
relativedelta(years=+25, months=+5, days=+11, hours=+12)
"""


The use case I'm thinking of is converting the difference between 2
dates into "3 years, 2 months, 5 days" or whatever. I've got an
application which needs to get this right for one of the dates being
29th Feb, so I *really* get to exercise the corner cases :-)


I believe relativedelta() is intended for this use case although it may resolve 
ambiguities in a different way than you were hoping.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue5434: datetime.monthdelta

[Robert Kern]

On 2009-04-16 21:55, s...@pobox.com wrote:

 Jess>  If, on the other hand, one of the committers wants to toss this in
 Jess>  at some point, whether now or 3 versions down the road, the patch
 Jess>  is up at bugs.python.org (and I'm happy to make any suggested
 Jess>  modifications).

Again, I think it needs to bake a bit.  I understand the desire and need for
doing date arithmetic with months.  Python is mature enough though that I
don't think you can just "toss this in".  It should be available as a module
outside of Python so people can beat on it, flush out bugs, make suggestions
for enhancements, whatever.  I believe you mentioned putting it up on PyPI.
I think that's an excellent idea.

I've used parts of Gustavo Niemeyer's dateutil package for a couple years
and love it.  It's widely used.  Adding it to dateutil seems like another
possibility.  That would guarantee an instant user base.  From there, if it
is found to be useful it could make the leap to be part of the datetime
module.


dateutil.relativedelta appears to do everything monthdelta does and more in a 
general way. Adding monthdelta to dateutil doesn't seem to make much sense.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Easy way to detect filesystem case-sensitivity?

[Robert Kern]

On 2009-05-07 18:29, Brett Cannon wrote:

[my python-dev sabbatical is still in effect, so make sure I am at least
cc'ed on any replies to this email]

I cannot be the only person who has a need to run tests conditionally
based on whether the file system is case-sensitive or not, so I feel
like I am re-inventing the wheel for issue 5442 to handle OS X with a
case-sensitive filesystem. Is there a boolean somewhere that I can
simply check or get to know whether the filesystem is case-sensitive?


Since one may have more than one filesystem side-by-side, this can't be just be 
a system-wide boolean somewhere. One would have to query the target directory 
for this information. I am not aware of the existence of code that does such a 
query, though.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Indentation oddness...

[Robert Kern]

On 2009-05-29 19:08, Dino Viehland wrote:

Consider the code:

code = "def  Foo():\n\npass\n\n  "

This code is malformed in that the final indentation (2 spaces) does not agree 
with the previous indentation of the pass statement (4 spaces).  Or maybe it's 
just fine if you take the blank lines should be ignored statement from the docs 
to be true.  So let's look at different ways I can consume this code.

If I use compile to compile this:

compile(code, 'foo', 'single')

I get an IndentationError: unindent does not match any outer indentation level

But if I put this in a file:

f= file('indenttest.py', 'w')
f.write(code)
f.close()
import indenttest

It imports just fine.


The 'single' mode, which is used for the REPL, is a bit different than 'exec', 
which is used for modules. This difference lets you insert "blank" lines of 
whitespace into a function definition without exiting the definition. Ending 
with a truly empty line does not cause the IndentationError, so the REPL can 
successfully compile the code, signaling that the user has finished typing the 
function.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Indentation oddness...

[Robert Kern]

On 2009-05-30 21:02, Greg Ewing wrote:

Robert Kern wrote:


The 'single' mode, which is used for the REPL, is a bit different than
'exec', which is used for modules. This difference lets you insert
"blank" lines of whitespace into a function definition without exiting
the definition.


All that means is that the REPL needs to keep reading
lines until it gets a completely blank one. I don't
see why the compiler has to treat the source any
differently once the REPL has decided how much text
to feed it.


Not true. The REPL will keep reading lines until 
compile(code,'','single') passes without a SyntaxError. You can put in 
blank lines when line continuation is implicit, like in the middle of a list. 
This is the reason that there is a 'single' mode in the first place, to 
determine when you've stopped typing. It's easier to add the grammar rule that a 
block does not end with a line of whitespace to the compiler than to implement 
all of the context-specific special cases for pure empty lines outside of the 
compiler.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] syntax warnings don't go through warnings.warn?

[Robert Kern]

On 2009-06-01 11:50, Dino Viehland wrote:

I’m just a little surprised by this - Is there a reason why syntax
warnings are special and untrappable via warnings.warn?

 >>> import warnings

 >>> def mywarn(*args): print 'xx', args

...

 >>> warnings.warn = mywarn

 >>> compile("def f():\n a = 1\n global a\n", "", "exec")

:3: SyntaxWarning: name 'a' is assigned to before global declaration

 at 012DB410, file "", line 1>


symtable.c uses PyErr_WarnExplicit() to provide file and line number 
information. You need to stub warnings.warn_explicit().


>>> import warnings
>>> def mywarn_explicit(*args):
... print 'xx', args
...
>>> warnings.warn_explicit = mywarn_explicit
>>> compile("def f():\n a = 1\n global a\n", "", "exec")
xx ("name 'a' is assigned to before global declaration", 'exceptions.SyntaxWarning'>, '', 3, None, None)

 at 0x39e9f8, file "", line 1>

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update to Python Documentation Website Request

[Robert Kern]

On 2009-07-25 17:28, Brett Cannon wrote:



On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone mailto:exar...@divmod.com>> wrote:

On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney
mailto:ben%2bpyt...@benfinney.id.au>>
wrote:

[snip]


If that is not your intent, then your application shouldn't be
mentioned
in standard Python documentation.


Hm.  But docutils isn't part of the standard library, and the
documentation
refers to it.


Why do you think distutils is not part of the standard library? The
official code location is Lib/distutils within the standard library.


He said "docutils", not "distutils".

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] deleting setdefaultencoding iin site.py is evil

[Robert Kern]

On 2009-08-25 12:37 PM, Guido van Rossum wrote:

In retrospect, it should have been called sys._setdefaultencoding().
That sends an extra signal that it's not meant for general use.


Considering all of the sys._getframe() hacks out there, I suspect that this 
would encourage more abuse of the function than the current situation.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] deleting setdefaultencoding iin site.py is evil

[Robert Kern]

On 2009-08-25 13:29 PM, Guido van Rossum wrote:

On Tue, Aug 25, 2009 at 11:10 AM, Robert Kern  wrote:

On 2009-08-25 12:37 PM, Guido van Rossum wrote:


In retrospect, it should have been called sys._setdefaultencoding().
That sends an extra signal that it's not meant for general use.


Considering all of the sys._getframe() hacks out there, I suspect that this
would encourage more abuse of the function than the current situation.


Why? It would still be deleted by site.py.


Ah, yes. You're right. For whatever reason I thought it lived as 
site.setdefaultencoding() when I read your message and thought that you were 
proposing to move it to sys. Never mind me.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Robert Kern]

On 2009-09-28 10:37 AM, Steven Bethard wrote:

On Mon, Sep 28, 2009 at 8:26 AM, Michael Foord
  wrote:

m h wrote:


Perhaps this is OT, but since command line parsing is part of
configuration, I figure I'd throw it out there.  My scripts often have
configuration that the command line can override and I loosely follow
the example hierarchy[0] listed in The Art of Unix Programming.

Some configuration I want in a config file (but I want to override
from the command line) and sometimes it's very nice to use environment
variables for configuration.  So I do something like this:


Integration with command line options is an occasionally-requested feature
for ConfigObj. I've always said I would be open to patches...

In other words, yes I think there is demand for it. Whether it belongs
*immediately* in the standard library is another matter, but if you wrote a
layer that unified ConfigParser and argparse I think you will find that
people use it.

It is well outside the realm of this PEP however.


What Michael said. ;-) If you'd like to provide a patch for argparse
that provides such functionality, I'd be happy to review it.


cfgparse appears to fill this role for optparse. It might be straightforward to 
port it to argparse.


http://cfgparse.sourceforge.net/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Robert Kern]

On 2009-09-29 18:38 PM, Steven Bethard wrote:


I don't really use GUI libraries, so I wouldn't be able to come up
with such an example. I'd also rather not make API changes based on
speculative use cases, so before I spend time documenting these
things, I'd really like to hear from someone who has already, say,
used getopt or optparse in conjunction with a GUI library, and what
feedback they have about that.


I use argparse (and previously optparse) frequently to handle the command line 
arguments of GUI apps. I tend to use them in the same way as CLI programs, 
though, since I usually only use command line arguments when starting the GUIs 
from the terminal. I am blissfully unaware of the problems Paul mentioned about 
Windows GUI-mode programs. I'm not sure what would make a program "GUI-mode" or 
not. Certainly, I have written Python programs that use wxPython and PyQt on 
Windows that print to stdout/stderr, and they appear to work fine.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Robert Kern]

On 2009-09-30 11:38 AM, Guido van Rossum wrote:

On a tangent -- a use case that I see happening frequently but which
is pretty poorly supported by optparse is a command-line that has a
bunch of general flags, then a 'subcommand name', and then more flags,
specific to the subcommand. Most here are probably familiar with this
pattern from SVN, Hg, and other revision control systems (P4 anyone?)
with a rich command-line interface. There are some variants, e.g.
whether global and subcommand-specific flags may overlap, and whether
flags may follow positional args (Hg and SVN seem to differ here a
bit).

I've helped write at least two tools at Google that have this
structure; both used different approaches, and neither was
particularly easy to get right. Getting all the different --help
output to make sense was mostly a manual process. (E.g. "foo --help"
should print the general flags and the list of known subcommands,
whereas "foo --help subcommand" should print flags and other usage
info about the specific subcommand.) Also switching out to different
calls based on the subcommand should/might be part of this.

I would be willing to live with a third option parser in the stdlib if
it solved this problem well.


I don't think argparse supports the "foo --help subcommand" OOB. I think it 
would be simple to modify argparse to make it do so. It does support general 
options followed by a subcommand with options, though.


http://argparse.googlecode.com/svn/tags/r101/doc/other-methods.html#sub-commands

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Robert Kern]

On 2009-09-30 15:17 PM, Andrew McNabb wrote:

On Wed, Sep 30, 2009 at 02:22:53PM -0400, Barry Warsaw wrote:


Right.  I've made it kind of work in Mailman 3, but it would be nice for
argparse to support this out of the box.  Note that I think you want two
forms:

foo help subcommand
foo subcommand --help

to basically print the same help text.  This is the way bzr does it for
example and it works great.


In some commands, options as well as subcommands can change subsequent
parsing.  The iptables command is a good example of a command-line
program that follows this practice.  From the man page:

   after [a module name is specified], various extra command line options
   become available, depending on the specific module.  You can specify
   multiple extended match modules in one line, and you can use the -h or
   --help options after the module has been specified to receive help
   specific to that module.

In the case of iptables, module names are specified as options, not as
subcommands.


From my cursory reading of the documentation, it looks like argparse can

only add subparsers for subcommands.  Is there any way to add subparsers
based on options instead (as iptables does)?


I have not done so, but I suspect so. The implementation of .add_subparsers() 
adds to the positional argument list, but one could be written to append to the 
option list.



Also, is it possible to add these subparsers dynamically?  For example,
you would want to be able to load a module immediately after parsing the
name instead of having to keep a predetermined list of all module names.
I'm pretty sure that bzr dynamically loads modules this way.  Can
argparse help with this?


Not out-of-box, but it looks fairly straightforward to plug in. The subparser 
logic is mostly encapsulated in the _SubparsersAction class. You can register a 
new class for it:


parser.register('action', 'parsers', MySubParsersAction)

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 389: argparse - new command line parsing module

[Robert Kern]

Steven D'Aprano wrote:

On Sun, 4 Oct 2009 04:46:19 am Yuvgoog Greenle wrote:


I just think that if a parser error is causing the SystemExit, I
would rather catch a parser error than catching a SystemExit for the
sake of readability. It saves me the comments:

# Catching SystemExit because parse_args() throws SystemExit on
parser errors.


But why are you catching the error? As a general rule, you *want* your 
command line app to exit if it can't understand the arguments you pass 
to it. What else can it do? Guess what you wanted?


There are uses of argparse outside of command line apps. For example, I use it 
to parse --options for IPython %magic commands. Of course, I just subclass 
ArgumentParser and replace the .error() method. I require a particular IPython 
exception type in order for the error to be recognized correctly in the %magic 
subsystem.


The other use case, as mentioned earlier, was for debugging your parser on the 
interactive prompt. A custom subclass may also be able to hold more 
machine-readable information about the error than the formatted error message, 
but I don't have any particular use cases about what may be the most useful.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backport new float repr to Python 2.7?

[Robert Kern]

Guido van Rossum wrote:

On Sun, Oct 11, 2009 at 11:28 AM, Mark Dickinson  wrote:

In a recent #python-dev IRC conversation, it was suggested that we
should consider backporting the new-style float repr from py3k to
trunk.  I'd like to get people's opinions on this idea.

[...]

Possible problems:

 - breaking docstrings in third party code.  Though Eric reminded me
  that when we implemented this for 3.1, there were essentially no
  standard library test breakages resulting from the changed repr
  format.


I think you mean doctests? These are the primary reason I've always
been hesitant to change this in 2.x.


I think that doctests inherently unsuited to testing floating point algorithms. 
Leaving aside the platform differences in actual floating point arithmetic that 
cause minute real differences in the results, Python 2.x relies on string 
conversion routines which give different doctest results on different platforms. 
Using a consistent routine would actually improve the ability to use doctests in 
that one regard. It certainly would make writing examples much more consistent, 
particularly for those of us that use infs and nans frequently.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Refactoring installation schemes

[Robert Kern]

On 2009-11-05 16:29 PM, David Lyon wrote:


But I've been on the list for some twelve months asking for work
to help out with, and haven't been assigned a single task to do
yet.

Seriously, if you won't allocate work out to people then how can
it get done?


With rare exceptions, open source projects don't really "assign" work. If you 
want to help, help. Don't wait for someone to tell you exactly what to do. No 
one will. Go to the tracker, find something interesting, and do it.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] decimal.py: == and != comparisons involving NaNs

[Robert Kern]

Antoine Pitrou wrote:

Stefan Krah  bytereef.org> writes:

Are there cases where == and != are actually needed to give a result
for NaNs?


It is a common expectation that == and != always succeed. They return True or
False, but don't raise an exception even on unrelated operands:


It is a common expectation, but a false one. __eq__ and __ne__ are explicitly 
allowed to return anything, not just bools.


http://www.python.org/dev/peps/pep-0207/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyPI comments and ratings, *really*?

[Robert Kern]

A.M. Kuchling wrote:

On Fri, Nov 13, 2009 at 11:44:42AM +1100, Ben Finney wrote:

There's a problem with the poll's placement: on the front page of the
PyPI website.


I've posted a tweet to the ThePSF account about the poll.  If the poll
runs for a week or two, that would provide time for word of the poll
to propagate through Twitter, blogs, etc.


You should also make an announcement on python-announce.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyPI comments and ratings, *really*?

[Robert Kern]

[We really should be discussing this on catalog-sig, but moving things 
mid-thread never works, so here we go. I apologize to python-dev. I also 
apologize for the length.]


On 2009-11-13 17:18 PM, Steven D'Aprano wrote:

On Sat, 14 Nov 2009 09:57:18 am Ben Finney wrote:

"A.M. Kuchling"  writes:

If popular vote is ruled out, I don't see who else could possibly
make the decision to disable comments and/or ratings.


Reasoned argument with the person who decides. A bad idea with many
people's support is still a bad idea; a good idea with few people's
support is still a good idea.


Okay, let's talk reasoned debate.

I understand the reason for making comments compulsory: they're for the
benefit of the users, not the package owner. It helps prevent
information about the package from being fragmented: there is One
Obvious place to find out about a package on PyPI, which is the PyPI
page, instead of having to search for blogs where people may or may not
have made comments about the package. If individual package owners
don't like this, too bad, because PyPI is run for the benefit of the
community, not individual package owners.

I understand the reason for making comments optional: personal choice of
the package owner is valuable in and of itself, even if it is against
the best interests of the community.

But for the life of me, I can't understand the 1/3 of the votes that
have been cast in favour of prohibiting comments for everybody, even
those who want comments.


While I do have a couple of packages on PyPI, I use PyPI as a consumer of 
packages much more frequently, every day in fact. I voted against comments and 
ratings because I think it is a detriment to my experience of PyPI as a user (I 
also think that they will be a detriment to the community, but that's a 
prediction, not a fact). Short form comments are simply not useful to me. 
Ratings are worse. They do not carry reliable information; they carry short 
statements of opinion from a wide variety of experiences, most of which are 
entirely unrelated to my own needs.


To make an example, I have a lot of experience making ornery stuff build. A lot 
of other people don't. Their personal experience of not managing to install a 
package correctly turns into a weird, objective-seeming statement that the 
"package is difficult to build". People have different thresholds, different 
experiences, and different standards.


When such opinions get numerically aggregated by a monolithic rating system, 
that's even worse. Even with short-form comments, they have the ability, though 
often not the propensity, to give a little bit of information ("I had trouble 
building it on Windows.") that can help me tease out whether their experiences 
will be relevant to mine, but one star is just one star.


I *do* like to read long-form reviews where people relate what their needs were, 
what they tried to use the package for, and exactly where the package succeeded 
and where it failed. I can compare that context with my own needs and extract 
the relevant parts of their experience. Blogs are great for that.


Now I do appreciate ratings and comments on shopping sites if they don't provide 
the capability for long-form reviews. But that's because the product is locked 
behind the barrier of payment and often shipping. There is no such barrier on 
PyPI. If I can get to a web view of their repository, thirty seconds with it 
will give a much better idea of whether the package is worth trying than any 
amount of comments I could read in that time. I can easily see how much 
documentation they have, if they have funny build requirements, if they are just 
prototyping, etc. without needing to trust that everyone else has needs and 
standards like mine. That's the key difference between comments and ratings on 
shopping sites and why I don't think that their success in that field translates 
to here.


If you want one idea that would make my use of PyPI much more pleasant and 
informative, it would be to add a "Repository-URL" entry to the recommended PyPI 
metadata so that I could always start looking at the code in one click. Or 
integrate the source code browsing into PyPI itself; it could open up the sdists 
and eggs and show a WebVCS-like browser interface.


Now, these are all reasons why commenting and rating are not beneficial to me. 
Where I think they are harmful is that when one is exposed to them, one cannot 
just ignore them. They have an emotional, unreasonable impact whether I want 
them to or not. And that's why I do not want to see them. Give me access to 
information, not opinions. If authors do want comments and ratings on their 
packages, let them use the services that already exist for *just* that purpose 
like Ohloh. They have the time and energy to implement these mechanisms with the 
care and attention that they need.


--
Rob

Re: [Python-Dev] Modules that run other scripts

[Robert Kern]

Nick Coghlan wrote:


For that second part:
1. Is it even worth doing at this stage? I'm not sure to what degree the
new command line flexibility has even been adopted by third party
application packagers, so I have no idea how large the pool of potential
users might be. Should I instead wait until we start seeing complaints
that these tools can't be used with script references that the main
command line will handle quite happily?


There is a small, but important class of "scripts that run scripts", which are 
mostly all development tools (e.g. coverage, my line_profiler, etc.). Doing this 
correctly in all of the corner cases is reasonably tricky, so I think this is a 
perfect case for having the functionality implemented once in the standard library.


For what its worth, I think Ned Batchelder did the most thorough job of 
implementing this in the latest version of coverage:


  http://bitbucket.org/ned/coveragepy/src/tip/coverage/execfile.py

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Kern]

On 2010-02-09 15:57 PM, Ben Finney wrote:


Is there a better third-party framework for use in these cases? As
Olemis points out later in this thread, I don't think it's good for the
‘unittest’ module to keep growing for uses that aren't focussed on unit
tests (as contrasted with other kinds of tests).


nosetests allows you to write such module-level and class-level setup and 
teardown functions.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Kern]

On 2010-02-11 16:20 PM, Ben Finney wrote:

Guido van Rossum  writes:



The argument that a unittest framework shouldn't be "abused" for
regression tests (or integration tests, or whatever) is also bizarre
to my mind. Surely if a testing framework applies to multiple kinds of
testing that's a good thing, not something to be frowned upon?


To my mind, an API should take a stand on the “right” way to use it,
rather than being a kitchen-sink of whatever ideas had any support.
Doing something the right way should be easy, and doing something the
wrong way should be awkward.


setUpClass and setUpModule are the "right" way to do many types of integration 
and functional tests. Integration and functional tests are vital tasks to 
perform, and unittest provides a good framework otherwise for implementing such 
tests.



There are several alternative testing frameworks available outside the
standard library. The provide useful competition with the stlib's
unittest and doctest modules, and useful inspiration for potential new
features. They also, by and large, evolve much faster than a stdlib
module ever could, and including anyone of these in the stdlib might
well be the death of it (just as unittest has evolved much slower
since it was included).


Right. This is an argument in favour of being assertive and parsimonious
in the design of the standard-library ‘unittest’ API: this is the clear
and obvious way to use this API, and if someone wants to do it a
different way there are alternatives available.


I would agree if the requirements for unit testing and integration/functional 
tests were so different. However, unittest provides most of the necessary 
infrastructure that is common to all of those kinds of testing. It's just that 
the latter kinds of tests also could use setUpClass/setUpModule. It would be a 
waste (and honestly kind of ridiculous) to force people to use a whole new 
framework (which would duplicate unittest in almost its entirety) for want of 
those two methods.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Kern]

On 2010-02-11 17:57 PM, Holger Krekel wrote:

On Fri, Feb 12, 2010 at 12:00 AM, Robert Kern  wrote:

On 2010-02-11 16:20 PM, Ben Finney wrote:


Guido van Rossumwrites:



The argument that a unittest framework shouldn't be "abused" for
regression tests (or integration tests, or whatever) is also bizarre
to my mind. Surely if a testing framework applies to multiple kinds of
testing that's a good thing, not something to be frowned upon?


To my mind, an API should take a stand on the “right” way to use it,
rather than being a kitchen-sink of whatever ideas had any support.
Doing something the right way should be easy, and doing something the
wrong way should be awkward.


setUpClass and setUpModule are the "right" way to do many types of
integration and functional tests. Integration and functional tests are vital
tasks to perform, and unittest provides a good framework otherwise for
implementing such tests.


Ben just expressed his opinion about API design and you claim some
truth about testing in general.


My first sentence was about API design. My second was justification that the use 
case is worth designing and API for. You can add implicit "in my opinion"s to 
just about anything I say, if you wish.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ugliness

[Robert Kern]

On 2010-03-08 15:20 PM, Greg Ewing wrote:

Mark Russell wrote:

Boolean flags are a common enough case that I'd be inclined to add a
wrapper method,

parser.add_bool_argument('--plot')


+1, this looks good.


I've added it to the argparse bugtracker, along with my suggested spelling 
add_flag():


  http://code.google.com/p/argparse/issues/detail?id=62

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] argparse ugliness

[Robert Kern]

On Tue, Mar 9, 2010 at 11:31, R. David Murray  wrote:
> On Mon, 08 Mar 2010 15:35:46 -0600, Robert Kern  wrote:
>> On 2010-03-08 15:20 PM, Greg Ewing wrote:
>> > Mark Russell wrote:
>> >> Boolean flags are a common enough case that I'd be inclined to add a
>> >> wrapper method,
>> >>
>> >> parser.add_bool_argument('--plot')
>> >
>> > +1, this looks good.
>>
>> I've added it to the argparse bugtracker, along with my suggested spelling
>> add_flag():
>>
>>    http://code.google.com/p/argparse/issues/detail?id=62
>
> Shouldn't argparse bugs/enhancement tickets be going to bugs.python.org
> now?

Probably. Having used the the argparse tracker before, though, it was
the first place I thought to file it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Decimal <-> float comparisons in py3k.

[Robert Kern]

On 2010-03-18 13:27 PM, Glenn Linderman wrote:


As any non-naïve float user is aware, the proper form of float
comparisons is not to use < or > or == or !=, but rather, instead of
using < (to follow along with your example), one should use:

Decimal('1.1') - 2.2 < epsilon


Not at all. This is quite incorrect for most use cases. Fuzzy comparisons are 
sometimes useful for equality testing, but almost never for ordering.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is nan != nan?

[Robert Kern]

On 2010-03-27 00:32 , David Cournapeau wrote:

On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
  wrote:


On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:

How about raising an exception instead of creating nans in the first place,
except maybe within specific contexts (so that the IEEE-754 minded can get
their nans working as they currently do)?

-1
The numeric community uses NaNs as placeholders in vectorized calculations.


But is this relevant to python itself ? In Numpy, we indeed do use and
support NaN, but we have much more control on what happens compared to
python float objects. We can control whether invalid operations raises
an exception or not, we had isnan/isfinite for a long time, and the
fact that nan != nan has never been a real problem AFAIK.


Nonetheless, the closer our float arrays are to Python's float type, the happier 
I will be.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is nan != nan?

[Robert Kern]

On 2010-03-27 13:36 , Adam Olsen wrote:

On Fri, Mar 26, 2010 at 17:16, Raymond Hettinger
  wrote:

Of the ideas I've seen in this thread, only two look reasonable:
* Do nothing.  This is attractive because it doesn't break anything.
* Have float.__eq__(x, y) return True whenever x and y are
the same NaN object.  This is attractive because it is a
minimal change that provides a little protection for
simple containers.
I support either of those options.


What's the flaw in using isnan()?


There are implicit comparisons being done inside list.__contains__() and other 
such methods. They do not, and should not, know about isnan().


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is nan != nan?

[Robert Kern]

On 2010-03-29 01:17 AM, David Cournapeau wrote:

On Sun, Mar 28, 2010 at 9:28 AM, Robert Kern  wrote:

On 2010-03-27 00:32 , David Cournapeau wrote:


On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
wrote:


On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:

How about raising an exception instead of creating nans in the first
place,
except maybe within specific contexts (so that the IEEE-754 minded can
get
their nans working as they currently do)?

-1
The numeric community uses NaNs as placeholders in vectorized
calculations.


But is this relevant to python itself ? In Numpy, we indeed do use and
support NaN, but we have much more control on what happens compared to
python float objects. We can control whether invalid operations raises
an exception or not, we had isnan/isfinite for a long time, and the
fact that nan != nan has never been a real problem AFAIK.


Nonetheless, the closer our float arrays are to Python's float type, the
happier I will be.


Me too, but I don't see how to reconcile this with the intent of
simplifying nan handling because they are not intuitive, which seems
to be the goal of this discussion.


"Do nothing" is still on the table, I think.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] what to do if you don't want your module in Debian

[Robert Kern]

On 4/26/10 4:46 PM, Barry Warsaw wrote:

On Apr 26, 2010, at 09:39 PM, Tarek Ziadé wrote:


You should be permissive on that one. Until we know how to describe resource
files properly, __file__ is what developer use when they need their projects
to be portable..


Until then, isn't pkg_resources the best practice for this?  (I'm pretty sure
we've talked about this before.)


I don't think the OP is really speaking against using __file__ per se, but 
rather putting data into the package however it is accessed. The Linux-packager 
preferred practice is to install into the appropriate /usr/shared/ subdirectory. 
Writing portable libraries (with portable setup.py files!) is difficult to do 
that way, though.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] query: docstring formatting in python distutils code

[Robert Kern]

On 7/7/10 1:53 PM, Éric Araujo wrote:

I promised to write a PEP about that some time in the future.  (Probably after
3.2 final.)


Nice.

It seems that projects putting Sphinxy reST in their doc are using
automatic doc generation. This is however not always the best way to
make good doc, as demonstrated by Python’s hand-written,
very-high-quality documentation.


This is a false dichotomy. Many of those projects using Sphinxy reST in their 
docstrings are using the semi-automatic[1] doc generation provided by Sphinx to 
construct *part* of their documentation. Namely, the reference of functions, 
classes and methods. A large part of Python's library reference consists of 
exactly this. Having a function's docstring provide the content for its entry in 
the library reference has the substantial DRY benefit of having exactly one 
source for the comprehensive documentation of that function available from both 
help() and the manual. As someone who uses the interactive prompt to play with 
things and read docstrings intensively, I would really like to see docstrings 
providing the same information as the manual.


Of course, opinions differ about how comprehensive docstrings should be compared 
to the reference manual's entries. And there are technical reasons for not 
wanting to try to extract docstrings from code (e.g. platform-specific modules). 
But one should not fear that the quality of the manual would decline.


[1] That's the really nice thing about Sphinx's docstring extraction features in 
contrast with other such tools. It doesn't generate a manual from the 
docstrings; it makes you explicitly reference the docstrings into the manual's 
text. This would fit in very naturally with Python's library reference.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] We should be using a tool for code reviews

[Robert Kern]

On 9/29/10 3:33 PM, Guido van Rossum wrote:

On Wed, Sep 29, 2010 at 1:31 PM, Alexander Belopolsky
  wrote:

On Wed, Sep 29, 2010 at 4:23 PM, Guido van Rossum  wrote:
..

But maybe with Hg it's less of a burden to ask people to use a checkout.


I thought with Hg it would be more of a burden for casual contributors
to use a checkout simply because the checkout is many times bigger.


Isn't it still faster though?


For what it's worth, when I tried it, the relationship is reversed:

[hg]$ time hg clone http://code.python.org/hg/branches/py3k/
...
hg clone http://code.python.org/hg/branches/py3k/  24.44s user 3.43s system 10% 
cpu 4:15.48 total


[hg]$ du -hsc py3k
105Mpy3k

[svn]$ time svn co http://svn.python.org/projects/python/branches/py3k/
...
svn co http://svn.python.org/projects/python/branches/py3k/  5.03s user 7.01s 
system 12% cpu 1:35.97 total

[svn]$ du -hsc py3k
131Mpy3k


Mercurial's checkout with the whole history is actually smaller than the SVN 
checkout because it applies some very nice compression to the history whereas 
the SVN checkout has an uncompressed copy of every single file tucked away in 
its .svn/ directory.


My mercurial checkout happens to be slower, but I don't know whose fault that 
is. I'm still using Mercurial 1.5.1 on my OS X box, and while I wasn't doing 
much that would take up CPU or bandwidth, I wasn't trying especially hard to 
prevent such interference, either.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __setcall__

[Robert Kern]

On 10/26/10 5:05 PM, Bj Raz wrote:

I'll look into SAGE, I am still curious if there is, a way, to write this in 
native python, cause I'm currently plotting in Autodesk Maya, and using Python.


This thread is off-topic for python-dev, which is intended for the development 
*of* the Python interpreter, not development *in* Python. We should continue 
this discussion on python-list, instead:


  http://mail.python.org/mailman/listinfo/python-list

That said, I will answer your question. No, there is no way to do this in 
Python. However, I don't think that is what the code you are trying to emulate 
does. It looks like it is building up a list of values, not defining a function. 
If you take your question over to python-list, I can explain more.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Robert Kern]

Martin v. Löwis wrote:
> Travis E. Oliphant schrieb:
>> What is needed is a definitive way to describe data and then have
>>
>> array
>> struct
>> ctypes
>>
>> all be compatible with that same method.That's why I'm proposing the 
>> PEP.  It's a unification effort not yet-another-method.
> 
> As I unification mechanism, I think it is insufficient. I doubt it
> can express all the concepts that ctypes supports.

What do you think is missing that can't be added?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Improvements / Rewrite

[Robert Kern]

Phillip J. Eby wrote:
> At 12:16 AM 1/6/2007 -0500, Barry Warsaw wrote:
>> If you've already explained it, that's fine, but if not, could you
>> outline what you have against epydoc?
> 
> The last I tried to work with it, it had even more hardcoded typechecking 
> than pydoc does, spread out over more of the code base.  Also, over at 
> OSAF, I've been repeatedly called upon to sort out some peculiarity in how 
> it discovers things or how it handles types it doesn't recognize.
> 
> My net impression has been that it's very brittle, even more so than pydoc.
> 
> On the plus side, there are some very good ideas and a *lot* of good 
> execution in there, but its extensibility has historically struck me as 
> non-existent.
> 
> To be fair, the last time I had to deal with any problems with it at OSAF 
> was almost a year ago, if memory serves.  I don't know if anything has 
> improved in it since then.

FWIW, a 3.0a3 was released in August 2006, and according to the History,
"Significant portions of epydoc were written for version 3.0." It seems a lot of
that was to add parsing as a complementary means to extract documentation. I'm
not particularly familiar with the introspection code of either 2.1 or 3.0a3,
but a cursory examination shows that 3.0a3 has an introspecter registry that 2.1
doesn't:

# In epydoc.docintrospecter:

def register_introspecter(applicability_test, introspecter, priority=10):
"""
Register an introspecter function.  Introspecter functions take
two arguments, a python value and a C{ValueDoc} object, and should
add information about the given value to the the C{ValueDoc}.
Usually, the first line of an inspecter function will specialize
it to a sublass of C{ValueDoc}, using L{ValueDoc.specialize_to()}:

>>> def typical_introspecter(value, value_doc):
... value_doc.specialize_to(SomeSubclassOfValueDoc)
... 

@param priority: The priority of this introspecter, which determines
the order in which introspecters are tried -- introspecters with lower
numbers are tried first.  The standard introspecters have priorities
    ranging from 20 to 30.  The default priority (10) will place new
introspecters before standard introspecters.
"""

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 12/7/10 2:26 PM, Vinay Sajip wrote:

This issue was brought to my notice today:

http://bugs.python.org/issue10626

and reference was made in the comments to possible obstacles facing stdlib
maintainers who might wish to use logging in the stdlib and in its unit tests.


From my perspective and as mentioned in the logging documentation, library code

which uses logging should add a NullHandler instance to any top-level logger,
which will avoid any "No handlers could be found for logger XXX" message if no
logging handlers have been set up.


I've done that before in my own library code, then quickly realized that it was 
a bad idea. Adding a NullHandler silently prevents logging.basicConfig() from 
working. Embedding that kind of restriction into library code (usually by a mere 
import!) made users unhappy and added an additional burden on the library 
developers who had to make sure to do that everywhere they used logging.


If I had my druthers, I would simply remove the "No handlers could be found for 
logger XXX" message. If we always wrote entire applications from the ground up, 
it makes a lot of sense. The person that writes the code that issues logs is the 
same person that writes the code to configure logging for reading. If you add 
logging in one place, you almost certainly want to use it somewhere else and not 
setting up logging is probably an error. But when you want to write reusable 
libraries, those roles become separated.


As a library author, I would dearly love to just add logging liberally without 
placing any additional burden to the users of my library. If my users wants to 
read those logs, he will configure logging. If he doesn't, he won't. With the 
current behavior, I can't do that. If I add logging, he has to add code just to 
silence a message that is meaningless to him (after I get the support emails 
asking if things are broken and explain how to silence it). If I add a 
NullHandler, I remove the ability for him to use logging.basicConfig(), the 
easiest and most straightforward way for him to add logging to his application.


This is only my personal anecdotal experience, but the current behavior has 
always wasted my time and never saved any of my time.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 12/7/10 4:59 PM, Robert Kern wrote:

On 12/7/10 2:26 PM, Vinay Sajip wrote:

This issue was brought to my notice today:

http://bugs.python.org/issue10626

and reference was made in the comments to possible obstacles facing stdlib
maintainers who might wish to use logging in the stdlib and in its unit tests.


From my perspective and as mentioned in the logging documentation, library code

which uses logging should add a NullHandler instance to any top-level logger,
which will avoid any "No handlers could be found for logger XXX" message if no
logging handlers have been set up.


I've done that before in my own library code, then quickly realized that it was
a bad idea. Adding a NullHandler silently prevents logging.basicConfig() from
working.


Only on the root handler. Forget this bit of my argument (and the statements 
that directly follow from it). The rest of my argument should stand on its own, 
though.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 2010-12-07 17:58 , Vinay Sajip wrote:

Robert Kern  gmail.com>  writes:



If I had my druthers, I would simply remove the "No handlers could be
found for logger XXX" message. If we always wrote entire applications
from the ground up, it makes a lot of sense. The person that writes the
code that issues logs is the same person that writes the code to configure
logging for reading. If you add logging in one place, you almost certainly
want to use it somewhere else and not setting up logging is probably an
error. But when you want to write reusable  libraries, those roles become
separated.


Exactly - we do use third-party libraries. When logging was added there was some
debate about whether this one-off message should be printed, and in balance it
was thought better to print this, not least because people would be unfamiliar
with logging (as it was then new) and so be not unlikely to misconfigure it. No
indication of this at all would be double-plus-ungood.


I really don't understand how this view can be consistent with the practice of 
adding NullHandler to loggers. If this message is so important to prevent 
misconfiguration, then why should a library author decide to silence it for his 
users?


I think that the chances of a misconfiguration that the warning would catch. are 
small in practice. I don't have any solid survey data on how people configure 
their loggers, but I strongly suspect that almost all configurations include a 
catch-all root logger and that most of those *only* consist of that root logger.



As a library author, I would dearly love to just add logging liberally
without placing any additional burden to the users of my library.
If my users wants to read those logs, he will configure logging. If he
doesn't, he won't. With the current behavior, I can't do that. If I add
logging, he has to add code just to silence a message that is meaningless
to him (after I get the support emails asking if things are broken and
explain how to silence it). If I add a NullHandler, I remove the ability
for him to use logging.basicConfig(), the easiest and most straightforward
way for him to add logging to his application.


You don't remove the ability for him to use basicConfig() - that's another
mistake in your post (you already corrected the other mistake in your
self-response). See my example with mylib.py/myapp.py in another post on this
thread, in response to Antoine.


Same mistake. I intended the correction to apply to all such statements in my 
post.


This is only my personal anecdotal experience, but the current behavior has
always wasted my time and never saved any of my time.


That's because I think you're doing it wrong, or else I've misunderstood your
use case.


I will try to state it more plainly: the message has always been a false 
positive in my experience. It has never helped me fix a real problem and has 
otherwise wasted my time.



NullHandler is the way to go, and doesn't preclude the use of
basicConfig() UNLESS you choose to add the NullHandler to the root logger (not
what you're supposed to do - you're supposed to add it to the top-level logger
*of your library*, not the top-level logger of the logging hierarchy.)

See the mylib/myapp example and tell me which it is - your mistake or mine?


I think that boilerplate should be minimized. If using getLogger() should almost 
always be followed by adding a NullHandler, then it should be the default 
behavior. The easiest way to achieve this effect is to simply not issue the 
warning message.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 12/8/10 2:51 AM, Vinay Sajip wrote:

Robert Kern  gmail.com>  writes:



I really don't understand how this view can be consistent with the
practice of adding NullHandler to loggers. If this message is so important
to prevent misconfiguration, then why should a library author decide to
silence it for his users?


Because the application developer knows more about the end-user audience for
their application, they are better placed to know how logging should work for
their application. It's not an error for a particular application developer to
decide that nothing should be produced by logging for a particular application;
they (particularly when casual users) would be confused by the misconfiguration
message due to logging by a library they're using.

The library author's users are the application developers who use the library,
not the end users who use their applications. Sometimes they're the same people,
I know, but I just think of them as wearing different hats :-)


I'm sorry, but it's not at all clear that you have understood my point. There is 
no way for me to parse your words as a sensible reply to what I said.


Let's say I write a library called Foo. I want to add logging to my functions. 
You want to write an application called Bar that uses Foo and you want to 
configure logging for your application (at the very least to provide a default 
if not production). The warning is supposed to help you not make mistakes when 
configuring logging in your application. If I, library author, attach 
NullHandlers to all of Foo's loggers, then you will not get that warning if you 
forget to add handlers the Foo loggers. My adding the NullHandler to Foo 
prevented that warning that you consider to be so important.


I don't think the warning helps much, if at all.


[...] I strongly suspect that almost all configurations include a
catch-all root logger and that most of those *only* consist of that
root logger.


That doesn't seem right: your comment might be conflating loggers with handlers.
The common pattern would be (or should be) to name loggers according to __name__
in the modules which use logging, but only configure *handlers* for the root
logger. That way, logging messages indicate their origin (because of the
__name__ convention) but you only need to add handlers at the root logger to
capture all the logging information.


Yes. That's what I meant.


I think that boilerplate should be minimized. If using getLogger() should
almost always be followed by adding a NullHandler, then it should be the
default behavior. The easiest way to achieve this effect is to simply not
issue the warning message.


getLogger() should NOT "almost always be followed by adding a NullHandler". For
example, in Django, only the logger named "django" would have that handler
added; no other Django logger (e.g. "django.db.models") would need to have that
handler added.


In a large package (particularly a namespace package), I can't guarantee that 
any particular module will get imported. I will want to be able to import just 
foo.bar.baz without needing to worry about whether foo.setup_logging got 
imported and ran the logging configuration as a side-effect. I want to be able 
to loosen the coupling between modules across my package, not add more coupling.


But in any case, while adding a NullHandler to just the package's root logger 
helps you to avoid needing a NullHandler on every logger, the effect is the 
same. Almost all loggers effectively terminate in a NullHandler either directly 
or through a chain of parent loggers. So why not just make it the default?


Personally, I would back the proposals being made elsewhere in this thread, that 
in the absence of configuration, warnings and errors should be printed to stderr 
no matter where they come from. This gives useful behavior out-of-the-box 
without configuration but remains completely configurable. Library errors don't 
pass silently, but logging allows people to silence them explicitly. It 
separates the concerns of library authors (who should never touch logging 
configuration and shouldn't be required to think about it) from those of the 
application authors and application users.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

If we're all converging on adding a "handler of last resort" and dropping the 
warning message, we can just let this branch of the thread drop. But if you want 
to continue, I already had most of the following already written. I hope it 
clears some things up more than it muddies them further. :-)


On 12/8/10 11:43 AM, Vinay Sajip wrote:

Robert Kern  gmail.com>  writes:



I don't think the warning helps much, if at all.


Perhaps not. I'm not so hung up on the importance of the message now, but it
certainly *was* important when logging was first introduced, and users couldn't
expect to get the same level of help on comp.lang.python that they now can.
Today there are quite a few people who can help people with finger-trouble
logging issues.


I am not commenting on the reasonableness of the behavior when it was 
introduced, just what I think is the best behavior now.



Consider the scenarios when not having the current behaviour would bite you:

1. You're new to Python/new to logging. You write some code, perhaps across
several modules, which code makes logging calls, but you omit to configure any
handlers, whether through forgetting or not understanding what needs to be done.
Without the message, you've no idea why no logging messages appear, no matter
how much you fiddle with levels.
2. You write some code and decide you don't want to use logging, but use some
third party libraries which do. You don't care about that, so it's annoying to
have "no handlers could be found for logger XXX" messages printed to console.
You berate the library developer for their lack of consideration.

Perhaps you don't find yourself in these situations, but surely you sympathize
with people who do? How would you propose to address both those scenarios?


I am quite familiar with the latter in the third-party library author role. 
However, the messages I get aren't about lack of consideration but "What is 
wrong with my code? Are things broken?" They think the warning comes from my code.


As for the former, I'm not sure how that would cause much confusion. If I'm 
interested in getting the logged information, wouldn't I first find out how to 
configure logging? It's just about the very first thing you see reading the 
docs. It's one of the fundamental operations using the logging package. I can't 
imagine that this is a common failure mode. I think that the documentation 
ameliorates it better than the warning message.



In a large package (particularly a namespace package), I can't guarantee
that any particular module will get imported. I will want to be able to
import just foo.bar.baz without needing to worry about whether
foo.setup_logging got imported and ran the logging configuration as a
side-effect. I want to be able to loosen the coupling between modules
across my package, not add more coupling.


I'm not sure what coupling you're talking about - perhaps you can illustrate
with an example. If I develop a package "foo.bar" which is part of namespace
package "foo", and use loggers named __name__ in my code, and add a NullHandler
to logger "foo.bar", that's all I have to do. Likewise, if another person
develops "foo.baz" and they add a NullHandler to "foo.baz", then where's the
coupling between the two packages? They needn't even know about each other.


You were saying (in the part of the conversation that you snipped) that one 
would just add a NullHandler to the "django" logger to take care of messages 
sent to "django.db.models" and all of the other child loggers. *That* introduces 
a tighter coupling than I am comfortable with. All modules that use logging must 
make sure that the code that adds the NullHandler to the "django" logger gets 
executed. It's certainly reasonable to make that effort sometimes, but there are 
good reasons to avoid it, too. Just using logging should not force my hand.


If I do keep things decoupled, then I am adding a NullHandler to nearly every 
logger anyways, and I return to my original point that this is repetitive 
boilerplate and should be eliminated by appropriate defaults.



But in any case, while adding a NullHandler to just the package's root logger
helps you to avoid needing a NullHandler on every logger, the effect is the
same. Almost all loggers effectively terminate in a NullHandler either
directly or through a chain of parent loggers. So why not just make it the
default?


There's no "termination" when a NullHandler is encountered.


All I meant is that it eventually reaches a NullHandler so no warning message is 
issued. Whether it actually stops there or not is irrelevant to my point. My 
point is that it is rarely the case that you deliberately want to have a logger 
which will cause the “No handlers could be found&q

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 2010-12-12 18:42 , Glenn Linderman wrote:

On 12/12/2010 2:26 PM, Paul Moore wrote:

The thing*I*  hit very early was wanting to add a command lime option
to my script to set the logging level. I'd have liked to be able to
add --log=INFO/DEBUG/... but to do that I seem to need to write my own
mapping between level names and numbers. A simple example of how to
tie command line options to logging config would be a great addition
to the documentation.


Oh?

import * from logger # change some details to avoid this
basicConfig( level= eval( opt.loglevel )


level = getattr(logging, opt.logLevel)

or

level = logging._levelNames[opt.logLevel]

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using logging in the stdlib and its unit tests

[Robert Kern]

On 2010-12-12 21:30 , Nick Coghlan wrote:

On Mon, Dec 13, 2010 at 11:22 AM, Robert Kern  wrote:

level = getattr(logging, opt.logLevel)


While this is the approach I would recommend, it does have a couple of
downsides:

1. An upper() call is also needed to allow strings like "info" instead
of "INFO":
2. If an integer is available, it would be nice to return it
unmodified (or, if we ever get named values in the standard library,
convert it to that)
3. The asymmetry with "logging.getLevelName" grates a bit

So it would be far more obvious if there was a "logging.getLevel"
counterpart to "logging.getLevelName" that looked something like:

def getLevel(level):
   try:
 return operator.index(level) # Integers and equivalents
   except TypeError:
 pass
   try:
 key = level.upper()
   except Exception as ex:
 raise TypeError("Log level must be an integer or string") from ex
   return globals()[key]


I don't think that the implementation should use globals(). I wouldn't want 
logging.getLevel('basic_format') to work. Instead, it should look it up in the 
known set of levels.



level = logging._levelNames[opt.logLevel]


That doesn't work (_levelNames maps from integers to strings, we want
the mapping from strings to integers and it is only the module globals
that provides that).


At least in Python 2.6, it maps both ways. But then again, it is an 
_implementation _detail that should not be relied upon in your programs. I would 
suggest that there should be two dictionaries as part of the documented API, one 
mapping numbers to names and one mapping names to numbers. Or functions/methods 
returning said dictionaries. Having the entire mappings at hand is more useful 
than having functions that do the translation. They would allow you to 
auto-generate UIs (e.g. the help text for a --log-level argument or a radio box 
widget in a GUI). Having separate mappings makes them easier to work with than 
the 2.6-style _levelNames mapping.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] public visibility of python-dev decisions "before it's too late" (was: PyCObject_AsVoidPtr removed from python 3.2 - is this documented?)

[Robert Kern]

On 3/14/11 5:30 PM, Lennart Regebro wrote:


Many projects, not only the Zope Toolkit needs to support a lot of
versions. The Zope component architecture currently supports 2.4, 2.5
and 2.6 and is expected to work on 2.7. I don't know if 2.4 or 2.5 can
be dropped, but it definitely will be *years* until we can drop
support for 2.6.  But if I move the PyCObject API to the PyCapsule
API, the zope packages will **only work on Python 2.7 and 3.2**. This
is obviously not an option. If I do *not* switch, I can't support
Python 3.2. That's bad.


For what it's worth, numpy simultaneously supports Python 2.5-2.7 and 3.1-3.2. 
It uses PyCObject or PyCapsule APIs as appropriate. We do this from the same 
codebase. We had to add another layer of indirection, but nothing too bad. You 
can steal our code here:


https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/npy_3kcompat.h#L299

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 4/27/11 12:44 PM, Terry Reedy wrote:

On 4/27/2011 10:53 AM, Guido van Rossum wrote:



Maybe we should just call off the odd NaN comparison behavior?


Eiffel seems to have survived, though I do not know if it used for numerical
work. I wonder how much code would break and what the scipy folks would think.


I suspect most of us would oppose changing it on general backwards-compatibility 
grounds rather than actually *liking* the current behavior. If the behavior 
changed with Python floats, we'd have to mull over whether we try to match that 
behavior with our scalar types (one of which subclasses from float) and our 
arrays. We would be either incompatible with Python or C, and we'd probably end 
up choosing Python to diverge from. It would make a mess, honestly. We already 
have to explain why equality is funky for arrays (arr1 == arr2 is a rich 
comparison that gives an array, not a bool, so we can't do containment tests for 
lists of arrays), so NaN is pretty easy to explain afterward.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 2011-04-27 22:16 , Guido van Rossum wrote:

On Wed, Apr 27, 2011 at 11:48 AM, Robert Kern  wrote:

On 4/27/11 12:44 PM, Terry Reedy wrote:


On 4/27/2011 10:53 AM, Guido van Rossum wrote:



Maybe we should just call off the odd NaN comparison behavior?


Eiffel seems to have survived, though I do not know if it used for
numerical
work. I wonder how much code would break and what the scipy folks would
think.


I suspect most of us would oppose changing it on general
backwards-compatibility grounds rather than actually *liking* the current
behavior. If the behavior changed with Python floats, we'd have to mull over
whether we try to match that behavior with our scalar types (one of which
subclasses from float) and our arrays. We would be either incompatible with
Python or C, and we'd probably end up choosing Python to diverge from. It
would make a mess, honestly. We already have to explain why equality is
funky for arrays (arr1 == arr2 is a rich comparison that gives an array, not
a bool, so we can't do containment tests for lists of arrays), so NaN is
pretty easy to explain afterward.


So does NumPy also follow Python's behavior about ignoring the NaN
special-casing when doing array ops?


By "ignoring the NaN special-casing", do you mean that identity is checked 
first? When we use dtype=object arrays (arrays that contain Python objects as 
their data), yes:


[~]
|1> nan = float('nan')

[~]
|2> import numpy as np

[~]
|3> a = np.array([1, 2, nan], dtype=object)

[~]
|4> nan in a
True

[~]
|5> float('nan') in a
False


Just like lists:

[~]
|6> nan in [1, 2, nan]
True

[~]
|7> float('nan') in [1, 2, nan]
False


Actually, we go a little further by using PyObject_RichCompareBool() rather than 
PyObject_RichCompare() to implement the array-wise comparisons in addition to 
containment:


[~]
|8> a == nan
array([False, False,  True], dtype=bool)

[~]
|9> [x == nan for x in [1, 2, nan]]
[False, False, False]


But for dtype=float arrays (which contain C doubles, not Python objects) we use 
C semantics. Literally, we use whatever C's == operator gives us for the two 
double values. Since there is no concept of identity for this case, there is no 
cognate behavior of Python to match.


[~]
|10> b = np.array([1.0, 2.0, nan], dtype=float)

[~]
|11> b == nan
array([False, False, False], dtype=bool)

[~]
|12> nan in b
False

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 2011-04-27 23:01 , Guido van Rossum wrote:

On Wed, Apr 27, 2011 at 8:42 PM, Robert Kern  wrote:



But for dtype=float arrays (which contain C doubles, not Python objects) we
use C semantics. Literally, we use whatever C's == operator gives us for the
two double values. Since there is no concept of identity for this case,
there is no cognate behavior of Python to match.

[~]
|10>  b = np.array([1.0, 2.0, nan], dtype=float)

[~]
|11>  b == nan
array([False, False, False], dtype=bool)

[~]
|12>  nan in b
False


And I wouldn't want to change that. It sounds like NumPy wouldn't be
much affected if we were to change this (which I'm not saying we
would).


Well, I didn't say that. If Python changed its behavior for (float('nan') == 
float('nan')), we'd have to seriously consider some changes. We do like to keep 
*some* amount of correspondence with Python semantics. In particular, we like 
our scalar types that match Python types to work as close to the Python type as 
possible. We have the np.float64 type, which represents a C double scalar and 
corresponds to a Python float. It is used when a single item is indexed out of a 
float64 array. We even subclass from the Python float type to help working with 
libraries that may not know about numpy:


[~]
|5> import numpy as np

[~]
|6> nan = np.array([1.0, 2.0, float('nan')])[2]

[~]
|7> nan == nan
False

[~]
|8> type(nan)
numpy.float64

[~]
|9> type(nan).mro()
[numpy.float64,
 numpy.floating,
 numpy.inexact,
 numpy.number,
 numpy.generic,
 float,
 object]


If the Python float type changes behavior, we'd have to consider whether to keep 
that for np.float64 or change it to match the usual C semantics used elsewhere. 
So there *would* be a dilemma. Not necessarily the most nerve-wracking one, but 
a dilemma nonetheless.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 2011-04-27 23:24 , Guido van Rossum wrote:

On Wed, Apr 27, 2011 at 9:15 PM, Alexander Belopolsky
  wrote:

On Wed, Apr 27, 2011 at 2:48 PM, Robert Kern  wrote:
..

I suspect most of us would oppose changing it on general
backwards-compatibility grounds rather than actually *liking* the current
behavior. If the behavior changed with Python floats, we'd have to mull over
whether we try to match that behavior with our scalar types (one of which
subclasses from float) and our arrays. We would be either incompatible with
Python or C, and we'd probably end up choosing Python to diverge from. It
would make a mess, honestly. We already have to explain why equality is
funky for arrays (arr1 == arr2 is a rich comparison that gives an array, not
a bool, so we can't do containment tests for lists of arrays), so NaN is
pretty easy to explain afterward.


Most NumPy applications are actually not exposed to NaN problems
because it is recommended that NaNs be avoided in computations and
when missing or undefined values are necessary, the recommended
solution is to use ma.array or masked array which is a drop-in
replacement for numpy array type and carries a boolean "mask" value
with every element.  This allows to have undefined elements is arrays
of any type: float, integer or even boolean.  Masked values propagate
through all computations including comparisons.


So do new masks get created when the outcome of an elementwise
operation is a NaN?


No.


Because that's the only reason why one should have
NaNs in one's data in the first place -- not to indicate missing
values!


Yes. I'm not sure that Alexander was being entirely clear. Masked arrays are 
intended to solve just the missing data problem and not the occurrence of NaNs 
from computations. There is still a persistent part of the community that really 
does like to use NaNs for missing data, though. I don't think that's entirely 
relevant to this discussion[1].


I wouldn't say that numpy applications aren't exposed to NaN problems. They are 
just as exposed to computational NaNs as you would expect any application that 
does that many flops to be.


[1] Okay, that's a lie. I'm sure that persistent minority would *love* to have 
NaN == NaN, because that would make their (ab)use of NaNs easier to work with.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 4/27/11 11:54 PM, Guido van Rossum wrote:

On Wed, Apr 27, 2011 at 9:25 PM, Robert Kern  wrote:

On 2011-04-27 23:01 , Guido van Rossum wrote:

And I wouldn't want to change that. It sounds like NumPy wouldn't be
much affected if we were to change this (which I'm not saying we
would).


Well, I didn't say that. If Python changed its behavior for (float('nan') ==
float('nan')), we'd have to seriously consider some changes.


Ah, but I'm not proposing anything of the sort! float('nan') returns a
new object each time and two NaNs that are not the same *object* will
still follow the IEEE std. It's just when comparing a NaN-valued
*object* to *itself* (i.e. the *same* object) that I would consider
following the lead of Python's collections.


Ah, I see!


We do like to
keep *some* amount of correspondence with Python semantics. In particular,
we like our scalar types that match Python types to work as close to the
Python type as possible. We have the np.float64 type, which represents a C
double scalar and corresponds to a Python float. It is used when a single
item is indexed out of a float64 array. We even subclass from the Python
float type to help working with libraries that may not know about numpy:

[~]
|5>  import numpy as np

[~]
|6>  nan = np.array([1.0, 2.0, float('nan')])[2]

[~]
|7>  nan == nan
False


Yeah, this is where things might change, because it is the same
*object* left and right.


[~]
|8>  type(nan)
numpy.float64

[~]
|9>  type(nan).mro()
[numpy.float64,
  numpy.floating,
  numpy.inexact,
  numpy.number,
  numpy.generic,
  float,
  object]


If the Python float type changes behavior, we'd have to consider whether to
keep that for np.float64 or change it to match the usual C semantics used
elsewhere. So there *would* be a dilemma. Not necessarily the most
nerve-wracking one, but a dilemma nonetheless.


Given what I just said, would it still be a dilemma? Maybe a smaller one?


Smaller, certainly. But now it's a trilemma. :-)

1. Have just np.float64 and np.complex128 scalars follow the Python float 
semantics since they subclass Python float and complex, respectively.

2. Have all np.float* and np.complex* scalars follow the Python float semantics.
3. Keep the current IEEE-754 semantics for all float scalar types.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 4/28/11 12:37 AM, Alexander Belopolsky wrote:

On Thu, Apr 28, 2011 at 12:33 AM, Robert Kern  wrote:

On 2011-04-27 23:24 , Guido van Rossum wrote:

..

So do new masks get created when the outcome of an elementwise
operation is a NaN?


No.


Yes.


from MA import array
print array([0])/array([0])

[-- ]

(I don't have numpy on this laptop, so the example is using Numeric,
but I hope you guys did not change that while I was not looking:-)


This behavior is not what you think it is. Rather, some binary operations have 
been augmented with a domain of validity, and the results will be masked out 
when the domain is violated. Division is one of them, and division by zero will 
cause the result to be masked. You can produce NaNs in other ways that will not 
be masked in both numpy and old Numeric:


[~]
|4> minf = np.ma.array([1e300]) * np.ma.array([1e300])
Warning: overflow encountered in multiply

[~]
|5> minf
masked_array(data = [ inf],
 mask = False,
   fill_value = 1e+20)


[~]
|6> minf - minf
masked_array(data = [ nan],
 mask = False,
   fill_value = 1e+20)

[~]
|14> import MA

[~]
|15> minf = MA.array([1e300]) * MA.array([1e300])

[~]
|16> minf
array([  inf,])

[~]
|17> (minf - minf)[0]
nan

[~]
|25> (minf - minf)._mask is None
True


Numeric has a bug where it cannot print arrays with NaNs, so I just grabbed the 
element out instead of showing it. But I guarantee you that it is not masked.


Masked arrays are not a way to avoid NaNs arising from computations. NaN 
handling is an important part of computing with numpy.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] the role of assert in the standard library ?

[Robert Kern]

On 4/28/11 3:27 PM, Holger Krekel wrote:

On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum  wrote:

On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé  wrote:

In my opinion assert should be avoided completely anywhere else than
in the tests. If this is a wrong statement, please let me know why :)


I would turn that around. The assert statement should not be used in
unit tests; unit tests should use self.assertXyzzy() always.


FWIW this is only true for the unittest module/pkg policy for writing and
organising tests. There are other popular test frameworks like nose and pytest
which promote using plain asserts within writing unit tests and also allow to
write tests in functions.  And judging from my tutorials and others places many
people appreciate the ease of using asserts as compared to learning tons
of new methods. YMMV.

Holger


regular code, assert should be about detecting buggy code. It should
not be used to test for error conditions in input data. (Both these
can be summarized as "if you still want the test to happen with -O,
don't use assert.)


Regardless of whether those frameworks encourage it, it's still the wrong thing 
to do for the reason that Guido states. Some bugs only show up under -O, so you 
ought to be running your test suite under -O, too.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 4/28/11 11:55 AM, Guido van Rossum wrote:


On Thu, Apr 28, 2011 at 8:52 AM, Robert Kern  wrote:

Smaller, certainly. But now it's a trilemma. :-)

1. Have just np.float64 and np.complex128 scalars follow the Python float
semantics since they subclass Python float and complex, respectively.
2. Have all np.float* and np.complex* scalars follow the Python float
semantics.
3. Keep the current IEEE-754 semantics for all float scalar types.


*If* my proposal gets accepted, there will be a blanket rule that no
matter how exotic an type's __eq__ is defined, self.__eq__(self)
(i.e., __eq__ called with the same *object* argument) must return True
if the type's __eq__ is to be considered well-behaved; and Python
containers may assume (for the purpose of optimizing their own
comparison operations) that their elements have a well-behaved __eq__.


*If* so, then we would then just have to decide between #2 and #3.

With respect to this proposal, how does that interact with types that do not 
return bools for rich comparisons? For example, numpy arrays return bool arrays 
for comparisons. SQLAlchemy expressions return other SQLAlchemy expressions, 
etc. I realize that by being "not well-behaved" in this respect, we give up our 
rights to be proper elements in sortable, containment-checking containers. But 
in this and your followup message, you seem to be making a stronger statement 
that self.__eq__(self) not returning anything but True would be a bug in all 
contexts.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PyObject_RichCompareBool identity shortcut

[Robert Kern]

On 4/28/11 6:13 PM, Guido van Rossum wrote:

On Thu, Apr 28, 2011 at 4:04 PM, Nick Coghlan  wrote:

On Fri, Apr 29, 2011 at 8:55 AM, Guido van Rossum  wrote:

Sorry, we'll have to make an exception for those of course. This will
somewhat complicate the interpretation of well-behaved, because those
are *not* well-behaved as far as containers go (both dict key lookup
and list membership are affected) but it is not a bug -- however it is
a bug to put these in containers and then use container comparisons on
the container.


That's a point in favour of the status quo, though - with the burden
of enforcing reflexivity placed on the containers, types are free to
make use of rich comparisons to return more than just simple
True/False results.


Possibly. Though for types that *do* return True/False, NaN's behavior
can still be infuriating.


I hadn't really thought about it that way before this discussion - it
is the identity checking behaviour of the builtin containers that lets
us sensibly handle cases like sets of NumPy arrays.


But do they? For non-empty arrays, __eq__ will always return something
that is considered true,


Actually, numpy.ndarray.__nonzero__() raises an exception. We've decided that 
there are no good conventions for deciding whether an array should be considered 
True or False that won't mislead people. It's quite astonishing how many people 
will just test "if x == y:" or "if x != y:" for a single set of inputs and 
"confirm" their guess as to the general rule from that.


But your point stands, numpy arrays cannot be members of sets or keys of dicts 
or orderable/"in-able" elements of lists.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not-a-Number

[Robert Kern]

On 4/28/11 8:44 PM, Steven D'Aprano wrote:

Greg Ewing wrote:

Taking a step back from all this, why does Python allow
NaNs to arise from computations *at all*?


The real question should be, why does Python treat all NANs as signalling NANs
instead of quiet NANs? I don't believe this helps anyone.


Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not-a-Number

[Robert Kern]

On 4/29/11 1:35 AM, Nick Coghlan wrote:

On Fri, Apr 29, 2011 at 3:28 PM, Steven D'Aprano  wrote:

Robert Kern wrote:

Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs.


Sorry, did I get that backwards? I thought it was signalling NANs that cause
a signal (in Python terms, an exception)?

If I do x = 0.0/0 I get an exception instead of a NAN. Hence a signalling
NAN.


Aside from the divide-by-zero case, we treat NaNs as quiet NaNs.


And in fact, 0.0/0.0 is covered by the more general rule that x/0.0 raises 
ZeroDivisionError, not a rule that converts IEEE-754 INVALID exceptions into 
Python exceptions. Other operations that produce a NaN and issue an IEEE-754 
INVALID signal do not raise a Python exception.


But that's not the difference between signalling NaNs and quiet NaNs. A 
signalling NaN is one that when it is used as an *input* to an operation, it 
issues an INVALID signal, not whether a signal is issued when it is the *output* 
of an operation.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not-a-Number

[Robert Kern]

On Fri, Apr 29, 2011 at 11:35, Alexander Belopolsky
 wrote:
> On Fri, Apr 29, 2011 at 11:31 AM, Robert Kern  wrote:
> ..
>> And in fact, 0.0/0.0 is covered by the more general rule that x/0.0 raises
>> ZeroDivisionError, not a rule that converts IEEE-754 INVALID exceptions into
>> Python exceptions.
>
> It is unfortunate that official text of IEEE-754 is not freely
> available and as a result a lot of discussion in this thread is based
> on imperfect information.
>
> I find Kahan's "Lecture Notes on the Status of IEEE Standard 754 for
> Binary Floating-Point Arithmetic" [1] a reasonable reference in the
> absence of the official text.   According to Kahan's notes, INVALID
> operation is defined as follows:
>
> """
> Exception: INVALID operation.
>
> Signaled by the raising of the INVALID flag whenever an operation's
> operands lie outside its domain, this exception's default, delivered
> only because any other real or infinite value would most likely cause
> worse confusion, is NaN , which means “ Not a Number.” IEEE 754
> specifies that seven invalid arithmetic operations shall deliver a NaN
> unless they are trapped:
>
>    real √(Negative) , 0*∞ , 0.0/0.0 , ∞/∞,
>    REMAINDER(Anything, 0.0) , REMAINDER( ∞, Anything) ,
>    ∞ - ∞ when signs agree (but ∞ + ∞ = ∞ when signs agree).
>
> Conversion from floating-point to other formats can be INVALID too, if
> their limits are violated, even if no NaN can be delivered.
> """
>
> In contrast, Kahan describes DIVIDE by ZERO exception as "a misnomer
> perpetrated for historical reasons. A better name for this exception
> is 'Infinite result computed Exactly from Finite operands.'"

Nonetheless, the reason that *Python* raises a ZeroDivisionError is
because it checks that the divisor is 0.0, not because 0.0/0.0 would
issue an INVALID signal. I didn't mean that 0.0/0.0 is a "Division by
Zero" error as defined in IEEE-754. This is another area where Python
ignores the INVALID signal and does its own thing.

>> Other operations that produce a NaN and issue an IEEE-754
>> INVALID signal do not raise a Python exception.
>
> Some do:
>
>>>> math.sqrt(-1)
> Traceback (most recent call last):
>  File "", line 1, in 
> ValueError: math domain error

Right. Elsewhere I gave a more exhaustive list including this one. The
other is int(nan), though that becomes a Python exception for a more
fundamental reason (there is no integer value that can represent it)
than that the IEEE-754 standard specifies that the operation should
signal INVALID. Arithmetic operations on signalling NaNs don't raise
an exception either.

These are the minority *exceptions* to the majority of cases where
operations on Python floats that would issue an INVALID signal do not
raise Python exceptions. If you want to lump all of the inf-related
cases together, that's fine, but arithmetic operations on signalling
NaNs and comparisons with NaNs form two more groups of INVALID
operations that do not raise Python exceptions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docs, reloaded

[Robert Kern]

Neal Becker wrote:

> There is an effort as part of numpy to come up with a new system using
> docstrings.  It seems to me it would be unfortunate if these two efforts
> were not coordinated.

I don't think so. The issue with numpy is getting our act together and making
parseable docstrings for auto-generated API documentation using existing tools
or slightly modified versions thereof. No one is actually contemplating building
a new tool. AFAICT, Georg's (excellent) work doesn't address that use. I don't
think there is anything to coordinate, here. Provided that Georg's system
doesn't place too many restrictions on the reST it handles, we could use the
available reST math options if we wanted to use Georg's system.

I'd much rather see Georg spend his time working on the docs for the Python
language and the feature set it requires. If the numpy project needs to extend
that feature set, we'll provide the manpower ourselves.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Product function patch [issue 1093]

[Robert Kern]

Guido van Rossum wrote:
> On 9/4/07, Steven H. Rogers <[EMAIL PROTECTED]> wrote:
>> Guido van Rossum wrote:
>>> I still don't see why the standard library needs to be weighed down
>>> with a competitor to numpy. Including a subset of numpy was considered
>>> in the past, but it's hard to decide on the right subset. In the end
>>> it was decided that numpy is too big to become a standard library.
>>> Given all the gyrations it has gone through I definitely believe this
>>> was the right decision.
>> A competitor to NumPy would be counter-productive, but including a core
>> subset in the standard library that NumPy could be built upon would add
>> valuable functionality to Python out of the box.  It was probably the
>> best decision to not include NumPy when it was previously considered,
>> but I think it should be reconsidered for Python 3.x.  While defining
>> the right subset to include has it's difficulties, I believe it can be
>> done.  What would be a reasonable target size for inclusion in the
>> standard library?
> 
> What makes 3.0 so special? Additions to the stdlib can be considered
> at any feature release.

The 3.x compatibility break (however alleviated by 2to3) makes a nice clean
cutoff. The numpy that works on Pythons 3.x would essentially be a port from the
current numpy. Consequently, we could modify the numpy for Pythons 3.x to always
rely on the stdlib API to build on top of. We couldn't do that for the version
targeted to Pythons 2.x because we could only rely on its presence for 2.6+. I
don't mind maintaining two versions of numpy, one for Python 2.x and one for
3.x, but I don't care to maintain three.

I invite Greg and Steven and whoever else is interested to discuss ideas for the
PEP on numpy-discussion. I'm skeptical, seeing what currently has been
suggested, but some more details could easily allay that.

  http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Product function patch [issue 1093]

[Robert Kern]

Guido van Rossum wrote:
> On 9/4/07, Robert Kern <[EMAIL PROTECTED]> wrote:
>> The 3.x compatibility break (however alleviated by 2to3) makes a nice clean
>> cutoff. The numpy that works on Pythons 3.x would essentially be a port from 
>> the
>> current numpy. Consequently, we could modify the numpy for Pythons 3.x to 
>> always
>> rely on the stdlib API to build on top of. We couldn't do that for the 
>> version
>> targeted to Pythons 2.x because we could only rely on its presence for 2.6+. 
>> I
>> don't mind maintaining two versions of numpy, one for Python 2.x and one for
>> 3.x, but I don't care to maintain three.
> 
> I just had a discussion with Glyph "Twisted" Lefkowitz about this. He
> warns that if every project using Python uses 3.0's incompatibility as
> an excuse to make their own library/package/project incompatible as
> well, we will end up with total pandemonium (my paraphrase). I think
> he has a good point -- we shouldn't be injecting any more instability
> into the world than absolutely necessary.

I agree. I didn't mean to imply that the 3.x version of numpy would be
incompatible to users of it, just that the codebase that implements it will be
different, whether it is automatically or manually translated.

Of course, if the API is introduced in 3.(x>0), we end up with the same problem
I wanted to avoid. Ah well. See you on the flip side of the PEP.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C Decimal - is there any interest?

[Robert Kern]

Mateusz Rukowicz wrote:
> Grzegorz Makarewicz wrote:
> 
>> Python in high aritmetic - nice, but in standard ? - using computation
>> effective representation as in this "demand" should be put as *must*
>> have ? in standard python ? - decimal with it's speed is sufficient - if
>> U need more speed use more spohisticated data structures - numpy/scipy
>> ... - I dont like those additions to python standard.
>>
> First of all C Decimal isn't meant to be substitute to numpy in any way.
> Second of all, I don't think numpy/scipy is an alternative to Decimal -
> it is more complicated, and I am not even sure if it supports Decimal
> arithmetic (which is crucial for financial application ie.).
> 
> [I couldn't find on google whether scipy supports decimal arithmetic, so
> I assumed it doesn't because of name '*scientific* tools for python'.
> However, if it supports - ignore following text block]

You are right. We have no tools in numpy or scipy for decimal arithmetic. The
focus of numpy is arrays. This decimal module is complementary to numpy and vice
versa.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Error in PEP3118?

[Robert Kern]

Thomas Heller wrote:

> Here is another typo (?) in the pep; I think it should be changed:
> 
> Index: pep-3118.txt
> ===
> --- pep-3118.txt  (revision 60037)
> +++ pep-3118.txt  (working copy)
> @@ -338,7 +338,7 @@
>  
>  ``len``
>  the total bytes of memory the object uses.  This should be the
> -same as the product of the shape array multiplied by the number of
> +same as the length of the shape array multiplied by the number of
>  bytes per item of memory.
>  
>  ``readonly``

While the original could be reworded ("product of the elements of the shape 
array"), the amendment is incorrect. For a shape array like {4*5*6}, the number 
of bytes is (4*5*6)*bytes_per_item, not 3*bytes_per_item.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory"

[Robert Kern]

[EMAIL PROTECTED] wrote:
> >> Note the "rm: conftest.dSYM: is a directory".  This occurred a few
> >> times during the configure process.  Didn't cause it to conk out, but
> >> is annoying.
> 
> Brett> I am assuming this is on your OS X machine, Skip?
> 
> Yes, sorry.  I forgot to mention that.  As long as someone else sees it and
> feels fine to ignore it, I'm fine with it.  I suspect it's a configure
> problem anyway, not a Python problem.

I've seen it with just about every ./configure script on OS X 10.5. It's not 
specific to Python, and I haven't seen it cause any actual problems.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unittest PEP do's and don'ts (BDFL pronouncement)

[Robert Kern]

Guido van Rossum wrote:

On Wed, Jul 16, 2008 at 2:03 PM, Raymond Hettinger <[EMAIL PROTECTED]> wrote:

From: "Michael Foord" <[EMAIL PROTECTED]>

assertIn / assertNotIn I use very regularly for collection membership

- self.assert_(func(x) in result_set)
+ self.assertIn(func(x), result_set)

Yawn.  The gain is zero.  Actually, it's negative because the second
doesn't read as nicely as the pure python expression.


I disagree. The reason why we have assertEquals(x, y) is that the
error message can show the values of x and y, whereas assert x == y
can't show those. Showing the values can be tremendously useful to
debugging the failure. (Doing an intelligent comparison, e.g. a string
or list "diff" instead of showing the two values, can be even more
useful, and I'd be in favor of that rather than adding new methods
like assertListsEqual.)

(Titus asks if the assert statement could be adjusted to do better
reporting. But that's not going to happen -- it would require a
tremendous amount of compiler support that would have to be
implemented in every Python implementation (last I counted there were
at least five). In addition, python -O removes all asserts from your
code -- that's why we use assertXxx functions in the first place.)


The assert statement itself does not have to be modified. nose and py.test are 
already capable of picking out the variables used in a failing assert expression 
and print them out. For example:



[~]$ cat test_nicefail.py
def test_nicefail():
x = 12
assert x == 10

[~]$ nosetests --detailed-errors test_nicefail.py
F
==
FAIL: test_nicefail.test_nicefail
--
Traceback (most recent call last):
  File 
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/nose-0.10.3-py2.5.egg/nose/case.py", 
line 182, in runTest

self.test(*self.arg)
  File "/Users/rkern/test_nicefail.py", line 3, in test_nicefail
assert x == 10
AssertionError:
12 = 12
>>  assert 12 == 10


--
Ran 1 test in 0.044s

FAILED (failures=1)


[~]$ py.test test_nicefail.py
== test process starts ===
executable: 
/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python 
 (2.5.1-final-0)
using py lib: 
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/py 



test_nicefail.py[1] F

__
___ entrypoint: test_nicefail 

def test_nicefail():
x = 12
E   assert x == 10
>   assert 12 == 10

[/Users/rkern/test_nicefail.py:3]
______
 tests finished: 1 failed in 0.09 seconds 

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Matrix product

[Robert Kern]

Terry Reedy wrote:


That said, I am curious what working scientists using Python think.


Well, we'll let you know more after SciPy '08, but I suspect the answer is that 
they just want one teensy little wafer-thin operator to do matrix multiplication 
on numpy arrays or their favorite matrix object. I don't think there are many 
scientists/engineers/whatnot who want to double the number of operators to learn 
or who care if the matmult operator works on lists of lists or anything else in 
the standard library.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pprint(iterator)

[Robert Kern]

On 2009-01-29 08:20, Aahz wrote:


The reason I'm chiming in is that I would welcome a PEP that created a
__pprint__ method as an alternative to special-casing.  I think that it
would be generically useful for user-created objects, plus once you've
added this feature other people can easily do some of the grunt work of
extending this through the Python core.  (Actually, unless someone
objects, I don't think a PEP is required, but it would be good for the
usual reasons that PEPs are written, to provide a central place
documenting the addition.)


I think it's worth looking at Armin Ronacher's pretty.py for a starting point.

  http://dev.pocoo.org/hg/sandbox/file/tip/pretty

I've been using it as my default displayhook under IPython for a few weeks now. 
It uses a combination of a function registry and a __pretty__ special method to 
find the right pretty printer.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pprint(iterator)

[Robert Kern]

On 2009-02-02 14:50, Nick Coghlan wrote:

Really drastic performance degradations would be on the radar as well -
slowing pprint() down by 10% is unlikely to bother anyone, but slowing
it down by 100% would be a bad thing (not that I think such a
degradation is likely, I'm just trying to give an impression of the
magnitude of change I'll be trying to measure).


Using the pretty module I referenced earlier, which basically uses 
simplegeneric's lookup algorithm:


In [11]: %timeit x=pprint.pformat(sys.modules)
10 loops, best of 3: 27.5 ms per loop

In [12]: %timeit x=pretty.pretty(sys.modules)
10 loops, best of 3: 39.9 ms per loop

In [13]: len(sys.modules)
Out[13]: 517


The comparison is somewhat dodgy, though. pretty is not so much a refactoring of 
pprint as a reimplementation, so the time differences may not be due to the 
dispatch.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] readd u'' literal support in 3.3?

[Robert Kern]

On 12/8/11 9:27 PM, "Martin v. Löwis" wrote:

[Glyph writes:]

Much of it is being in FORTRAN 77


Can you prove this? I trust that existing code is being maintained
in FORTRAN 77. For new code, I'm skeptical.


Personally, I've written more new code in FORTRAN 77 than in Fortran 90+. Even 
with all of the quirks in FORTRAN 77 compilers, it's still substantially easier 
to connect FORTRAN 77 code to C and Python than 90+. When they introduced some 
of the nicer language features, they left the precise details of memory 
structures of the new types undefined, so compilers chose different ways to 
implement them. Some of the very latest developments in modern Fortran have 
begun to standardize the FFI for these features (or at least let you write a 
standardized shim for them) and compilers are catching up.


For people writing new whole programs in Fortran, yes, they are probably mostly 
using 90+.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] requirements for moving __import__ over to importlib?

[Robert Kern]

On 2/9/12 10:15 PM, Antoine Pitrou wrote:

On Thu, 9 Feb 2012 17:00:04 -0500
PJ Eby  wrote:

On Thu, Feb 9, 2012 at 2:53 PM, Mike Meyer  wrote:


For those of you not watching -ideas, or ignoring the "Python TIOBE
-3%" discussion, this would seem to be relevant to any discussion of
reworking the import mechanism:

http://mail.scipy.org/pipermail/numpy-discussion/2012-January/059801.html

Interesting.  This gives me an idea for a way to cut stat calls per

sys.path entry per import by roughly 4x, at the cost of a one-time
directory read per sys.path entry.


Why do you even think this is a problem with "stat calls"?


All he said is that reading about that problem and its solution gave him an idea 
about dealing with stat call overhead. The cost of stat calls has demonstrated 
itself to be a significant problem in other, more typical contexts.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP czar for PEP 3144?

[Robert Kern]

On 2/21/12 4:52 AM, Guido van Rossum wrote:

On Mon, Feb 20, 2012 at 4:53 PM, Stephen J. Turnbull  wrote:

Steven D'Aprano writes:

  >  Also, "Czar" is commonly used in US politics as an informal term for the 
top
  >  official responsible for an area.

I think here the most important connotation is that in US parlance a
"czar" does not report to a committee, and with the exception of a
case where Sybil is appointed czar, cannot bikeshed.  Decisions get
made (what a concept!)


I'm curious how old that usage is. I first encountered it around '88
when I interned for a summer at DEC SRC (long since subsumed into HP
Labs); the person in charge of deciding a particular aspect of their
software or organization was called a czar, e.g. the documentation
czar.


From the Wikipedia article Steven cited:

"""
The earliest known use of the term for a U.S. government official was in the 
administration of Franklin Roosevelt (1933–1945), during which eleven unique 
positions (or twelve if one were to count "Economic Czar" and "Economic Czar of 
World War II" as separate) were so described. The term was revived, mostly by 
the press, to describe officials in the Nixon and Ford administrations and 
continues today.

"""

http://en.wikipedia.org/wiki/List_of_U.S._executive_branch_%27czars%27

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: [Pythonmac-SIG] Re: Re: [Python-checkins] python/dist/src/Mac/OSX fixapplepython23.py, 1.1, 1.2

[Robert Kern]

Bob Ippolito wrote:
I have no idea, but I remember this issue was brought up at some point 
during one of the many discussions of this issue.  I think it was also 
mentioned that SciPy's distutils extensions (or is it a fork?) does 
something similarly stupid when frobbing in a Fortran compiler.
Extension, and yes it does something similarly stupid.
--
Robert Kern
[EMAIL PROTECTED]
"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: license issues with profiler.py and md5.h/md5c.c

[Robert Kern]

A.M. Kuchling wrote:
On Sat, Feb 12, 2005 at 01:54:27PM +1100, Donovan Baarda wrote:
Are there any potential problems with making the md5sum module availability
"optional" in the same way as this?

The md5 module has been a standard module for a long time; making it
optional in the next version of Python isn't possible.  We'd have to
require OpenSSL to compile Python.
I'm happy to replace the MD5 and/or SHA implementations with other
code, provided other code with a suitable license can be found.
How about this one:
http://sourceforge.net/project/showfiles.php?group_id=42360
From an API standpoint, it's trivially different from the one currently 
in Python.

From md5.c:
/*
  Copyright (C) 1999, 2000, 2002 Aladdin Enterprises.  All rights reserved.
  This software is provided 'as-is', without any express or implied
  warranty.  In no event will the authors be held liable for any damages
  arising from the use of this software.
  Permission is granted to anyone to use this software for any purpose,
  including commercial applications, and to alter it and redistribute it
  freely, subject to the following restrictions:
  1. The origin of this software must not be misrepresented; you must not
 claim that you wrote the original software. If you use this software
 in a product, an acknowledgment in the product documentation would be
 appreciated but is not required.
  2. Altered source versions must be plainly marked as such, and must 
not be
 misrepresented as being the original software.
  3. This notice may not be removed or altered from any source 
distribution.

  L. Peter Deutsch
  [EMAIL PROTECTED]
 */
/* $Id: md5.c,v 1.6 2002/04/13 19:20:28 lpd Exp $ */
/*
  Independent implementation of MD5 (RFC 1321).
  This code implements the MD5 Algorithm defined in RFC 1321, whose
  text is available at
http://www.ietf.org/rfc/rfc1321.txt
  The code is derived from the text of the RFC, including the test suite
  (section A.5) but excluding the rest of Appendix A.  It does not include
  any code or documentation that is identified in the RFC as being
  copyrighted.
[etc.]
--
Robert Kern
[EMAIL PROTECTED]
"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: license issues with profiler.py and md5.h/md5c.c

[Robert Kern]

David Ascher wrote:
FWIW, I agree.  Personnally, I think that if Debian has a problem with
the above, it's their problem to deal with, not Python's.
The OSI may also have a problem with the license if they were to be made 
aware of it.

See section 8 of the Open Source Definition:
"""8. License Must Not Be Specific to a Product
The rights attached to the program must not depend on the program's 
being part of a particular software distribution. If the program is 
extracted from that distribution and used or distributed within the 
terms of the program's license, all parties to whom the program is 
redistributed should have the same rights as those that are granted in 
conjunction with the original software distribution.
"""

I'm not entirely sure if this affects the PSF's use of OSI's trademark.
IANAL. TINLA.
--
Robert Kern
[EMAIL PROTECTED]
"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Ye don't be needin' these!

[Robert Kern]

Herman Toothrot wrote:
Avast!  Why be there builtins divmod and pow, when operators **, /, and 
% should be good enough for ya?  It runs counter to TOOWTDI, I be 
thinking.  Arr.
Well, divmod(x, y) does both / and % in one shot, which can be very 
useful. pow(x, y[, z]) has an optional third argument ((x**y) % z), 
which is necessary for really large numbers like the ones you play with 
in cryptography.

--
Robert Kern
[EMAIL PROTECTED]
"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] INPLACE_ADD and INPLACE_MULTIPLY oddities in ceval.c

[Robert Kern]

Guido van Rossum wrote:
> On 3/27/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
> 
>>If you have Numeric or numpy installed try this:
>>
>>#import Numeric as N
>>import numpy as N
>>
>>a = range(10)
>>b = N.arange(10)
>>
>>a.__iadd__(b)
>>
>>print a
>>
>>Result:
>>
>>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>
>>
>>Contrast the returned output with
>>
>>import numpy as N
>>
>>a = range(10)
>>b = N.arange(10)
>>
>>a += b
>>
>>print a
>>
>>Result:
>>
>>[ 0  2  4  6  8 10 12 14 16 18]
>>
>>Having "a+=b" and "a.__iadd__(b)" do different things seems like an
>>unfortunate bug.
>>
>>It seems to me that the problem is that the INPLACE_ADD and
>>INPLACE_MULTIPLY cases in ceval.c  use PyNumber_InPlace instead of
>>trying PySequence_InPlace when the object doesn't support the
>>in-place number protocol.
>>
>>I could submit a patch if there is agreement that this is a problem.
> 
> Well how is the interpreter to know whether += meanse numeric add or
> sequence add in any particular case?
> 
> Shouldn't it be numpy's responsibility to implement both operations 
> identically?

"a" is a list object. Travis's question is, "why doesn't the list object
implement both operations identically?" I think the answer is, "it never gets a
chance to."

When a numpy array is on the LHS of an inplace operation, we always want the
numeric add. numpy arrays can't be extended inplace like a list, and a lone "+"
is always a numeric operation for arrays, never a sequence operation. When numpy
arrays are on the LHS of an inplace operation, everything works consistently.

The issue seems to arise from the fact that PyNumber_InPlaceAdd tries to coerce
the LHS and the RHS to a common numerical type first before trying
PySequence_InPlaceConcat. Lists of numbers coerce to numpy arrays just fine, so
this code path is taken, and nothing tests for the sq_inplace_concat or
sq_concat slots.

Personally, I'm willing to just document it in the numpy documentation as a
rarely-seen gotcha. This behavior pretty much affects just us.

-- 
Robert Kern
[EMAIL PROTECTED]

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] I'm not getting email from SF when assigned abug/patch

[Robert Kern]

Brett Cannon wrote:
> On 3/28/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> 
>>[EMAIL PROTECTED] wrote:
>>
>>>Roundup is there now, right (sans SF export)?
>>
>>Richard Jones has an SF importer for one of the two XML-like formats,
>>the one that is correct XML but with incomplete data. The other format,
>>which has complete data but is ill-formed XML, has no importer into
>>roundup at the moment.
>>
>>
>>>Trac is being used by the
>>>folks doing the new website.  What other candidates are being considered?
>>
>>My view is that nothing should be "considered" unless there is
>>a) a volunteer to operate the tracker (or, failing that, somebody who
>>   does it for money), and
>>b) an importer of whatever data SF can provide.
>>
> 
> 
> And as the chair of the infrastructure committee, those are the base
> requirements for a tracker to be considered.
> 
> Once we have the complete SF data I will put out a call for proposals
> for trackers along with who has volunteered to manage them so that
> people can band together to help push their favorite tracker.
> 
> But as of right now the committe just wants to get a clean dump of our
> SF data to work off of (yes, we could start a tracker from scratch but
> we want the SF data anyway and so we are going through the hassle of
> getting it off now and as a test data set for the various trackers).
> 
> -Brett
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org


-- 
Robert Kern
[EMAIL PROTECTED]

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] I'm not getting email from SF when assigned abug/patch

[Robert Kern]

Brett Cannon wrote:
> On 3/28/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:

>>My view is that nothing should be "considered" unless there is
>>a) a volunteer to operate the tracker (or, failing that, somebody who
>>   does it for money), and
>>b) an importer of whatever data SF can provide.
> 
> And as the chair of the infrastructure committee, those are the base
> requirements for a tracker to be considered.
> 
> Once we have the complete SF data I will put out a call for proposals
> for trackers along with who has volunteered to manage them so that
> people can band together to help push their favorite tracker.

FWIW: Trac has a Sourceforge bug tracker import script:

http://projects.edgewall.com/trac/browser/trunk/contrib/sourceforge2trac.py

Apologies: for the other blank reply.

-- 
Robert Kern
[EMAIL PROTECTED]

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Allow Empty Subscript List Without Parentheses

[Robert Kern]

Greg Ewing wrote:

> I'm having trouble seeing a real use for a 0D array as
> something distinct from a scalar, as opposed to them
> just being an oddity that happens to arise as a side
> effect of the way Numeric/Numpy are implemented.

This has been rehashed over and over again on numpy-discussion. The upshot is
that numpy no longer creates rank-zero arrays unless if the user really asks for
one. The remaining use cases are documented here:

  http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a note in random.shuffle.__doc__ ...

[Robert Kern]

Alex Martelli wrote:
> ...claims:
> 
> Note that for even rather small len(x), the total number of
> permutations of x is larger than the period of most random number
> generators; this implies that "most" permutations of a long
> sequence can never be generated.
> 
> Now -- why would the behavior of "most" random number generators be  
> relevant here?  The module's docs claim, for its specific Mersenne  
> Twister generator, a period of 2**19997-1, which is (e.g.) a  
> comfortable  
> 130128673800676351960752618754658780303412233749552410245124492452914636 
> 028095467780746435724876612802011164778042889281426609505759158196749438 
> 742986040468247017174321241233929215223326801091468184945617565998894057 
> 859403269022650639413550466514556014961826309062543 times larger than  
> the number of permutations of 2000 items, which doesn't really feel  
> to me like a "rather small len(x)" in this context (what I'm most  
> often shuffling is just a pack of cards -- len(x)==52 -- for example).

I wouldn't be too comfortable with that margin. The fun thing about factorials
is that they add up really quickly. The crossover point is between 2080 and 
2081.


In [28]: from scipy import *

In [29]: def f(x):
   : return special.gammaln(x-1) - 19937*log(2)
   :

In [30]: optimize.brentq(f, 2000, 3000)
Out[30]: 2082.4031300820125

In [31]: import gmpy

In [32]: mtperiod = 2**19937 - 1

In [33]: for i in range(2075, 2085):
   : if gmpy.fac(i) > mtperiod:
   : print i
   : break
   :
   :
2081


I think that documenting this boundary might be worthwhile rather than making
vague references to "small len(x)." A note along the lines of Josiah wrote about
there being no guarantees despite period size would also be useful.

OTOH, isn't the exact PRNG algorithm considered an implementation detail? It
certainly was when the module migrated from Wichmann-Hill to the Mersenne
Twister. OTGH, I don't foresee the random module ever using an algorithm with
worse characteristics than the Mersenne Twister.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] External Package Maintenance (was Re: Please stopchanging wsgiref on the trunk)

[Robert Kern]

Paul Moore wrote:

> This is purely speculation (and fairly negative speculation at that!)
> but I'd advocate caution for a while yet. Maybe ActiveState or
> Enthought could provide some input on how/if sumo bundles can address
> the need to upgrade "parts" of the distribution?

We at Enthought are moving towards distributing a smallish bootstrap
distribution and provide most of the packages as eggs. Upgrades should then be
an easy_install away.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3 design principles

[Robert Kern]

Oren Tirosh wrote:

> While a lot of existing code will break on 3.0 it is still generally
> possible to write code that will run on both 2.x and 3.0: use only the
> "proper" forms above, do not assume the result of zip or range is a
> list, use absolute imports (and avoid static types, of course). I
> already write all my new code this way.
> 
> Is this "common subset" a happy coincidence or a design principle? 

I think it's because those are the most obvious things right now. The
really radical stuff won't come up until active development on Python
3000 actually starts. And it will, so any "common subset" will probably
not be very large.

IMO, if we are going to restrict Python 3000 enough to protect that
"common subset," then there's not enough payoff to justify breaking
*any* backwards compatibility. If my current codebase[1] isn't going to
be supported in Python 3000, I'm going to want the Python developers to
use that opportunity to the fullest advantage to make a better language.

[1] By which I mean the sum total of the code that I use not just code
that I've personally written. I am a library-whore.

-- 
Robert Kern
[EMAIL PROTECTED]

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] reference counting in Py3K

[Robert Kern]

Fredrik Lundh wrote:
> Josiah Carlson wrote:
> 
>>Perhaps a bit into the future, extending import semantics to notice .pyx
>>files, compare their checksum against a stored md5 in the compiled
>>.pyd/.so, and automatically recompiling them if they (or their includes)
>>have changed: +10 (I end up doing this kind of thing by hand with
>>phantom auto-build modules).

http://www.prescod.net/pyximport/

> which reminds me...  does anyone know what happened to the various
> "inline C" versions that were quite popular a few years ago.  e.g.
> 
> http://mail.python.org/pipermail/python-dev/2002-January/019178.html
> 
> (I've been using an extremely simple home-brewn version in a couple of
> projects, and it's extremely addictive, at least if you're a C/C++ veteran...)

weave is alive and kicking and actively used although it could use some TLC.

http://svn.scipy.org/svn/scipy_core

-- 
Robert Kern
[EMAIL PROTECTED]

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problems with the Python Memory Manager

[Robert Kern]

Josiah Carlson wrote:
> Travis Oliphant <[EMAIL PROTECTED]> wrote:

>>I think definitely, his usage pattern represented a "bad" corner case.  
>>An unusable "corner" case in fact.   At any rate, moving to use the 
>>system free and malloc fixed the immediate problem.  I mainly wanted to 
>>report the problem here just as another piece of anecdotal evidence.
> 
> On the one hand, using PyObjects embedded in an array in scientific
> Python is a good idea; you can use all of the standard Python
> manipulations on them.  On the other hand, other similar projects have
> found it more efficient to never embed PyObjects in their arrays, and
> just allocate them as necessary on access.

That's not what we're doing[1]. The scipy_core arrays here are just
blocks of C doubles. However, the offending code (I believe Chris
Fonnesbeck's PyMC, but I could be mistaken) frequently indexes into
these arrays to get scalar values. In scipy_core, we've defined a set of
numerical types that generally behave like Python ints and floats but
have the underlying storage of the appropriate C data type and have the
various array attributes and methods. When the result of an indexing
operation is a scalar (e.g., arange(10)[0]), it always returns an
instance of the appropriate scalar type. We are "just allocat[ing] them
as necessary on access."

[1] There *is* an array type for general PyObjects in scipy_core, but
that's not being used in the code that blows up and has nothing to do
with the problem Travis is talking about.

-- 
Robert Kern
[EMAIL PROTECTED]

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problems with the Python Memory Manager

[Robert Kern]

Martin v. Löwis wrote:

> That the code is complex would not so much be a problem: we often
> analyse complex code here. It is a problem that the code is not
> available, and it would be a problem if the problem was not
> reproducable even if you had the code (i.e. if the problem would
> sometimes occur, but not the next day when you ran it again).

You can get the version of scipy_core just before the fix that Travis
applied:

  svn co -r 1488 http://svn.scipy.org/svn/scipy_core/trunk

The fix:

  http://projects.scipy.org/scipy/scipy_core/changeset/1489
  http://projects.scipy.org/scipy/scipy_core/changeset/1490

Here's some code that eats up memory with rev1488, but not with the HEAD:

"""
import scipy

a = scipy.arange(10)
for i in xrange(10000000):
x = a[5]
"""

-- 
Robert Kern
[EMAIL PROTECTED]

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com