[Python-Dev] HTMLParser and HTML5

[Matt]

Hello all,

I wanted to ask a few questions and start a discussion about HTML5
support within the HTMLParser class(es). Over on issue 670664, an
inconsistency with the way browsers and the HTMLParser parse script
and style tags was discovered. Currently, HTMLParser adheres strictly
to the HTML4 standard, which says that these tags should exit CDATA
mode when the start of *any* closing tag is found. No browsers, to my
knowledge, have ever supported this (at least in the 21st century).
Instead, all browsers implement the behavior described in the HTML5
spec, which states that script tags should exit their "raw text mode"
when the full closing tag for that element is encountered.

The repercussions of adhering to the HTML4 standard in HTMLParser are
somewhat serious: a good number of documents will either encounter
exceptions for broken markup (which aren't actually broken). Libraries
like Beautiful Soup (which depend on HTMLParser) are also affected,
requiring the use of hacks just to get the document to parse at all.

Rather than bore you all with another paragraph about how HTML4 is
terrible, feel free to look at the issue
(http://bugs.python.org/issue670664), which quite thoroughly outlines
the pros and cons of this particular change. Any feedback/input  on
the proposed changes is welcome.

So here are my questions:

- What plans, if any, are there to support HTML5 parsing behaviors,
since the HTML5 spec effectively describes current web browser
behavior?
- What policies are in place for keeping parity with other HTML
parsers (such as those in web browsers)?

Given the semi-backward-compatible nature of HTML5's syntax, this
seems like a rather unique problem that could use some more
discussion.


Thanks

Matt Basta
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] HTMLParser and HTML5

[Matt]

On Fri, Jul 29, 2011 at 11:03 AM, Glyph Lefkowitz
wrote:

>
> On Jul 29, 2011, at 7:46 AM, Stefan Behnel wrote:
>
> > Joao S. O. Bueno, 29.07.2011 13:22:
> >> On Fri, Jul 29, 2011 at 1:37 AM, Stefan Behnel wrote:
> >>> Brett Cannon, 28.07.2011 23:49:
> >>>>
> >>>> On Thu, Jul 28, 2011 at 11:25, Matt wrote:
> >>>>>
> >>>>> - What policies are in place for keeping parity with other HTML
> >>>>> parsers (such as those in web browsers)?
> >>>>
> >>>> There aren't any beyond "it would be nice".
> >>>> [...]
> >>>> It's more of an issue of someone caring enough to do the coding work
> to
> >>>> bring the parser up to spec for HTML5 (or introduce new code to live
> >>>> beside
> >>>> the HTML4 parsing code).
> >>>
> >>> Which, given that html5lib readily exists, would likely be a lot more
> work
> >>> than anyone who is interested in HTML5 handling would want to invest.
> >>>
> >>> I don't think we need a new HTML5 parsing implementation only to have
> it in
> >>> the stdlib. That's the old sunny Java way of doing it.
> >>
> >> I disaagree.
> >> Having proper html parsing out of the box is part of the "batteries
> >> included" thing.
> >
> > Well, you can easily prove me wrong by implementing this.
>

As far as the issue described in my initial message goes, there is a patch
and tests for the patch.


>
> Please don't implement this just to profe Stefan wrong :).
>
> The thing to do, if you want html parsing in the stdlib, is to
> _incorporate_ html5lib, which is already a perfectly good, thoroughly tested
> HTML parser, and simply deprecate HTMLParser and friends.  Implementing a
> new parser would serve no purpose I can see.
>

I don't see any real reason to drop a decent piece of code (HTMLParser, that
is) in favor of a third party library when only relatively minor updates are
needed to bring it up to speed with the latest spec. As far as structure
goes, HTML4 and HTML5 are practically identical. The differences between the
two that are applicable to HTMLParser involve the way the specs deal with
special element types and broken syntax. For what it's worth, the rules
HTML4 does define are (in many cases) ignored in favor of more modern,
Postel's Law-agreeable rules. HTML5 simply standardized what browsers
actually do.

Deprecating HTMLParser in favor of a newer/better/faster HTML library is a
bad thing for everybody that's already using HTMLParser, whether directly or
indirectly. html5lib does not have an interface compatible with HTMLParser,
so code would largely need to be rewritten from scratch to gain the benefits
of HTML5's support for broken code. Developers using HTMLParser would be
permanently stuck using a library that throws exceptions for perfectly valid
HTML. Keep in mind that these are solved problems: all of the thinking on
how to handle broken code has been done for us by the folks at the WHATWG.
It's simply a matter of updating our existing code with these new rules.

While I agree that there are merits to dropping support for the old code, it
does not solve the existing problems that folks are having right now (namely
incorrect parser output or exceptions). It would be more ideal to perhaps
patch the obvious issues stemming from HTML4 support for now, leaving
anything that goes beyond parity with browsers for a later time or
implementing as an opt-in feature (i.e.: enabled by a parameter).

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Matt Billenstein]

Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
deprecated (and no longer ships the header files for) the version shipped with
recent versions of osx.

Perhaps this is an option to support the various flavors of Linux as well?

m

On Sun, Jan 14, 2018 at 02:48:49AM +, Paul G wrote:
>One thing to note is that if getting Travis working with Python 3.7 is a
>pain, a huge number of libraries on PyPI probably just won't test against
>Python 3.7, which is not a great situation to be in.
> 
>It's probably worth contacting Travis to give them a head's up and see how
>likely it is that they'll be able to support Python 3.7 if it requires a
>newer version of these libraries.
> 
>On January 14, 2018 2:16:53 AM UTC, Brett Cannon  wrote:
> 
>  On Sat, Jan 13, 2018, 14:45 Christian Heimes, 
>  wrote:
> 
>On 2018-01-13 21:02, Brett Cannon wrote:
>> +1 from me as well for the improved security.
> 
>Thanks, Brett!
> 
>How should we handle CPython's Travis CI tests? The 14.04 boxes have
>OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer
>16.04.
>We could either move to container-based testing with a 16.04
>container,
>which would give us 1.0.2 Or we could compile our own copy of OpenSSL
>with my multissl builder and use some rpath magic.
> 
>In order to test all new features, Ubuntu doesn't cut it. Even current
>snapshot of Ubuntu doesn't contain OpenSSL 1.1. Debian Stretch or
>Fedora
>would do the trick, though.
> 
>Maybe Barry's work on official test container could leveraged testing?
> 
>  My guess is we either move to containers on Travis, see if we can
>  manually install -- through apt or something -- a newer version of
>  OpenSSL, or we look at alternative CI options.
>  -Brett
> 
>Regards,
>Christian

> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/matt%40vazor.com


-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Matt Billenstein]

On Sun, Jan 14, 2018 at 10:54:57AM -0500, Ned Deily wrote:
> On Jan 14, 2018, at 08:39, Christian Heimes  wrote:
> > On 2018-01-14 09:24, Matt Billenstein wrote:
> >> Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has
> >> deprecated (and no longer ships the header files for) the version shipped 
> >> with
> >> recent versions of osx.
> >> 
> >> Perhaps this is an option to support the various flavors of Linux as well?
> > 
> > AFAK Apple has decided to compile and statically link CPython's ssl with
> > an ancient, customized LibreSSL version. Cory posted [1] a couple of
> > months ago
> 
> What Matt is likely thinking of is the Python 3 versions provided by the
> python.org macOS binary installers where we do build and link with our
> own 1.0.2 (and soon 1.1.0 for 3.7) versions of OpenSSL.

Yes, referring to the Python3 python.org installers -- I'm seeing this practice
of bundling libs (particularly ssl) become more common as operating system
support lags behind.  In my mind it becomes easier to bundle deps in a binary
installer across the board (Linux, OSX, Windows) rather than rely on whatever
version the operating system provides.

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

[Matt Billenstein]

On Tue, Jan 30, 2018 at 09:42:07AM -0800, Chris Barker wrote:
>IT dept has been making me upgrade, so I"m going to guess 10.8 or newer...

OSX is in a sad state linking to system libs on the later releases -- maybe
10.11 and on, not sure of the exact release -- they stopped shipping the
headers for things like ssl and ffi since they don't want 3rd parties linking
to deprecated versions of those libraries versus, in the case of ssl, their
newer security framework.  Recommendation is to bundle what you need if you're
not using the framework -- something to think about.

thx

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] libxml2 installation/binding issue

[Priest, Matt]

Hello,

I am not sure if this is the correct place to post an issue/question like this, 
but here goes...

I've successfully (?) installed Python 3.6.4 and libxml2, with the ultimate 
goal of installing GTK+ 3.22.0.
However, I'm running into this error:


python3
Python 3.6.4 (default, Feb  5 2018, 13:28:04)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libxml2
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib/python3.6/site-packages/libxml2.py",
 line 1, in 
import libxml2mod
ImportError: 
/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib/python3.6/site-packages/libxml2mod.so:
 undefined symbol: _PyVerify_fd


Here are the details on the version, cflags, and ldflags.
python3 --version ;
Python 3.6.4
python3-config --cflags
-I/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/include/python3.6m
-I/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/include/python3.6m
-Wno-unused-result
-Wsign-compare
-fPIC -DNDEBUG
-g
-fwrapv
-O3
-Wall
-Wstrict-prototypes

python3-config -ldflags;
-L/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib/python3.6/config-3.6m-x86_64-linux-gnu
-L/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib
-lpython3.6m
-lpthread
-ldl
-lutil
-lrt
-lm
-Xlinker
-export-dynamic

Anyhelp or hint would be appreciated...



Matt

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move ensurepip blobs to external place

[Matt Billenstein]

As i recall git LFS makes storing large binary objects in some external object 
storage fairly seamless - might be a good fit for keeping the same workflow and 
not bloating the repo.

M

--
Matt Billenstein
m...@vazor.com

Sent from my iPhone 6 (this put here so you know I have one)

> On Mar 24, 2018, at 8:27 PM, Nick Coghlan  wrote:
> 
>> On 25 March 2018 at 06:52, Ned Deily  wrote:
>> On Mar 24, 2018, at 16:13, Steve Dower  wrote:
>> > Or we could just pull the right version directly from PyPI? (Note that 
>> > updating the version should be an explicit step, as it is today, but the 
>> > file should be identical to what’s on PyPI, right? And a urlretrieve is 
>> > easier than pulling from a git repo.)
>> 
>> I think the primary original rationale for having the pip wheel and its 
>> dependencies checked into the cpython repo was so that users would be able 
>> to install pip even if they did not have an Internet connection.  But 
>> perhaps that requirement can be relaxed a bit if we say that the necessary 
>> wheels are vendored into all of our downloadable release items, that is, 
>> included in the packaging of source release files (the various tarballs) and 
>> the Windows and macOS binary installers.  The main change would likely be 
>> making ensurepip a bit smarter to download if the bundled wheels are not 
>> present in the source directory.  Assuming that people building from a 
>> cpython repo need to have a network connection if they want to run 
>> ensurepip, at least for the first time, is probably not an onerous 
>> requirement.
> 
> Right, having the wheels in the release artifacts is a requirement, as is 
> having them available for use when running the test suite, but having them in 
> the git repo isn't.
> 
> Adding them directly to the repo was just the simplest approach to getting 
> ensurepip working, since it didn't require any changes to the build process.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/matt%40vazor.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5

[Matt Billenstein]

On Wed, Jan 08, 2014 at 07:12:06PM +0100, Stefan Behnel wrote:
> Why can't someone write a third-party library that does what these projects
> need, and that works in both Py2 and Py3, so that these projects can be
> modified to use that library and thus get on with their porting to Py3?

Apologies if this is out of place and slightly OT and soap-boxey...

Does it not strike anyone here how odd it is that one would need a library to
manipulate binary data in a programming language with "batteries included" on a
binary computer?  And maybe you can do it with existing facilities in both
versions of Python, although in python3, I need to understand what bytes,
format, ascii, and surrogateescape mean - among other things.

I started in Python blissfully unaware of unicode - it was a different time for
sure, but what I knew from C worked pretty much the same in Python - I could
read some binary data out of a file, twiddle some bits, and write it back out
again without any of these complexities - life was good and granted I was
naive, but it made Python approachable for me and I enjoyed it.  I stuck with
it and learned about unicode and the complexities of encoding data and now I'm
astonished at how many professional programmers don't know the slightest bit
about it and how horribly munged some data you can consume on the web might be
- I agree it's all quite a mess.

So now I'm getting more serious about Python3 and my fear is that the
development community (python3) has fractured from the user community (python2)
in that they've built something that solves their problems (to oversimplify
lets say a webapp) - sure, a bunch of stuff got fixed along the way and we gave
the users division they would expect (3/2 == 1.5), but somewhere what I felt
was more like a hobbyist language has become big and complex and "we need to
protect our users from doing the wrong thing."

And I think everyone was well intentioned - and python3 covers most of the
bases, but working with binary data is not only a "wire-protocol programmer's"
problem.  Needing a library to wrap bytesthing.format('ascii', 
'surrogateescape')
or some such thing makes python3 less approachable for those who haven't
learned that yet - which was almost all of us at some point when we started
programming.

I appreciate everyone's hard work - I'm confident the community will cross the
2-3 chasm and I hope we preserve the approachability I first came to love about
Python when I started using it for all sorts of applications.

thx

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Matt Arcidy]

This cynical view on students is shocking!  Everyone on this list has
been a student or a learner for far longer than an educator, and the
perspective from students and learners are far more important than
educators to assess this angle regardless.  Can anyone adequately
explain why this specific modality of learning,  a student-in-a-seat
based educator, must outweigh all other modalities learners use to
increase knowledge and skill, from the perspectives of policy, tool
creation, and each of our time spent learning?

Shortest story:
Teach not to re-use names.

Short story:
1) What about the full mosaic of learning vs. this myopic view on
seat-based student-educator interaction?
2) What about smart, motivated, diligent and cautious students?
3) What weight should educator opinion be given with respect to
providing convenience to professional Python programmers?
4) Who is this Student Stupid von Densemeister anyways?
5) Are assignment expressions convenience and is any danger the pose
unmitagatble?
6) Consider adding an "Important Topics not Covered" or "Further
Reading" reading section to your class description
7) Creating examples showing this effect is easy, especially when not
actually re-using the name in the expression for explanatory purposes.
it's the same as creating examples showing how re-use works in
comprehensions.


Let's stop constructing these fake Students.  They only work as
appeals to the people we have come across whose lack of understanding
has made our life painful.  This construction is actively filtering
all the good students for the sake of influencing this decision, yet
again punishing or discounting the intelligent, quick, and diligent.

And what of this underlying premise that educator's should
_significantly_ influence language development?  Limiting Python's
tools to Student Straw-man's ability to learn is just dissonant, they
have nothing to do with each other, nor does this cause-effect
relationship actually exist.   Let's evaluate this reductionist
statement:
"I understand X, but this other person is not capable of understanding
X, therefore X should not exist"  Is has there ever been an X for
which this is true, let alone the backwardation necessary to fully
close the statement?

The actual argument is far less reductionist, yet even more ridiculous:
"I understand X,  this other person may take time to learn X, and may
use X wrong, therefore X should not exist"
"I understand assignment expressions, but this other class of person
may take time to learn assignment expressions, and may use assignment
expressions wrong, therefore assignment expressions should not be
accepted"

Rhetorically I disagree with how teaching is being presented, to the
point of near insult (for me lacking a better term).  You are saying
these statements about _my_ learning path, (though not personally of
course.)  Each of you occupied a role of student at some point, and
each of these statements are being made about your path as well.  Do
these ring true of your student experience?  What about your much
broader experience as a _learner_?  You think a tool shouldn't exist
because it took you time to learn it and you wrote some hard to debug
code, and possibly crashed production, got fired, lost your house and
your pet snake, and crashed the planet into the sun?

Now I yield, I will accept this position: all/some students cannot
learn this (or it's too complex to teach), but they must learn this
during some class to quickly become effective python developers.  How
much weight should this position have in this decision?  Let's appeal
to the learner in us.  How much of our learner's path, percentage of
total time learning all things python related, has been in a seat
listening to someone else, and that's the only place from which we
gained the knowledge to meet the educator's objective?  This time
spent in a class, how does that compare to hours in other learning
modalities?  Is this percentage not exactly the weight assigned to
that position?  Are people hired from pure class-room based experience
expected to require zero further learning?  Are people more valuable
based on classroom hours or work hours?

As for handling teaching the subject or not, this is easily remedied
with how I do it: "Important Topics not Covered", with resources.

Anyone here can rightfully claim educator status by having taught
another person something related to this language, which includes
at-work mentoring, informal discussions, posting/replying on SO,
blogging, etc.  Are they not being solicited to comment as well?  It's
possible to answer this question while vehemently disagreeing with the
PEP.  This focus on people who are being ostensibly paid to teach is
myopic.

Concretely, it's clear to me that parent-local effects can be
dangerously non-obvious when reading and mimicking code without
undertsanding.  But when?  And how to guard against?  How about this:
teach proper (i.e. not) re-using names.  The name will still be
ej

Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Matt Arcidy]

On Mon, Jul 2, 2018 at 2:34 AM Michael Selik  wrote:
>
> On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy  wrote:
>>
>> [...] Can anyone adequately explain why this specific modality of learning,  
>> a student-in-a-seat based educator, must outweigh all other modalities [...]?
>
>
> 1. It doesn't.
> 2. It's a proxy for the other modes.
>
> I hope this was an adequate explanation.

Absolutely, thank you.  We agree it doesn't out weigh other methods.
Clearly I disagree about the proxying.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does the Contributor Agreement need my address?

[Matt Arcidy]

On Sun, Sep 9, 2018, 12:59 Antoine Pitrou  wrote:

>
>
> I'm not sure why anyone would ask that question.


because if they can discredit a witness, they will.

Matt

>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/marcidy%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: What to do about invalid escape sequences

[Matt Billenstein]

On Mon, Aug 05, 2019 at 04:22:50AM -, raymond.hettin...@gmail.com wrote:
> This once seemed like a reasonable and innocuous idea to me; however, I've
> been using the 3.8 beta heavily for a month and no longer think it is a good
> idea.  The warning crops up frequently, often due to third-party packages
> (such as docutils and bottle) that users can't easily do anything about.

Perhaps those packages could be flagged now via pylint and problems raised with
the respective package maintainers before the actual 3.8 release?  Checking the
top 100 or top 1000 packages on PyPI?

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZS5B2EOCBTJGSHMVTL4ZXNEEGGBL2RN6/

[Python-Dev] Re: What to do about invalid escape sequences

[Matt Billenstein]

On Tue, Aug 06, 2019 at 04:32:04PM +, Matt Billenstein wrote:
> Perhaps those packages could be flagged now via pylint and problems raised 
> with
> the respective package maintainers before the actual 3.8 release?  Checking 
> the
> top 100 or top 1000 packages on PyPI?

fwiw, ran pylint on the top 100 pypi pkgs from:

https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json

The list of packages is pretty small:

https://gist.github.com/mattbillenstein/ad862d032b8575f8d6e08384850f2223

but some have quite a few errors...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TKDVLMRD4NRNQD3X3GYKSG5D2I3AKPBO/

[Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?

[Matt Newell]

I have encountered what I believe to be a bug but I'm sure there is some 
reason things are done as they are and I am hoping someone can shed some light 
or confirm it is indeed a bug.

As a bit of background I have a c++ class that I use sip to generate python 
bindings.  The potential python bug manifests itself as:

>>> rl = RecordList()
>>> rl += []
>>> rl
NotImplemented

The bindings fill in nb_inplace_add which appears to be implemented properly, 
returning a new reference to Py_NotImplemented if the right hand argument is 
not as expected.

Where things appear to go wrong is that PyNumber_InPlaceAdd, after getting a 
NotImplemented return value from nb_inplace_add, then attempts to call 
sq_inplace_concat.  From reading the code it appears that sq_inplace_concat is 
not supposed to return NotImplemented, instead it should set an exception and 
return null if the right hand arg is not supported.  In my case 
sq_inplace_concat ends up being the same function as nb_inplace_add, which 
results in the buggy behavior.

When I figured this out I tried to find out why sq_inplace_concat was set to 
the 
same function as nb_inplace_add, and ended up having to set a watchpoint in 
gdb which finally gave me the answer that python itself is setting 
sq_inplace_concat during type creation in the various functions in 
typeobject.c. Stack trace is below.

I don't really understand what the fixup_slot_dispatchers function is doing, 
but it does seem like there must be a bug either in what it's doing, or in 
PyNumber_InPlaceAdd's handling of a NotImplemented return value from 
sq_inplace_concat.

Thanks,
Matt



Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2


Stack trace where a watch on sq->sq_inplace_concat reveals the change:

Hardware watchpoint 5: *(binaryfunc *) 0xcf6f88

Old value = (binaryfunc) 0
New value = (binaryfunc) 0x74d41c78 

#0  update_one_slot.25588 (type=type@entry=0xcf6c70, p=0x86ba90) at 
../Objects/typeobject.c:6203
#1  0x004b96d0 in fixup_slot_dispatchers (type=0xcf6c70) at 
../Objects/typeobject.c:6299
#2  type_new.part.40 (kwds=, args=0x0, metatype=) at ../Objects/typeobject.c:2464
#3  type_new.25999 (metatype=, args=0x0, kwds=) 
at ../Objects/typeobject.c:2048
#4  0x00463c08 in type_call.25547 (type=0x765953a0, 
args=('RecordList', (,), 
{'__module__': 'blur.Stone'}), kwds=0x0) at ../Objects/typeobject.c:721
#5  0x004644eb in PyObject_Call (func=, 
arg=, kw=) at ../Objects/abstract.c:2529


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is nb_inplace_add copied to sq_inplace_concat?

[Matt Newell]

On Thursday, May 16, 2013 08:41:32 PM you wrote:
> On Fri, May 17, 2013 at 1:38 PM, Nick Coghlan  wrote:
> > On Fri, May 17, 2013 at 9:17 AM, Matt Newell  wrote:
> >> I don't really understand what the fixup_slot_dispatchers function is
> >> doing, but it does seem like there must be a bug either in what it's
> >> doing, or in PyNumber_InPlaceAdd's handling of a NotImplemented return
> >> value from sq_inplace_concat.
> > 
> > I didn't read your post in detail, but operand precedence in CPython
> > is known to be broken for types which only populate the sq_* slots
> > without also populating the corresponding nb_* slots:
> > http://bugs.python.org/issue11477
In this case it's the other way around.  Only nb_inplace_add is populated, and 
python forces the buggy behavior that you describe below by copying 
nb_inplace_add to sq_inplace_concat. 

> 
> Oops, I meant to state that one of the consequences of the bug is that
> returning NotImplemented from the sq_* methods doesn't work at all -
> it's never checked and thus never turned into a TypeError. That's why
> changing to delegation from the nb_* slots is the most promising
> approach - all that handling is there and correct for the numeric
> types, but pure sequence types (which can only be created from C code)
> bypass that handling.
> 
> I *did* read enough of the original post to know that was the symptom
> you were seeing, I just failed to mention that in my initial reply...
> 

I read through the bug and it looks like whatever solution you choose will fix 
this problem also.  

In the meantime I guess the solution for me is to always define 
sq_inplace_concat with a function that simply raises a TypeError.  Hmm, even 
simpler would be to reset sq_inplace_concat to 0 after python sets it.  I 
actually tested the latter in gdb and it gave the correct results.  I'll just 
have to keep an eye out to make sure my workaround doesn't break things when 
the real fix gets into python.

Matt

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

It seems unittest.TestSuite holds references to unittest.TestCase instances
after the test runs, until the test suite finishes. In a large suite, where
the TestCase instances consume memory during execution, that can lead to
exhausting all available memory and the OS killing the test process.

What do you think of a change like this?

$ hg diff
diff -r 3bd55ec317a7 Lib/unittest/suite.py
--- a/Lib/unittest/suite.py Thu Aug 01 23:57:21 2013 +0200
+++ b/Lib/unittest/suite.py Fri Aug 02 07:42:22 2013 -0400
@@ -90,7 +90,12 @@
 if getattr(result, '_testRunEntered', False) is False:
 result._testRunEntered = topLevel = True

-for test in self:
+while True:
+try:
+test = self._tests.pop(0)
+except IndexError:
+break
+
 if result.shouldStop:
 break

See also the conversation on django-developers[1] that led me here.

[1]: https://groups.google.com/forum/#!topic/django-developers/XUMetDSGVT0

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

On Fri, Aug 2, 2013 at 11:13 AM, Michael Foord wrote:

> There's an open bug for this with some discussion and a proposed fix:
>
> http://bugs.python.org/issue11798
>
> The agreed on approach just needs doing.
>

Thanks for the link. I hadn't found that yet. I'll see if I can contribute
there.

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

Michael Foord  voidspace.org.uk> writes:
> On 2 Aug 2013, at 19:19, Antoine Pitrou  pitrou.net> wrote:
> > The patch is basically ready for commit, except for a possible doc
> > addition, no?
>
> Looks to be the case, reading the patch it looks fine. I'm currently on
holiday until Monday.
> If anyone is motivated to do the docs too and commit that would be great.
Otherwise I'll
> get to it on my return.

It looks like the patch is based on what will become 3.4. Would backporting
it to 2.7 be feasible?  What's involved in doing so?

I took a crack at the docs.

# HG changeset patch
# User Matt McClure 
# Date 1375538965 14400
# Node ID d748d70201929288c230862da4dbdba33d61ae9f
# Parent  bf43956356ffe93e75ffdd5a7a8164fc68cf14ae
[11798] Document TestSuite.{__iter__, run} changes

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -1470,15 +1470,24 @@

   Tests grouped by a :class:`TestSuite` are always accessed by
iteration.
   Subclasses can lazily provide tests by overriding :meth:`__iter__`.
Note
-  that this method maybe called several times on a single suite
-  (for example when counting tests or comparing for equality)
-  so the tests returned must be the same for repeated iterations.
+  that this method may be called several times on a single suite (for
+  example when counting tests or comparing for equality) so the tests
+  returned by repeated iterations before :meth:`TestSuite.run` must be
the
+  same for each call iteration. After :meth:`TestSuite.run`, callers
should
+  not rely on the tests returned by this method unless the caller uses
a
+  subclass that overrides :meth:`TestSuite._removeTestAtIndex` to
preserve
+  test references.

   .. versionchanged:: 3.2
  In earlier versions the :class:`TestSuite` accessed tests
directly rather
  than through iteration, so overriding :meth:`__iter__` wasn't
sufficient
  for providing tests.

+  .. versionchanged:: 3.4
+ In earlier versions the :class:`TestSuite` held references to each
+ :class:`TestCase` after :meth:`TestSuite.run`. Subclasses can
restore
+ that behavior by overriding :meth:`TestSuite._removeTestAtIndex`.
+
In the typical usage of a :class:`TestSuite` object, the :meth:`run`
method
is invoked by a :class:`TestRunner` rather than by the end-user test
harness.

diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py
--- a/Lib/unittest/suite.py
+++ b/Lib/unittest/suite.py
@@ -65,6 +65,7 @@
 return result

 def _removeTestAtIndex(self, index):
+"""Stop holding a reference to the TestCase at index."""
     try:
 self._tests[index] = None
 except TypeError:

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

On Aug 3, 2013, at 12:07 PM, "R. David Murray"  wrote:

> Thanks.  Please post your patch to the issue, it will get lost here.

I'm trying to register, but I'm not receiving a confirmation email to complete 
the registration. 

-- 
http://matthewlmcclure.com
http://about.mapmyfitness.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

On Sat, Aug 3, 2013 at 3:27 PM, Michael Foord wrote:

> It smells to me like a new feature rather than a bugfix, and it's a
> moderately big change. I don't think it can be backported to 2.7 other than
> through unittest2.
>

Is http://hg.python.org/unittest2 the place to backport to unittest2?

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] unittest.TestSuite holding references to unittest.TestCase instances too long

[Matt McClure]

Hi Michael,

On Tue, Aug 6, 2013 at 4:25 AM, Michael Foord wrote:

> unittest itself has changed so extensively since the last release of
> unittest2 that I'm not sure whether a completely fresh start for unittest2
> might be needed. (Although I intend to do another bugfix release of this
> version as well.)
>
> Making unittest2 will involve:
>
> * Taking the Python 3 unittest and porting code plus tests to run
> on python 2
> [ ... ]
>

I took a different approach and simply applied the patch of diffs[1] from
the Python 3 issue to the unittest2 tip.

There was a small amount of renaming "unittest" to "unittest2" required,
but other than that, the patch applied pretty cleanly, and seems to pass
the unit tests and avoid the ever-increasing memory problem in my private
test suite.

Do you think it's sufficient to port just this feature? Or if not, what am
I missing that requires resyncing more of unittest2 with the changes from
Python 3?

Is Google Code[2] still the right place for unittest2 issues? I found that
via PyPI[3].

It looks like there have been a lot of commits in the unittest2 repository
since the last PyPI release (2010-07-12 -- 0.5.1). Would you plan to do
another PyPI release of unittest2 with this feature? Or would you recommend
using unittest2 from the repository to get it? Or am I missing a more
recent packaged release somewhere else?

[1]:
https://bitbucket.org/matthewlmcclure/unittest2/compare/issue11798-tip..issue11798-base#diff
[2]: https://code.google.com/p/unittest-ext/issues/detail?id=76&sort=-id
[3]: https://pypi.python.org/pypi/unittest2

-- 
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] A question concerning named pipes, multiprocessing.Queue (multiprocessing.queues.JoinableQueue)

[Matt Gregory]

I raised this issue and question on StackExchange and #python (FreeNode) 
and have received little or no feedback.  I fear that the only answer 
will lie in profiling the python interpreter itself, which is beyond the 
scope of my capabilities at present.


The original question can be found here: 
http://stackoverflow.com/questions/38637282/multiprocessing-queue-seems-to-go-away-os-pipe-destruction-vs-python


Cliffs: after about one to two hours of processing, the consumer process 
reading the queue terminates on timeout during queue.get(). The thread 
writing objects to the queue receives no exception continues writing to 
the queue.  The thread keeps track of each child process using a simple 
local list of processes, and each time an object is added to the queue, 
the processes objects (consumers) are checked with .is_alive().  When 
the child processes suicide on timeout, is_alive() continues to return 
True, so they are not garbage collected nor are new processes started.


I sincerely apologize if my own understanding of the documentation is at 
fault, but to the best of my knowledge this code should work according 
to the documentation.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Snap Python for simple distribution across multiple Linux distros

[Matt Billenstein]

On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
> Is there someone here who'd be interested in doing the same for Python?

Do snaps support Windows and/or MacOS?

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socketserver ForkingMixin waiting for child processes

[Matt Billenstein]

Common pattern I've used is to wait a bit, then send a kill signal.

M

--
Matt Billenstein
m...@vazor.com

Sent from my iPhone 6 (this put here so you know I have one)

> On Aug 11, 2017, at 5:44 AM, Victor Stinner  wrote:
> 
> Hi,
> 
> I'm working on reducing the failure rate of Python CIs (Travis CI,
> AppVeyor, buildbots). For that, I'm trying to reduce test side effects
> using "environment altered" warnings. This week, I worked on
> support.reap_children() which detects leaked child processes (usually
> created with os.fork()).
> 
> I found a bug in the socketserver module: it waits for child processes
> completion, but only in non-blocking mode. If a child process takes
> too long, the server will never reads its exit status and so the
> server leaks "zombie processes". Leaking processes can increase the
> memory usage, spawning new processes can fail, etc.
> 
> => http://bugs.python.org/issue31151
> 
> I changed the code to call waitpid() in blocking mode on each child
> process on server_close(), to ensure that all children completed when
> on server close:
> 
> https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049
> 
> After pushing my change, I'm not sure anymore if it's a good idea.
> There is a risk that server_close() blocks if a child is stuck on a
> socket recv() or send() for some reasons.
> 
> Should we relax the code by waiting a few seconds (problem: hardcoded
> timeouts are always a bad idea), or *terminate* processes (SIGKILL on
> UNIX) if they don't complete fast enough?
> 
> I don't know which applications use socketserver. How I can test if it
> breaks code in the wild?
> 
> At least, I didn't notice any regression on Python CIs.
> 
> Well, maybe the change is ok for the master branch. But I would like
> your opinion because now I would like to backport the fix to 2.7 and
> 3.6 branches. It might break some applications.
> 
> If we *cannot* backport such change to 2.7 and 3.6 because it changes
> the behaviour, I will fix the bug in test_socketserver.py instead.
> 
> What do you think?
> 
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/matt%40vazor.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: python3 -bb and hash collisions

[Matt Billenstein]

On Tue, Sep 10, 2019 at 10:42:52AM -0400, Daniel Holth wrote:
> I stopped using Python 3 after learning about str(bytes) by finding it in my
> corrupted database. Ever since then I've been anxious about changing to the 
> new
> language, since it makes it so easy to convert from bytes to unicode by
> accident without specifying a valid encoding. So I would like to see a future
> where str(bytes) is effectively removed. I started working on a pull request
> that adds an API to toggle str(bytes) at runtime with a thread local (instead
> of requiring a command line argument), so you could do with no_str_bytes(): if
> you were worried about the feature, but got a bit stuck in the internals.

How is this different than all the str -> unicode bugs we had in python2?  If
you have special needs, you can always monkey-patch it in plain python code by
overriding __builtins__.str with something that asserts the given arg is not
bytes.

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J3MLBIRRUSHO7TSTZ54FFYQZBJBVJCMY/

[Python-Dev] Re: python3 -bb and hash collisions

[Matt Billenstein]

On Fri, Sep 13, 2019 at 08:37:26AM +1000, Cameron Simpson wrote:
> On 10Sep2019 10:42, Daniel Holth  wrote:
> [...]
> > I stopped using Python 3 after learning about str(bytes) by finding it
> > in
> > my corrupted database. [...]
> 
> Could you outline how this happened to you?

Not the OP, but I've actually seen something like this happen in postgres, but
it's postgres doing the adaptation of bytea into a text column, not python str
afaict:

>>> conn = psycopg2.connect(...)
>>>
>>> with conn.cursor() as cursor:
...   cursor.execute('update note set notes=%s where id=%s returning notes', 
('hi there', 'NwMVUksheafn'))
...   cursor.fetchall()
...   cursor.execute('update note set notes=%s where id=%s returning notes', 
(b'hi there', 'NwMVUksheafn'))
...   cursor.fetchall()
...
[{'notes': 'hi there'}]
[{'notes': '\\x6869207468657265'}]

We were storing the response of an api request from requests and had grabbed
response.content (bytes) instead of response.text (str).  I was still able to
decode the original data from this bytes representation, so not ideal, but no
data lost.

I did wish this sorta thing had raised an error instead of doing what it did.

m


-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CXFLFI3I3J4OQBAYLHRZSFFDU46Y4EGW/

[Python-Dev] Re: Recent PEP-8 change

[Matt White]

I am imagining the pain and hurt these conversations will cause for 
people who are not as well positioned and not as comfortable in the 
community.


Just want to throw out a counterpoint to the imagined people - as a real 
person who is not well positioned or comfortable in the community. I've 
been lurking on the list trying to wrap my head around the CPython 
process and how I might contribute.


I think it makes sense to remove the S&W standard, making it more open 
and less intimidating. I'm in a cohort that both agrees with the 
purported goal and disagrees with the execution. It's disappointing that 
anyone bringing up any counterpoint has to have a disclaimer so they 
aren't personally attacked, but this seems to be the level of discourse 
around these topics.


I'm not on Twitter for a reason; the commit message is much more suited 
to that environment. The goal, if it truly was for making a more 
inclusive community, has not been accomplished.


There is an effect - intended or not - making those of us who would 
rather focus on solving technical problems than having yet another 
exhausting emotionally charged rhetorical fight feel unwelcome. The 
personal attacks being thrown around are not helping.


If this was a necessary cost of inclusivity - which is the obvious 
fallback argument - it would be easier to believe the claim that this is 
entirely devoted to creating a more open and welcoming community. But 
the goal of opening things up by removing the S&W guideline could've 
been trivially accomplished without generating any of this drama.


It's baffling claim to promote cohesion and throw a partisan diatribe 
into the commit message.



Mmm. Well, we said what we had to say.


I think this captures the real intentions of the PR.

Best,
Matt White
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TMA4GK4JDFKOG2LV6ZBFPSXSYL6BFV6K/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] Forking and Multithreading - enemy brothers

[Matt Knox]

Jesse Noller  gmail.com> writes:

> We already have an implementation that spawns a
> subprocess and then pushes the required state to the child. The
> fundamental need for things to be pickleable *all the time* kinda
> makes it annoying to work with.
> 

just a lurker here... but this topic hits home with me so thought I'd chime
in. I'm a windows user and I would *love* to use multiprocessing a lot more
because *in theory* it solves a lot of the problems I deal with very nicely
(lot sof financial data number crunching). However, the pickling requirement
makes it very very difficult to actually get any reasonably complex code to
work properly with it.

A lot of the time the functions I want to call in the spawned processes are
actually fairly self contained and don't need most of the environment of the
parent process shoved into it, so it's annoying that it fails because some data
I don't even need in the child process can't be pickled.

What about having an option to skip all the parent environment data pickling
and require the user to manually invoke any imports that are needed in the
target functions as the first step inside their target function?

for example...

def target_function(object_from_module_xyz):
import xyz
return object_from_module_xyz.do_something()

and if I forgot to import all the stuff necessary for the arguments being
passed into my function to work, then it's my own problem.

Although maybe there is some obvious problem with this that I am not seeing.

Anyway, just food for thought.

- Matt


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "Missing" 2.5 feature

[Matt Fleming]

On 10/07/06, Alexander Schremmer <[EMAIL PROTECTED]> wrote:
> On Sun, 9 Jul 2006 20:45:05 -0700, Neal Norwitz wrote:
>
> > There hasn't been much positive response (in the original thread or
> > here).  Given you forgot about it for over a year, how important can
> > it be? :-)
>

I'm the SoC student working on the improvements for pdb, one of the
improvements is allowing thread debugging. I was in fact, going to use
the threadframe module if it was available on the system, having this
method in the Python core is an even better solution.

cheering-for-tim-ly yr's,
Matt


-- 
http://mattssanctuary.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] new guy

[matt westerburg]

Hi my name is Matt Westerburg, I am a student and have only recently gotten into Python.  But have fallen in love with the language thus far.  Fantastic language and thank you very much for making it what it is today.  I am looking into getting into working on Python.  Still need sometime working with the language, but I am interested in both the standard library and the interpreter.  Can anyone recommend a good starting place?
thank you
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] r50708 - in python/trunk: Lib/test/test_sys.py Misc/NEWS Python/pystate.c

[Matt Fleming]

On 19/07/06, Thomas Heller <[EMAIL PROTECTED]> wrote:
> Neal Norwitz schrieb:
> > On 7/18/06, Jack Diederich <[EMAIL PROTECTED]> wrote:
> >>
> >> were pre-2003 and talking about mod_python.  HURD and FreeBSD came up a
> >> couple times.  Do we need to add more *BSD buildbots?
> >
> > Yes.  We only have OpenBSD now.  It would be nice to have {Free,Net}BSD 
> > too..
>
> Maybe some of the buildbots should (in addition to the normal build?)
> configure Python with --without-threads?
>

I have an AMD64 NetBSD machine that isn't doing much at the moment, I
can regurlary run tests (I submitted a patch not long back to make
regrtest netbsd-3 aware). However, I can't turn it into a buildbot,
sorry.

Matt

--
http://mattssanctuary.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Improving unit tests for the standard library

[Matt Fleming]

Hi, after speaking with Neal off-list about writing tests for the
pkgutil module, we agreed it would be a good idea to start a page on
http://wiki.python.org/moin/ stating any tests for the standard
library that either,

a) need to be written
b) can be improved

I've started the page http://wiki.python.org/moin/ImprovingLibTests
that lists all the test files I could think of that need to be
written. Ive also included a table for improvements to existing tests,
along with a column that allows you to specify exactly what needs
improving.

I hope this will be of use to people, and I hope people will find time
to modify the page approriately. When I get some spare time from my
SoC project, I'll be working my way through the list.

Thanks, Matt

-- 
http://mattssanctuary.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] uuid test suite failing

[Matt Fleming]

> On 27/07/06, Ronald Oussoren <[EMAIL PROTECTED]> wrote:
> > IIRC at least some versions of HP-UX do not support the -a flag for
> > ifconfig, I'll check this tomorrow.
> >
> > Ronald
> >

  td192> /usr/sbin/ifconfig
  usage: ifconfig interface
 [ af [ address [ dest_addr ] ] [ up ] [ down ][ netmask mask ] ]
 [ metric n ]
 [ arp | -arp ]
 [ plumb | unplumb ]
  td192> /usr/sbin/ifconfig -a
  ifconfig: no such interface
  td192> uname -a
  HP-UX td192 B.11.11 U 9000/800 1839940656 unlimited-user license


Also fixed this test on my NetBSD machine by using 'ifconfig -a' and
checking for 'address:' in the output. But as Ronald said, not all
platforms support the '-a' flag. Not sure if this will fix the OpenBSD
buildbot, I don't have access to an OpenBSD machine.

Matt


-- 
http://mattssanctuary.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is this a bug?

[Matt Fleming]

On 09/08/06, Georg Brandl <[EMAIL PROTECTED]> wrote:
> Is this considered a bug? Sure, deleting modules from sys.modules
> isn't quite common, but it happened to me on one occasion.
>
> Python 2.4.3 (#1, Jul 29 2006, 10:52:20)
>  >>> import logging
>  >>> import sys
>  >>> del logging
>  >>> del sys.modules['logging']
>  >>> ^D
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
>File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
>  func(*targs, **kargs)
>File "/usr/lib/python2.4/logging/__init__.py", line 1328, in shutdown
>  for h in _handlerList[:]: # was _handlers.keys():
> TypeError: unsubscriptable object
> Error in sys.exitfunc:
> Traceback (most recent call last):
>File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
>  func(*targs, **kargs)
>File "/usr/lib/python2.4/logging/__init__.py", line 1328, in shutdown
>  for h in _handlerList[:]: # was _handlers.keys():
> TypeError: unsubscriptable object
>
> Obviously, _handlerList (as a global) is already cleaned up, which is why
> the subscript fails.
>
> Georg
>
> ___


Could it be considered a bug in the atexit module (or is that what you
meant)? Seeing as there's no _decent_ way to recover from this error,
perhaps it could just slip silently passed?

-- 
Matt

http://mattssanctuary.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Fri, Apr 15, 2011 at 05:03:55PM -0700, Bob Ippolito wrote:
> On Fri, Apr 15, 2011 at 4:12 PM, Antoine Pitrou  wrote:
> > On Fri, 15 Apr 2011 14:27:04 -0700
> > Bob Ippolito  wrote:
> >> On Fri, Apr 15, 2011 at 2:20 PM, Antoine Pitrou  
> >> wrote:
> >
> > Well, here's a crude microbenchmark. I'm comparing 2.6+simplejson 2.1.3
> > to 3.3+json, so I'm avoiding integers:
> >
> > * json.dumps:
> >
> > $ python -m timeit -s "from simplejson import dumps, loads; \
> > ?? ??d = dict((str(i), str(i)) for i in range(1000))" \
> > ?? "dumps(d)"
> >
> > - 2.6+simplejson: 372 usec per loop
> > - 3.2+json: 352 usec per loop
> >
> > * json.loads:
> >
> > $ python -m timeit -s "from simplejson import dumps, loads; \
> > ?? ??d = dict((str(i), str(i)) for i in range(1000)); s = dumps(d)" \
> > ?? ??"loads(s)"
> >
> > - 2.6+simplejson: 224 usec per loop
> > - 3.2+json: 233 usec per loop
> >
> >
> > The runtimes look quite similar.
> 
> That's the problem with trivial benchmarks. With more typical data
> (for us, anyway) you should see very different results.

Slightly less crude benchmark showing simplejson is quite a bit faster:

http://pastebin.com/g1WqUPwm

250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Fri, Apr 15, 2011 at 05:03:55PM -0700, Bob Ippolito wrote:
> On Fri, Apr 15, 2011 at 4:12 PM, Antoine Pitrou  wrote:
> > On Fri, 15 Apr 2011 14:27:04 -0700
> > Bob Ippolito  wrote:
> >> On Fri, Apr 15, 2011 at 2:20 PM, Antoine Pitrou  
> >> wrote:
> >
> > Well, here's a crude microbenchmark. I'm comparing 2.6+simplejson 2.1.3
> > to 3.3+json, so I'm avoiding integers:
> >
> > * json.dumps:
> >
> > $ python -m timeit -s "from simplejson import dumps, loads; \
> > ?? ??d = dict((str(i), str(i)) for i in range(1000))" \
> > ?? "dumps(d)"
> >
> > - 2.6+simplejson: 372 usec per loop
> > - 3.2+json: 352 usec per loop
> >
> > * json.loads:
> >
> > $ python -m timeit -s "from simplejson import dumps, loads; \
> > ?? ??d = dict((str(i), str(i)) for i in range(1000)); s = dumps(d)" \
> > ?? ??"loads(s)"
> >
> > - 2.6+simplejson: 224 usec per loop
> > - 3.2+json: 233 usec per loop
> >
> >
> > The runtimes look quite similar.
> 
> That's the problem with trivial benchmarks. With more typical data
> (for us, anyway) you should see very different results.

Slightly less crude benchmark showing simplejson is quite a bit faster:

http://pastebin.com/g1WqUPwm

250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Sat, Apr 16, 2011 at 01:30:13PM +0200, Antoine Pitrou wrote:
> On Sat, 16 Apr 2011 00:41:03 +
> Matt Billenstein  wrote:
> > 
> > Slightly less crude benchmark showing simplejson is quite a bit faster:
> > 
> > http://pastebin.com/g1WqUPwm
> > 
> > 250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...
> 
> This doesn't have much value if you don't say which version of Python
> you ran json with. You should use 3.2, otherwise you might miss some
> optimizations.

Yes, that was 2.6.5 -- 3.2 native json is comparable to simplejson here taking
about 330ms...

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of json (simplejson) in cpython

[Matt Billenstein]

On Sun, Apr 17, 2011 at 08:22:20AM +0200, Stefan Behnel wrote:
> Matt Billenstein, 17.04.2011 00:47:
> >On Sat, Apr 16, 2011 at 01:30:13PM +0200, Antoine Pitrou wrote:
> >>On Sat, 16 Apr 2011 00:41:03 +
> >>Matt Billenstein wrote:
> >>>
> >>>Slightly less crude benchmark showing simplejson is quite a bit faster:
> >>>
> >>>http://pastebin.com/g1WqUPwm
> >>>
> >>>250ms vs 5.5s encoding and decoding an 11KB json object 1000 times...
> >>
> >>This doesn't have much value if you don't say which version of Python
> >>you ran json with. You should use 3.2, otherwise you might miss some
> >>optimizations.
> >
> >Yes, that was 2.6.5 -- 3.2 native json is comparable to simplejson here 
> >taking
> >about 330ms...
> 
> From the POV of CPython 3.2, is "native" Python or C?

"Native" as in the version that ships with 3.2.

And actually I think my test with 2.6.5 wasn't using the C extension for some
reason so that 5.5s number isn't right -- a fresh build of 2.7.1 gives me a
runtime of around 350ms.

m

-- 
Matt Billenstein
m...@vazor.com
http://www.vazor.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] which SSL client protocols work with which server protocols?

[Matt Goodall]

Bill Janssen wrote:
> Here's the updated connection table:
> 
>   SSL2SSL3SS23TLS1
> SSL2  yes no  yes no
> SSL3  yes yes yes no
> SSL23 yes no  yes no
> TLS1  no  no  yes yes
> 
> Given this, I think the client-side default should be changed from
> SSLv23 to SSLv3, and the server-side default should be SSLv23.

I believe you are correct.

I did some experiments with this a while ago after hitting problems
connecting to some SSL servers although I can't remember the exact
results now.

More importantly, what you recommend is what Twisted does and I'd
believe them more than me any time ;-).

See Twisted's DefaultOpenSSLContextFactory [1] for the server side and
ClientContextFactory [2] for the client side.


Cheers, Matt


[1] DefaultOpenSSLContextFactory,
http://twistedmatrix.com/trac/browser/trunk/twisted/internet/ssl.py#L67

[2] ClientContextFactory,
http://twistedmatrix.com/trac/browser/trunk/twisted/internet/ssl.py#L102

-- 
Matt Goodall, Pollenation Internet Ltd
Technology House, 237 Lidgett Lane, Leeds LS17 6QR
Registered No 4382123
A member of the Brunswick MCL Group of Companies
w: http://www.pollenation.net/
e: [EMAIL PROTECTED]
t: +44 113 2252500
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Python source code on Bazaar vcs

[Matt Nordhoff]

[EMAIL PROTECTED] wrote:
> Barry> All the gory details are documented here:
> 
> Barry>  http://www.python.org/dev/bazaar
> 
> Thanks.  I checked out, made a branch named test3, changed Makefile.pre.in
> to have a test3 target, checked it in, then tried to push it:
> 
> % pwd
> /Users/skip/src/python-bzr/test3
> % bzr push bzr+ssh://[EMAIL PROTECTED]/python/users/skip/test3
> bzr: ERROR: Parent directory of bzr+ssh://[EMAIL 
> PROTECTED]/python/users/skip/test3 does not exist.
> You may supply --create-prefix to create all leading parent directories.
> 
> Did I misread the directions or do I really need the --create-prefix arg?
> 
> Skip

I don't know if there's a special setup here, but
 doesn't list a "skip" directory
yet. You'll need to use --create-prefix to get bzr to create it.
-- 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Hi all,

My first post to the list. In fact, first time Python hacker, long-time
Python user though. (Melbourne, Australia).

Some of you may have seen for the past week or so my bug report on Roundup,
http://bugs.python.org/issue3300

I've spent a heap of effort on this patch now so I'd really like to get some
more opinions and have this patch considered for Python 3.0.

Basically, urllib.quote and unquote seem not to have been updated since
Python 2.5, and because of this they implicitly perform Latin-1 encoding and
decoding (with respect to percent-encoded characters). I think they should
default to UTF-8 for a number of reasons, including that's what other
software such as web browsers use.

I've submitted a patch which fixes quote and unquote to use UTF-8 by
default. I also added extra arguments allowing the caller to choose the
encoding (after discussion, there was some consensus that this would be
beneficial). I have now completed updating the documentation, writing
extensive test cases, and testing the rest of the standard library for code
breakage - with the result being there wasn't really any, everything seems
to just work nicely with UTF-8. You can read the sordid details of my
investigation in the tracker.

Firstly, it'd be nice to hear if people think this is desirable behaviour.
Secondly, if it's feasible to get this patch in Python 3.0. (I think if it
were delayed to Python 3.1, the code breakage wouldn't justify it). And
thirdly, if the first two are positive, if anyone would like to review this
patch and check it in.

I have extensively tested it, and am now pretty confident that it won't
cause any grief if it's checked in.

Thanks very much,
Matt Giuca
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Thanks for all the replies, and making me feel welcome :)

>
> If what you are saying is true, then it can probably go in as a bug
> fix (unless someone else knows something about Latin-1 on the Net that
> makes this not true).
>

Well from what I've seen, the only time Latin-1 naturally appears on the net
is when you have a web page in Latin-1 (either explicit or inferred; and
note that a browser like Firefox will infer Latin-1 if it sees only ASCII
characters) with a form in it. Submitting the form, the browser will use
Latin-1 to percent-encode the query string.

So if you write a web app and you don't have any non-ASCII characters or
mention the charset, chances are you'll get Latin-1. But I would argue
you're leaving things to chance and you deserve to get funny behaviour. If
you do any of the following:

   - Use a non-ASCII character, encoded as UTF-8 on the page.
   - Send a Content-Type: ; charset=utf-8.
   - In HTML, set a .
   - In the form itself, set .

then the browser will encode the form data as UTF-8. And most "proper" web
pages should get themselves explicitly served as UTF-8.

That I can't say I can necessarily due; have my own bug reports to
> work through this weekend. =)


OK well I'm busy for the next few days; after that I can do a patch trade
with someone. (That is if I am allowed to do reviews; not sure since I don't
have developer privileges).


On Sun, Jul 13, 2008 at 5:58 AM, Mark Hammond <[EMAIL PROTECTED]>
wrote:

> > My first post to the list. In fact, first time Python hacker,
> > long-time Python user though. (Melbourne, Australia).
>
> Cool - where exactly?  I'm in Wantirna (although not at this very moment -
> I'm in Lithuania, but home again in a couple of days)


Cool :) Balwyn.


> * Please take Martin with a grain of salt ( \I would say "ignore him", but
> that is too strong ;)


Lol, he is a hard man to please, but he's given some good feedback.


On Sun, Jul 13, 2008 at 7:07 AM, Bill Janssen <[EMAIL PROTECTED]> wrote:

>
> The standard here is RFC 3986, from Jan 2005, which says,
>
>  ``When a new URI scheme defines a component that represents textual
> data consisting of characters from the Universal Character Set [UCS],
> the data should first be encoded as octets according to the UTF-8
> character encoding [STD63]; then only those octets that do not
> correspond to characters in the unreserved set should be
> percent-encoded.''


Ah yes, I was originally hung up on the idea that "URLs had to be encoded in
UTF-8", till Martin pointed out that it only says "new URI scheme" there.
It's perfectly valid to have non-UTF-8-encoded URIs. However in practice
they're almost always UTF-8. So I think introducing the new encoding
argument and having it default to "utf-8" is quite reasonable.

I'd say, treat the incoming data as either Unicode (if it's a Unicode
> string), or some unknown superset of ASCII (which includes both
> Latin-1 and UTF-8) if it's a byte-string (and thus in some unknown
> encoding), and apply the appropriate transformation.
>

Ah there may be some confusion here. We're only dealing with str->str
transformations (which in Python 3 means Unicode strings). You can't put a
bytes in or get a bytes out of either of these functions. I suggested a
"quote_raw" and "unquote_raw" function which would let you do this.

The issue is with the percent-encoded characters in the URI string, which
must be interpreted as bytes, not code points. How then do you convert these
into a Unicode string? (Python 2 did not have this problem, since you simply
output a byte string without caring about the encoding).

On Sun, Jul 13, 2008 at 9:10 AM, "Martin v. Löwis" <[EMAIL PROTECTED]>
wrote:

> > Very nice, I had this somewhere on my todo list to work on. I'm very much
> > in favour, especially since it synchronizes us with the RFCs (for all I
> > remember reading about it last time).
>
> I still think that it doesn't. The RFCs haven't changed, and can't
> change for compatibility reasons. The encoding of non-ASCII characters
> in URLs remains as underspecified as it always was.


Correct. But my patch brings us in-line with that unspecification. The
unpatched version forces you to use Latin-1. My patch lets you specify the
encoding to use.


> Now, with IRIs, the situation is different, but I don't think the patch
> claims to implement IRIs (and if so, it perhaps shouldn't change URL
> processing in doing so).


True. I don't claim to have implemented IRIs or even know enough about them
to do that. I'll read up on these things in the next few days.

However, this is a URI library, not IRI. From what I've seen, it

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> This POV is way too browser-centric...
>

This is but one example. Note that I found web forms to be the least
clear-cut example of choosing an encoding. Most of the time applications
seem to be using UTF-8, and all the standards I have read are moving towards
specifying UTF-8 (from being unspecified). I've never seen a standard
specify or even recommend Latin-1.

Where web forms are concerned, basically setting the form accept-charset or
the page charset is the *maximum amount* of control you have over the
encoding. As you say, it can be encoded by another page or the user can
override their settings. Then what can you do as the server? Nothing ...
there's no way to predict the encoding. So you just handle the cases you
have control over.

5) Different cultures do not choose necessarily between latin-1 and utf-8.
> They deal more with things like, say KOI8-R or Big5.


Exactly. This is exactly my point - Latin-1 is arbitrary from a standards
point of view. It's just one of the many legacy encodings we'd like to
forget. The UTFs are the only options which support all languages, and UTF-8
is the only ASCII-compatible (and therefore URI-compatible) encoding. So we
should aim to support that as the default.

Besides all that and without any offense: "most proper" and "should do" and
> the implication that all web browsers behave the same way are not a good
> location to argue from when talking about implementing a standard ;)


I agree. However if there *was* a proper standard we wouldn't have to argue!
"Most proper" and "should do" is the most confident we can be when dealing
with this standard, as there is no correct encoding.

Does anyone have a suggestion which will be more compatible with the rest of
the world than allowing the user to select an encoding, and defaulting to
"utf-8"?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

ncoded string, then encoded octets

Clearly the unquote is str->bytes,  You can't pass a Unicode string
> back
> as the result of unquote *without* passing in an encoding specifier,
> because the character set is application-specific.
>

So for unquote you're suggesting that it always return a bytes object UNLESS
an encoding is specified? As in:

>>> urllib.parse.unquote('h%C3%BCllo')
b'h\xc3\xbcllo'

I would object to that on two grounds. Firstly, I wouldn't expect or desire
a bytes object. The vast majority of uses for unquote will be to get a
character string out, not bytes. Secondly, there is a mountain of code
(including about 12 modules in the standard library) which call unquote and
don't give the user the encoding option, so it's best if we pick a default
that is what the majority of users will expect. I argue that that's UTF-8.

I'd prefer having a separate unquote_raw function which is str->bytes, and
the unquote function performs the same role as it always have, which is
str->str. But I agree on quote, I think it can be (bytes OR str)->str.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] str(container) should call str(item), not repr(item)

[Matt Giuca]

Another disadvantage of calling str recursively rather than repr is that it
places an onus on anyone writing a class to write both a repr and a str
method (or be inconsistent with the newly-accepted standard for container
types).

I personally write a repr method for most classes, which aids debugging.
This means all my classes behave like containers currently do - their str
will call repr on the items. This proposal will make all of my classes
behave inconsistently with the standard container types.

- Matt Giuca
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Hi folks,

This issue got some attention a few weeks back but it seems to have
fallen quiet, and I haven't had a good chance to sit down and reply
again till now.

As I've said before this is a serious issue which will affect a great
deal of code. However it's obviously not as clear-cut as I originally
believed, since there are lots of conflicting opinions. Let us see if
we can come to a consensus.

(For those who haven't seen the discussion, the thread starts here:
http://mail.python.org/pipermail/python-dev/2008-July/081013.html
continues here for some reason:
http://mail.python.org/pipermail/python-dev/2008-July/081066.html
and I've got a bug report with a fully tested and documented patch here:
http://bugs.python.org/issue3300)

Firstly, it looks like most of the people agree we should add an
optional "encoding" argument which lets the caller customize which
encoding to use. What we tend to disagree about is what the default
encoding should be.

Here I present the various options as I see it (and I'm trying to be
impartial), and the people who've indicated support for that option
(apologies if I've misrepresented anybody's opinion, feel free to
correct):

1. Leave it as it is. quote is Latin-1 if range(0,256), fallback to
UTF-8. unquote is Latin-1.
In favour: Anybody who doesn't reply to this thread
Pros: Already implemented; some existing code depends upon ord values
of string being the same as they were for byte strings; possible to
hack around it.
Cons: unquote is not inverse of quote; quote behaviour
internally-inconsistent; garbage when unquoting UTF-8-encoded URIs.

2. Default to UTF-8.
In favour: Matt Giuca, Brett Cannon, Jeroen Ruigrok van der Werven
Pros: Fully working and tested solution is implemented; recommended by
RFC 3986 for all future schemes; recommended by W3C for use with HTML;
UTF-8 used by all major browsers; supports all characters; most
existing code compatible by default; unquote is inverse of quote.
Cons: By default, URIs may have invalid octet sequences (not possible
to reverse).

3. quote default to UTF-8, unquote default to Latin-1.
In favour: André Malo
Pros: quote able to handle all characters; unquote able to handle all sequences.
Cons: unquote is not inverse of quote; totally inconsistent.

4. quote accepts either bytes or str, unquote default to outputting
bytes unless given an encoding argument.
In favour: Bill Janssen
Pros: Technically does what the spec says, which is treat it as an
octet encoding.
Cons: unquote will break most existing code; almost 100% of the time
people will want it as a string.



I'll just comment on #4 since I haven't already. Let's talk about
quote and unquote separately. For quote, I'm all for letting it accept
a bytes as well as a str. That doesn't break anything or surprise
anyone.

For unquote, I think it will break a lot and surprise everyone. I
think that while this may be "purely" the best option, it's pretty
silly. I reckon the vast majority of users will be surprised when they
see it spitting out a bytes object, and all that most people will do
is decode it as UTF-8. Besides, while you're reading the RFCs as "URLs
specify a method for encoding octet sequences", I'm reading them as
"URLs specify a method for encoding strings, and leave the character
encoding unspecified." The second reading supports the idea that
unquote outputs a str.

I'm also recommending we add unquote_to_bytes to do what you suggest
unquote should do. (So either way we'll get both versions of unquote;
I'm just suggesting the one called "unquote" do the thing everybody
expects). But that's less of a priority so I want to commit these
urgent fixes first.

I'm basically saying just two things: 1. The standards are undefined;
2. Therefore we should pick the most useful and/or intuitive default.
IMHO choosing UTF-8 *is* the most useful AND intuitive, and will be
more so in the future when more technologies are hard-coded as UTF-8
(which this RFC recommends they do in the future).

I am also quite adamant that unquote be the inverse of quote.

Are there any more opinions on this matter? It would be good to reach
a consensus. If anyone seriously wants to push a different alternative
to mine, please write a working implementation and attach it to issue
3300.

On the technical side of things, does anybody have time to review my
patch for this issue?
http://bugs.python.org/issue3300
Patch 5.
It's just a patch for unquote, quote, and small related functions, as
well as numerous changes to test cases and documentation.

Cheers
Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Arg! Damnit, why do my replies get split off from the main thread?
Sorry about any confusion this may be causing.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> Con: URI encoding does not encode characters.

OK, for all the people who say URI encoding does not encode characters: yes
it does. This is not an encoding for binary data, it's an encoding for
character data, but it's unspecified how the strings map to octets before
being percent-encoded. From RFC 3986, section
1.2.1<http://tools.ietf.org/html/rfc3986#section-1.2.1>
:

Percent-encoded octets (Section 2.1) may be used within a URI to represent
> characters outside the range of the US-ASCII coded character set if this
> representation is allowed by the scheme or by the protocol element in which
> the URI is referenced.  Such a definition should specify the character
> encoding used to map those characters to octets prior to being
> percent-encoded for the URI.


So the string->string proposal is actually correct behaviour. I'm all in
favour of a bytes->string version as well, just not with the names "quote"
and "unquote".

I'll prepare a new patch shortly which has bytes->string and string->bytes
versions of the functions as well. (quote will accept either type, while
unquote will output a str, there will be a new function unquote_to_bytes
which outputs a bytes - is everyone happy with that?)

Guido says:

> Actually, we'd need to look at the various other APIs in Py3k before we can
> decide whether these should be considered taking or returning bytes or text.
> It looks like all other APIs in the Py3k version of urllib treat URLs as
> text.


Yes, as I said in the bug tracker, I've groveled over the entire stdlib to
see how my patch affects the behaviour of dependent code. Aside from a few
minor bits which assumed octets (and did their own encoding/decoding) (which
I fixed), all the code assumes strings and is very happy to go on assuming
this, as long as the URIs are encoded with UTF-8, which they almost
certainly are.

Guido says:

> I think the only change is to remove the encoding arguments and ...


You really want me to remove the encoding= named argument? And hard-code
UTF-8 into these functions? It seems like we may as well have the optional
encoding argument, as it does no harm and could be of significant benefit.
I'll post a patch with the unquote_to_bytes function, but leave the encoding
arguments in until this point is clarified.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Alright, I've uploaded the new patch which adds the two requested
bytes-oriented functions, as well as accompanying docs and tests.
http://bugs.python.org/issue3300
http://bugs.python.org/file11009/parse.py.patch6

I'd rather have two pairs of functions, so that those who want to give
> the readers of their code a clue can do so. I'm not opposed to having
> redundant functions that accept either string or bytes though, unless
> others prefer not to.
>

Yes, I was in a similar mindset. So the way I've implemented it, quote
accepts either a bytes or a str. Then there's a new function
quote_from_bytes, which is defined precisely like this:

quote_from_bytes = quote
>

So either name can be used on either input type, with the idea being that
you should use quote on a str, and quote_from_bytes on a bytes. Is this a
good idea or should it be rewritten so each function permits only one input
type?

Sorry, I have yet to look at the tracker (only so many minutes in a day...).


Ah, I didn't mean offense. Just that one could read the sordid details of my
investigation on the tracker if one so desired ;)

I don't mind an encoding argument, as long as it isn't used to change
> the return type (as Bill was proposing).


Yeah, my unquote always outputs a str, and unquote_to_bytes always outputs a
bytes.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Bill wrote:

I'm not sure that's sufficient review, though I agree it's necessary.
>
The major consumers of quote/unquote are not in the Python standard
>
library.


I figured that Python 3.0 is designed to fix things, with the breaking
third-party code being an acceptable side-effect of that. So the most
important thing when 3.0 is released is that the stdlib is internally
consistent. All other code is "allowed" to be broken. So I've investigated
all the code necessary.

Having said this, my patch breaks almost no code. Your suggestion breaks a
hell of a lot.

Sure.  All I was asking was that we not break the existing usage of
>
the standard library "unquote" by producing a string by *assuming* a
>
UTF-8 encoded string is what's in those percent-encoded bytes (instead
>
of, say, ISO 2022-JP).  Let the "new" function produce a string:
>
"unquote_as_string".


You're assuming that a Python 2.x "str" is the same thing as a Python 3.0
"bytes". It isn't. (If it was, this transition would be trivial). A Python 2
"str" is a non-Unicode string. It can be printed, concatenated with Unicode
strings, etc etc. It has the semantics of a string. The Python 3.0 "bytes"
is not a string at all.

What you're saying is "the old behaviour was to output a bytes, so the new
behaviour should be consistent". But that isn't true - the old behaviour was
to output a string (a non-Unicode one). People, and code, expect it to
output something with string semantics. So making unquote output a bytes is
just as big a change as making it output a (unicode) str. Python 3.0 doesn't
have a type which is like Python 2's "str" type (which is good - that type
was very messy). So the argument that "Python 2 unquote outputs a bytes, so
we should too" is not legitimate.



If you want to keep pushing this, please install my new patch (patch 6).
Then rename "unquote" to "unquote_to_string" and rename "unquote_to_bytes"
to "unquote", and witness the havoc that ensues. Firstly, you break most
Internet-related modules in the standard library.

10 tests failed:
>
test_SimpleHTTPServer test_cgi test_email test_http_cookiejar
>
test_httpservers test_robotparser test_urllib test_urllib2
>
test_urllib2_localnet test_wsgiref
>

Fixing these isn't a matter of changing test cases (which all but one of my
fixes were). It would require changes to all the modules, to get them to
deal with bytes instead of strings (which would generally mean spraying
.decode("utf-8") all over the place). My code, on the other hand, "tends to
be" compatible with 2.x code.

Here I'm seeing:
BytesWarning: Comparison between bytes and string.
TypeError: expected an object with the buffer interface
http.client.BadStatusLine

For another example, try this:

>>> import http.server
>>> s = http.server.HTTPServer(('',8000),
http.server.SimpleHTTPRequestHandler)
>>> s.serve_forever()

The current (unpatched) build works, but links to files with non-ASCII
filenames (eg. '漢字') break, because of the URL. This is one example of my
patch directly fixing a bug in real code. With my patch applied, the links
work fine *because URL quoting and unquoting are consistent, and work on all
Unicode characters*.

If you change unquote to output a bytes, it breaks completely. You get a
"TypeError: expected an object with the buffer interface" as soon as the
user visits the page.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> so you can use quote_from_bytes on strings?

Yes, currently.

> I assumed Guido meant it was okay to have quote accept string/byte input and 
> have a function that was redundant but limited in what it accepted (i.e. 
> quote_from_bytes accepts only bytes)
>
> I suppose your implementation doesn't break anything... it just strikes me as 
> "odd"

Yeah. I get exactly what you mean. Worse is it takes an
encoding/replace argument.

I'm in two minds about whether it should allow this or not. On one
hand, it kind of goes with the Python philosophy of not artificially
restricting the allowed types. And it avoids redundancy in the code.

But I'd be quite happy to let quote_from_bytes restrict its input to
just bytes, to avoid confusion. It would basically be a
slightly-modified version of quote:

def quote_from_bytes(s, safe = '/'):
if isinstance(safe, str):
safe = safe.encode('ascii', 'ignore')
cachekey = (safe, always_safe)
if not isinstance(s, bytes) or isinstance(s, bytearray):
raise TypeError("quote_from_bytes() expected a bytes")
try:
quoter = _safe_quoters[cachekey]
except KeyError:
quoter = Quoter(safe)
_safe_quoters[cachekey] = quoter
res = map(quoter, s)
return ''.join(res)

(Passes test suite).

I think I'm happier with this option. But the "if not isinstance(s,
bytes) or isinstance(s, bytearray)" is not very nice.
(The only difference to quote besides the missing arguments is the two
lines beginning "if not isinstance". Maybe we can generalise the rest
of the function).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Has anyone had time to look at the patch for this issue? It got a lot of
support about a week ago, but nobody has replied since then, and the patch
still hasn't been assigned to anybody or given a priority.

I hope I've complied with all the patch submission procedures. Please let me
know if there is anything I can do to speed this along.

Also I'd be interested in hearing anyone's opinion on the "quote_from_bytes"
issue as raised in the previous email. I posted a suggested implementation
of a more restrictive quote_from_bytes in that email, but I haven't included
it in the patch yet.

Matt Giuca
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> After the most recent flurry of discussion I've lost track of what's
> the right thing to do. I also believe it was said it should wait until
> 2.7/3.0, so there's no hurry (in fact there's no way to check it -- we
> don't have branches for those versions yet).
>

I assume you mean 2.7/3.1.

I've always been concerned with the suggestion that this wait till 3.1. I
figure this patch is going to change the documented behaviour of these
functions, so it might be unacceptable to change it after 3.0 is released.
It seems logical that this patch be part of the
"incompatible-for-the-sake-of-fixing-things" set of changes in 3.0.

The current behaviour is broken. Any code which uses quote to produce a URL,
then unquotes the same URL later will simply break for characters outside
the Latin-1 range. This is evident in the SimpleHTTPServer class as I said
above (which presents users with URLs for the files in a directory using
quote, then gives 404 when they click on them, because unquote can't handle
it). And it will break any user's code which also assumes unquote is the
inverse of quote.

We could hack a fix into SimpleHTTPServer and expect other users to do the
same (along the lines of .encode('utf-8').decode('latin-1')), but then those
hacks will break when we apply the patch in 3.1 because they abuse Unicode
strings, and we'll have to have another debate about how to be backwards
compatible with them. (The patched version is largely compatible with the
2.x version, but the unpatched version isn't compatible with either the 2.x
version or the patched version).

Surely the sane option is to get this UTF-8 patch into version 3.0 so we
don't have to support this bug into the future? I'm far less concerned about
the decision with regards to unquote_to_bytes/quote_from_bytes, as those are
new features which can wait.

Matt Giuca
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> This whole discussion circles too much, I think. Maybe it should be pepped?


The issue isn't circular. It's been patched and tested, then a whole lot of
people agreed including Guido. Then you and Bill wanted the bytes
functionality back. So I wrote that in there too, and Bill at least said
that was sufficient.

On Thu, Jul 31, 2008, Bill Janssen wrote:
>
> But:  OK, OK, I yield.  Though I still think this is a bad idea, I'll shut
> up if we can also add "unquote_as_bytes" which returns a byte sequence
> instead of a string.  I'll just change my code to use that.
>

We've reached, to quote Guido, "as close as consensus as we can get on this
issue".

There is a bug in Python. I've proposed a working fix, and nobody else has.
Guido okayed it. I made all the changes the community suggested. What more
needs to be discussed here?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

> There are a lot of quotes around. Including "After the most recent flurry
> of
> discussion I've lost track of what's the right thing to do."
> But I don't talk for other people.
>

OK .. let me compose myself a little. Sorry I went ahead and assumed this
was closed.

It's just frustrating to me that I've now spent a month trying to push this
through, and while it seems everybody has an opinion, nobody seems to have
bothered trying my code. (I've even implemented your suggestions and posted
my feedback, and nobody replied to that). Nobody's been assigned to look at
it and it hasn't been given a priority, even though we all agree it's a bug
(though we disagree on how to fix it).


>
> > There is a bug in Python. I've proposed a working fix, and nobody else
> > has.
>
> Well, you proposed a patch ;-)
> It may fix things, it will break a lot. While this was denied over and over
> again, it's still gonna happen, because the axioms are still not accounting
> for the reality.


Well all you're getting from me is "it works". And all I'm getting from you
is "it might not". Please .. I've been asking for weeks now for someone to
review the patch. I've already spent hours (like ... days worth of hours)
testing this patch against the whole library. I've written reams of reports
on the tracker to try and convince people it works. There isn't any more *I*
can do. If you think it's going to break code, go ahead and try it out.

The claims I am making are based on my experience working with a) Python 2,
b) Python 3 as it stands, c) Python 3 with my patch, and d) Python 3 with
quote/unquote using bytes. In my experience, (c) is the only version of
Python 3 which works properly.

> I made all the changes the community suggested.
>
> I don't think so.
>

?


> > What more needs to be discussed here?
>
> Huh? You feel, the discussion is over? Then why are there still open
> questions? I admit, a lot of discussion is triggered by the assessments
> you're stating in your posts. Don't take it as a personal offense, it's a
> simple observation. There were made a lot of statements and nobody even
> bothered to substantiate them.


If you read the bug tracker <http://bugs.python.org/issue3300> all the way
to the beginning, you'll see I use a good many examples, and I also went
through the entire standard library <http://bugs.python.org/msg69591> to try
and substantiate my claims. (Admittedly, I didn't finish investigating the
request module, but that shouldn't prevent the patch from being reviewed).
As I've said all along, yes, it will break code, but then *all solutions
possible* will break code, including leaving it in. Mine *seems* to break
the least existing code. If there is ever a time to break code, Python 3.0
is it.


> A PEP could fix that.
>

I could write a PEP. But as you've read above, I'm concerned this won't get
into Python 3.0, and then we'll be locked into the existing functionality
and it'll never get accepted; hence I'd rather this be resolved as quickly
as possible. If you think it's worth writing a PEP, let's do it.

Apologies again for my antagonistic reply earlier. Not trying to step on
toes, just get stuff done.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] urllib.quote and unquote - Unicode issues

[Matt Giuca]

Wow .. a lot of replies today!

On Thu, Aug 7, 2008 at 2:09 AM, "Martin v. Löwis" <[EMAIL PROTECTED]>wrote:

> It hasn't been given priority: There are currently 606 patches in the
> tracker, many fixing bugs of some sort. It's not clear (to me, at least)
> why this should be given priority over all the other things such as
> interpreter crashes.


Sorry ... when I said "it hasn't been given priority" I mean "it hasn't been
given *a* priority" - as in, nobody's assigned a priority to it, whatever
that priority should rightfully be.


> We all agree it's a bug: no, I don't. I think it's a missing feature,
> at best, but I'm staying out of the discussion. As-is, urllib only
> supports ASCII in URLs, and that is fine for most purposes.


Seriously, Mr. L%C3%B6wis, that's a tremendously na%C3%AFve statement.


> URLs are just not made for non-ASCII characters. Implement IRIs if you
> want non-ASCII characters; the rules are much clearer for these.


Python 3.0 fully supports Unicode. URIs support *encoding* of arbitrary
characters (as of more recent revisions). The difference is that URIs *may
only consist* of ASCII characters (even though they can encode Unicode
characters), while IRIs may also consist of Unicode characters. It's our
responsibility to implement URIs here ... IRIs are a separate issue.

Having said this, I'm pretty sure Martin can't be convinced, so I'll leave
that alone.

On Thu, Aug 7, 2008 at 3:34 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:

> So unquote() should probably try to decode using UTF-8 first
>
and then fall back to Latin-1 if that doesn't work.


That's an interesting proposal. I think I don't like it - for a user
application that's a good policy. But for a programming language library, I
think it should not do guesswork. It should use the encoding supplied, and
have a single default. But I'd be interested to hear if anyone else wants
this.

As-is, it passes 'replace' to the errors argument, so encoding errors get
replaced by '�' characters.

OK I haven't looked at the review yet .. guess it's off to the tracker :)

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] String concatenation

[Matt Giuca]

Is the only issue with this feature that you might accidentally miss a comma
after a string in a sequence of strings? That seems like a significantly
obscure scenario compared to the usefulness of the current syntax, for
exactly the purpose Barry points out (which most people use all the time).

I think the runtime concatenation costs are less important than the
handiness of being able to break strings across lines without having to
figure out where to put that '+' operator.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bytes.tohex in Python 3

[Matt Giuca]

Hi,

I'm confused as to how you represent a bytes object in hexadecimal in Python
3. Of course in Python 2, you use str.encode('hex') to go to hex, and
hexstr.decode('hex') to go from hex.

In Python 3, they removed "hex" as a codec (which was a good move, I think).
Now there's the static method bytes.fromhex(hexbytes) to go from hex. But I
haven't figured out any (easy) way to convert a byte string to hex. Is there
some way I haven't noticed, or is this an oversight?

The easiest thing I can think of currently is this:
''.join(hex(b)[2:] for b in mybytes)

I think there should be a bytes.tohex() method. I'll add this as a bug
report if it indeed is an oversight, but I thought I'd check here first to
make sure I'm not just missing something.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bytes.tohex in Python 3

[Matt Giuca]

Well, whether there's community support for this or not, I thought I'd have
a go at implementing this, so I did. I've submitted a feature request +
working patch to the bug tracker:

http://bugs.python.org/issue3532

Matt

PS. I mean
''.join("%02x" % b for b in mybytes)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bytes.tohex in Python 3

[Matt Giuca]

Hi Guido,

Ah yes Martin just pointed this out on the tracker. I think it would still
be worthwhile having the tohex method, if not just to counter the obscurity
of the binascii.hexlify function (but for other reasons too).

Also there's an issue with all the functions in binascii - they return
bytes, not strings. Is this an oversight? (My version of tohex returns a
str).

See tracker:
http://bugs.python.org/issue3532

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] uuid test fails

[Matt Giuca]

Hi,

I thought I'd bring this up on both the tracker and mailing list, since it's
important. It seems the test suite breaks as of r65661. I've posted details
to the bug tracker and a patch which fixes the module in question (uuid.py).

http://bugs.python.org/issue3552

Cheers
Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ImportError message suggestion

[Matt Giuca]

> ImportError: cannot import name annotate from /usr//image.pyc


That could be handy.

Not sure it's necessary, however, and exposes some system information in the
error message. I can imagine a web app which prints out exception messages,
and this would mean the server file structure is printed to the world
(though arguably you should not be doing this on your web app, I work on an
open source web app and we do dump tracebacks to our users sometimes --
because it's open source we don't mind them seeing the code -- but we'd
rather not have them see our server details).

If you do get this issue (as a developer), I find the built-in help()
function very handy -- you can import a module then go help(that_module) and
it tells you the absolute path to the module. That might be a sufficient
alternative to this patch (though requiring a bit more manual labour).

So I am neither for nor against this suggestion.

"I think the acceptance for this wouldn't be that hard since there is
> no real issue for regression (the only one I could think of is for
> doctest module, although I'm not sure there are any reason to test for
> failed import in doctest)"


I agree. (I'm more familiar with unittest than doctest, where you'd just use
assertRaises(ImportError, ...) and not care what the exception message is --
is there any way in doctest to check for the exception type without caring
about the message?)

I can't write the C code myself, or evaluate the patch.
>

Go to http://bugs.python.org/ and add a new issue. Upload the patch as an
attachment when you enter the issue description. I think you'll have to put
it down as a feature request for 2.7/3.1, since the beta tomorrow will mean
no more features in 2.6/3.0.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ImportError message suggestion

[Matt Giuca]

I think this is not the place to be discussing the patch (the tracker is),
but while I think of it, I'll just say:

You need to DECREF the fn variable (both
PyObject_GetAttrString<http://www.python.org/doc/api/object.html>and
PyString_FromString
<http://www.python.org/doc/api/stringObjects.html>return new
references). If this makes no sense, read up on reference
counting (http://docs.python.org/ext/refcounts.html,
http://www.python.org/doc/api/countingRefs.html).


+PyString_AsString(name),
+ PyString_AsString(fn));
+   Py_DECREF(fn);
}

Also:

   - Do you really want "?" if you can't get the filename for some reason --
   why not just not say anything?
   - Perhaps don't create a new variable "fn", use one of the many defined
   at the top of the eval function.

Otherwise, looks like it will do the job.

But I haven't tested it, just eyeballed it.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Things to Know About Super

[Matt Giuca]

Hi Michele,

Do you have a URL for this blog?

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fwd: Things to Know About Super

[Matt Giuca]

Had a brief offline discussion with Michele - forwarding.

-- Forwarded message --
From: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Date: Mon, Aug 25, 2008 at 12:13 AM

On Aug 24, 3:43 pm, "Matt Giuca" <[EMAIL PROTECTED]> wrote:
> Hi Michele,
>
> Do you have a URL for this blog?

Sorry, here it is:
http://www.artima.com/weblogs/index.jsp?blogger=micheles

-- Forwarded message --
From: Matt Giuca <[EMAIL PROTECTED]>
Date: Mon, Aug 25, 2008 at 1:15 AM

I skimmed (will read in detail later). As an "intermediate" (I'll describe
myself as) Python developer, I tend not to use/understand super (I just call
baseclassname.methodname(self,...) directly, so I guess I'm the target
audience of this article. It's good - very informative and thorough.

It's a bit too informal, personal, and opinionative to be used as
"documentation" IMHO but it could certainly be cleaned up without being
rewritten.

Of interest though, is this:
"The first sentence is just plain wrong: super does not return the
superclass."

>From what I remember of using super, this statement is true, and the
documentation is wrong (or at least over-simplifies things).
http://docs.python.org/dev/library/functions.html#super
http://docs.python.org/dev/3.0/library/functions.html#super
Perhaps this should be amended? (A brief statement to the effect of super
creating a proxy object which can call the methods of any base class). It
actually mentions the "super object" later, even though it claims to be
returning the superclass.

Also Michele, looks as if super in Python 3 works about the same but has the
additional feature of supporting 0 arguments, in which case it defaults to
super(this_class, first_arg). (Does not create unbound super objects).

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation Error for __hash__

[Matt Giuca]

> This may have been true for old style classes, but as new style classes
> inherit a default __hash__ from object - mutable objects *will* be usable as
> dictionary keys (hashed on identity) *unless* they implement a __hash__
> method that raises a type error.
>

I always thought this was a bug in new-style classes (due to the fact that,
as you say, they inherit __hash__ from object whether it's wanted or not).
However, it seems to be fixed in Python 3.0. So this documentation is only
"wrong" for Python 2.x branch.

See:

Python 2.6b3+ (trunk:66055, Aug 29 2008, 07:50:39)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class X(object):
... def __eq__(self, other):
... return True
...
>>> x = X()
>>> hash(x)
-1211564180

versus

Python 3.0b3+ (py3k:66055M, Aug 29 2008, 07:52:23)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class X(object):
... def __eq__(self, other):
...     return True
...
>>> x = X()
>>> hash(x)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unhashable type: 'X'

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation Error for __hash__

[Matt Giuca]

> Being hashable is a different from being usable as dictionary key.
>
> Dictionaries perform the lookup based on the hash value, but will
> then have to check for hash collisions based on an equal comparison.
>
> If an object does not define an equal comparison, then it is not
> usable as dictionary key.
>

But if an object defines *neither* __eq__ or __hash__, then by default it is
usable as a dictionary key (using the id() of the object for both default
equality and hashing, which is fine, and works for all user-defined types by
default).

An object defining __hash__ but not __eq__ is not problematic, since it
still uses id() default for equality. (It just has potentially bad
dictionary performance, if lots of things hash the same even though they
aren't equal). This it not a problem by definition because *it is officially
"okay" for two objects to hash the same, even if they aren't equal, though
undesirable*.

So all hashable objects are usable as dictionary keys, are they not? (As far
as I know it isn't possible to have an object that does not have an equality
comparison, unless you explicitly override __eq__ and have it raise a
TypeError for some reason).

It's probably a good idea to implement __hash__ for objects that
> implement comparisons, but it won't always work and it is certainly
> not needed, unless you intend to use them as dictionary keys.
>

But from what I know, it is a *bad* idea to implement __hash__ for any
mutable object with non-reference equality (where equality depends on the
mutable state), as an unbreakable rule. This is because if they are inserted
into a dictionary, then mutated, they may suddenly be in the wrong bucket.
This is why all the mutable types in Python with non-reference equality (eg.
list, set, dict) are explicitly not hashable, while the immutable types (eg.
tuple, frozenset, str) are hashable, and so are the mutable types with
reference equality (eg. functions, user-defined classes by default).


>
> > and that mutable objects should raise a TypeError in __hash__.
>
> That's a good idea, even though it's not needed either ;-)
>

So I think my above "axioms" are a better (less restrictive, and still
correct) rule than this one. It's OK for a mutable object to define
__hash__, as long as its __eq__ doesn't depend upon its mutable state. For
example, you can insert a function object into a dictionary, and mutate its
closure, and it won't matter, because neither the hash nor the equality of
the object is changing. It's only types like list and dict, with deep
equality, where you run into this hash table problem.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation Error for __hash__

[Matt Giuca]

>
>> It's probably a good idea to implement __hash__ for objects that
>> implement comparisons, but it won't always work and it is certainly
>> not needed, unless you intend to use them as dictionary keys.
>>
>>
>>
>
>
> So you're suggesting that we document something like.
>
> Classes that represent mutable values and define equality methods are free
> to define __hash__ so long as you don't mind them being used incorrectly if
> treated as dictionary keys...
>
> Technically true, but not very helpful in my opinion... :-)


No, I think he was suggesting we document that if a class overrides __eq__,
it's a good idea to also implement __hash__, so it can be used as a
dictionary key.

However I have issues with this. First, he said:

"It's probably a good idea to implement __hash__ for objects that
implement comparisons, but it won't always work and it is certainly
not needed, unless you intend to use them as dictionary keys."

You can't say "certainly not needed unless you intend to use them as
dictionary keys", since if you are defining an object, you never know when
someone else will want to use them as a dict key (or in a set, mind!) So *if
possible*, it is a good idea to implement __hash__ if you are implementing
__eq__.

But also, it needs to be very clear that if you *should not* implement
__hash__ on a mutable object -- and it already is. So basically the docs
should suggest that it is a good idea to implement __hash__ if you are
implementing __eq__ on an immutable object.

HOWEVER,

There are two contradictory pieces of information in the docs.

a) "if it defines
__cmp__()<http://docs.python.org/dev/reference/datamodel.html#object.__cmp__>or
__eq__() <http://docs.python.org/dev/reference/datamodel.html#object.__eq__>but
not
__hash__()<http://docs.python.org/dev/reference/datamodel.html#object.__hash__>,
its instances will not be usable as dictionary keys."
versus
b) "User-defined classes have
__cmp__()<http://docs.python.org/dev/3.0/reference/datamodel.html#object.__cmp__>and
__hash__()<http://docs.python.org/dev/3.0/reference/datamodel.html#object.__hash__>methods
by default; with them, all objects compare unequal and
x.__hash__() returns id(x)."

Note that these statements are somewhat contradictory: if a class has a
__hash__ method by default (as b suggests), then it isn't possible to "not
have a __hash__" (as suggested by a).

In Python 2, statement (a) is true for old-style classes only, while
statement (b) is true for new style classes only. This distinction needs to
be made. (For old-style classes, it isn't the case that it has a __hash__
method by default - rather that the hash() function knows how to deal with
objects without a __hash__ method, by calling id()).

In Python 3, statement (a) is true always, while statement (b) is not (in
fact just the same as old-style classes are in Python 2). So the Python 3
docs can get away with being simpler (without having to handle that weird
case).

I just saw Marc-Andre's new email come in; I'll look at that now.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documentation Error for __hash__

[Matt Giuca]

> Note that only instances have the default hash value id(obj). This
> is not true in general. Most types don't implement the tp_hash
> slot and thus are not hashable. Indeed, mutable types should not
> implement that slot unless they know what they're doing :-)


By "instances" you mean "instances of user-defined classes"?
(I carefully avoid the term "instance" on its own, since I believe that was
phased out when they merged types and classes; it probably still exists in
the C API but shouldn't be mentioned in Python-facing documentation).

But anyway, yes, we should make that distinction.

Sorry, I wasn't clear enough: with "not defining an equal comparison"
> I meant that an equal comparison does not succeed, ie. raises an
> exception or returns Py_NotImplemented (at the C level).


Oh OK. I didn't even realise it was "valid" or "usual" to explicitly block
__eq__ like that.


> Again, the situation is better at the C level, since types
> don't have a default tp_hash implementation, so have to explicitly
> code such a slot in order for hash(obj) to work.


Yes but I gather that this "data model" document we are talking about is not
designed for C authors, but Python authors, so it should be written for the
point of view of people coding only in Python. (Only the "Extending and
Embedding" and the "C API" documents are for C authors).

The documentation should probably say:
>
> "If you implement __cmp__ or
> __eq__ on a class, also implement a __hash__ method (and either
> have it raise an exception or return a valid non-changing hash
> value for the object)."
>

I agree, except maybe not for the Python 3 docs. As long as the behaviour I
am observing is well-defined and not just a side-effect which could go away
-- that is, if you define __eq__/__cmp__ but not __hash__, in a user-defined
class, it raises a TypeError -- then I think it isn't necessary to recommend
implementing a __hash__ method and raising a TypeError. Better just to leave
as-is ("if it defines
__cmp__()<http://docs.python.org/dev/3.0/reference/datamodel.html#object.__cmp__>or
__eq__()<http://docs.python.org/dev/3.0/reference/datamodel.html#object.__eq__>but
not
__hash__()<http://docs.python.org/dev/3.0/reference/datamodel.html#object.__hash__>,
its instances will not be usable as dictionary keys") and clarify the later
statement.


>
> "If you implement __hash__ on classes, you should consider implementing
> __eq__ and/or __cmp__ as well, in order to control how dictionaries use
> your objects."


I don't think I agree with that. I'm not sure why you'd implement __hash__
without __eq__ and/or __cmp__, but it doesn't cause issues so we may as well
not address it.


> In general, it's probably best to always implement both methods
> on classes, even if the application will just use one of them.
>

Well it certainly is for new-style classes in the 2.x branch. I don't think
you should implement __hash__ in Python 3 if you just want a non-hashable
object (since this is the default behaviour anyway).

A lot of my opinion here, though, which doesn't count very much -- so I'm
just making suggestions.

Matt
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] patch for Cookie.py to add support for HttpOnly

[Matt Chisholm]

Eighteen months ago, Arvin Schnell contributed a really
straightforward three-line patch to Cookie.py adding support for the
HttpOnly flag on cookies:

http://bugs.python.org/issue1638033

In the last eighteen months, HttpOnly has become a de-facto extension
to the cookie standard. It is now supported by IE 7, Firefox 3, and
Opera 9.5 (and there's a bug open against WebKit to support it):

http://www.owasp.org/index.php/HTTPOnly

Ruby, Perl, and PHP all support creating HttpOnly cookies now too. 

This article explains why HttpOnly is a good way to make cross-site
scripting (XSS) attacks significantly more difficult:

http://www.codinghorror.com/blog/archives/001167.htmllop

Unfortunately this patch appears to have been ignored for the last
year.

The last thing I want is a delay in the release of 2.6/3.0, but
Antoine Pitrou posted on the bug that it will have to wait for Python
2.7/3.1, because it is a feature request.  If I'm not mistaken, that
means no support for HttpOnly until sometime in 2010.

Do we really have to wait two more years to apply a three-line patch
which will bring Python in line with the industry state of the art and
improve security for Python web applications?  Is there a way that
this could go in to 2.6.1/3.0.1? 

-matt


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

What if you broke up the read and built the final string object up. I
always assumed this is where the real gain was with read_into.
On Nov 25, 2011 5:55 AM, "Eli Bendersky"  wrote:

> On Thu, Nov 24, 2011 at 20:29, Antoine Pitrou  wrote:
>
>> On Thu, 24 Nov 2011 20:15:25 +0200
>> Eli Bendersky  wrote:
>> >
>> > Oops, readinto takes the same time as copying. This is a real shame,
>> > because readinto in conjunction with the buffer interface was supposed
>> to
>> > avoid the redundant copy.
>> >
>> > Is there a real performance regression here, is this a well-known
>> issue, or
>> > am I just missing something obvious?
>>
>> Can you try with latest 3.3 (from the default branch)?
>>
>
> Sure. Updated the default branch just now and built:
>
> $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()'
> 1000 loops, best of 3: 1.14 msec per loop
> $1 -m timeit -s'import fileread_bytearray'
> 'fileread_bytearray.readandcopy()'
> 100 loops, best of 3: 2.78 msec per loop
> $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()'
> 1000 loops, best of 3: 1.6 msec per loop
>
> Strange. Although here, like in python 2, the performance of readinto is
> close to justread and much faster than readandcopy, but justread itself is
> much slower than in 2.7 and 3.2!
>
> Eli
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

Eli,

Example coming shortly, the differences are quite significant.

On Fri, Nov 25, 2011 at 9:41 AM, Eli Bendersky  wrote:
> On Fri, Nov 25, 2011 at 00:02, Matt Joiner  wrote:
>>
>> What if you broke up the read and built the final string object up. I
>> always assumed this is where the real gain was with read_into.
>
> Matt, I'm not sure what you mean by this - can you suggest the code?
>
> Also, I'd be happy to know if anyone else reproduces this as well on other
> machines/OSes.
>
> Eli
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

It's my impression that the readinto method does not fully support the
buffer interface I was expecting. I've never had cause to use it until
now. I've created a question on SO that describes my confusion:

http://stackoverflow.com/q/8263899/149482

Also I saw some comments on "top-posting" am I guilty of this? Gmail
defaults to putting my response above the previous email.

On Fri, Nov 25, 2011 at 11:49 AM, Matt Joiner  wrote:
> Eli,
>
> Example coming shortly, the differences are quite significant.
>
> On Fri, Nov 25, 2011 at 9:41 AM, Eli Bendersky  wrote:
>> On Fri, Nov 25, 2011 at 00:02, Matt Joiner  wrote:
>>>
>>> What if you broke up the read and built the final string object up. I
>>> always assumed this is where the real gain was with read_into.
>>
>> Matt, I'm not sure what you mean by this - can you suggest the code?
>>
>> Also, I'd be happy to know if anyone else reproduces this as well on other
>> machines/OSes.
>>
>> Eli
>>
>>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

On Fri, Nov 25, 2011 at 12:07 PM, Antoine Pitrou  wrote:
> On Fri, 25 Nov 2011 12:02:17 +1100
> Matt Joiner  wrote:
>> It's my impression that the readinto method does not fully support the
>> buffer interface I was expecting. I've never had cause to use it until
>> now. I've created a question on SO that describes my confusion:
>>
>> http://stackoverflow.com/q/8263899/149482
>
> Just use a memoryview and slice it:
>
> b = bytearray(...)
> m = memoryview(b)
> n = f.readinto(m[some_offset:])

Cheers, this seems to be what I wanted. Unfortunately it doesn't
perform noticeably better if I do this.

Eli, the use pattern I was referring to is when you read in chunks,
and and append to a running buffer. Presumably if you know in advance
the size of the data, you can readinto directly to a region of a
bytearray. There by avoiding having to allocate a temporary buffer for
the read, and creating a new buffer containing the running buffer,
plus the new.

Strangely, I find that your readandcopy is faster at this, but not by
much, than readinto. Here's the code, it's a bit explicit, but then so
was the original:

BUFSIZE = 0x1

def justread():
# Just read a file's contents into a string/bytes object
f = open(FILENAME, 'rb')
s = b''
while True:
b = f.read(BUFSIZE)
if not b:
break
s += b

def readandcopy():
# Read a file's contents and copy them into a bytearray.
# An extra copy is done here.
f = open(FILENAME, 'rb')
s = bytearray()
while True:
b = f.read(BUFSIZE)
if not b:
break
s += b

def readinto():
# Read a file's contents directly into a bytearray,
# hopefully employing its buffer interface
f = open(FILENAME, 'rb')
s = bytearray(os.path.getsize(FILENAME))
o = 0
while True:
b = f.readinto(memoryview(s)[o:o+BUFSIZE])
if not b:
break
o += b

And the timings:

$ python3 -O -m timeit 'import fileread_bytearray'
'fileread_bytearray.justread()'
10 loops, best of 3: 298 msec per loop
$ python3 -O -m timeit 'import fileread_bytearray'
'fileread_bytearray.readandcopy()'
100 loops, best of 3: 9.22 msec per loop
$ python3 -O -m timeit 'import fileread_bytearray'
'fileread_bytearray.readinto()'
100 loops, best of 3: 9.31 msec per loop

The file was 10MB. I expected readinto to perform much better than
readandcopy. I expected readandcopy to perform slightly better than
justread. This clearly isn't the case.

>
>> Also I saw some comments on "top-posting" am I guilty of this?

If tehre's a magical option in gmail someone knows about, please tell.

>
> Kind of :)
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

On Fri, Nov 25, 2011 at 5:41 PM, Eli Bendersky  wrote:
>> Eli, the use pattern I was referring to is when you read in chunks,
>> and and append to a running buffer. Presumably if you know in advance
>> the size of the data, you can readinto directly to a region of a
>> bytearray. There by avoiding having to allocate a temporary buffer for
>> the read, and creating a new buffer containing the running buffer,
>> plus the new.
>>
>> Strangely, I find that your readandcopy is faster at this, but not by
>> much, than readinto. Here's the code, it's a bit explicit, but then so
>> was the original:
>>
>> BUFSIZE = 0x1
>>
>> def justread():
>>    # Just read a file's contents into a string/bytes object
>>    f = open(FILENAME, 'rb')
>>    s = b''
>>    while True:
>>        b = f.read(BUFSIZE)
>>        if not b:
>>            break
>>        s += b
>>
>> def readandcopy():
>>    # Read a file's contents and copy them into a bytearray.
>>    # An extra copy is done here.
>>    f = open(FILENAME, 'rb')
>>    s = bytearray()
>>    while True:
>>        b = f.read(BUFSIZE)
>>        if not b:
>>            break
>>        s += b
>>
>> def readinto():
>>    # Read a file's contents directly into a bytearray,
>>    # hopefully employing its buffer interface
>>    f = open(FILENAME, 'rb')
>>    s = bytearray(os.path.getsize(FILENAME))
>>    o = 0
>>    while True:
>>        b = f.readinto(memoryview(s)[o:o+BUFSIZE])
>>        if not b:
>>            break
>>        o += b
>>
>> And the timings:
>>
>> $ python3 -O -m timeit 'import fileread_bytearray'
>> 'fileread_bytearray.justread()'
>> 10 loops, best of 3: 298 msec per loop
>> $ python3 -O -m timeit 'import fileread_bytearray'
>> 'fileread_bytearray.readandcopy()'
>> 100 loops, best of 3: 9.22 msec per loop
>> $ python3 -O -m timeit 'import fileread_bytearray'
>> 'fileread_bytearray.readinto()'
>> 100 loops, best of 3: 9.31 msec per loop
>>
>> The file was 10MB. I expected readinto to perform much better than
>> readandcopy. I expected readandcopy to perform slightly better than
>> justread. This clearly isn't the case.
>>
>
> What is 'python3' on your machine? If it's 3.2, then this is consistent with
> my results. Try it with 3.3 and for a larger file (say ~100MB and up), you
> may see the same speed as on 2.7

It's Python 3.2. I tried it for larger files and got some interesting results.

readinto() for 10MB files, reading 10MB all at once:

readinto/2.7 100 loops, best of 3: 8.6 msec per loop
readinto/3.2 10 loops, best of 3: 29.6 msec per loop
readinto/3.3 100 loops, best of 3: 19.5 msec per loop

With 100KB chunks for the 10MB file (annotated with #):

matt@stanley:~/Desktop$ for f in read bytearray_read readinto; do for
v in 2.7 3.2 3.3; do echo -n "$f/$v "; "python$v" -m timeit -s 'import
readinto' "readinto.$f()"; done; done
read/2.7 100 loops, best of 3: 7.86 msec per loop # this is actually
faster than the 10MB read
read/3.2 10 loops, best of 3: 253 msec per loop # wtf?
read/3.3 10 loops, best of 3: 747 msec per loop # wtf??
bytearray_read/2.7 100 loops, best of 3: 7.9 msec per loop
bytearray_read/3.2 100 loops, best of 3: 7.48 msec per loop
bytearray_read/3.3 100 loops, best of 3: 15.8 msec per loop # wtf?
readinto/2.7 100 loops, best of 3: 8.93 msec per loop
readinto/3.2 100 loops, best of 3: 10.3 msec per loop # suddenly 3.2
is performing well?
readinto/3.3 10 loops, best of 3: 20.4 msec per loop

Here's the code: http://pastebin.com/nUy3kWHQ

>
> Also, why do you think chunked reads are better here than slurping the whole
> file into the bytearray in one go? If you need it wholly in memory anyway,
> why not just issue a single read?

Sometimes it's not available all at once, I do a lot of socket
programming, so this case is of interest to me. As shown above, it's
also faster for python2.7. readinto() should also be significantly
faster for this case, tho it isn't.

>
> Eli
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

You can see in the tests on the largest buffer size tested, 8192, that
the naive "read" actually outperforms readinto(). It's possibly by
extrapolating into significantly larger buffer sizes that readinto()
gets left behind. It's also reasonable to assume that this wasn't
tested thoroughly.

On Fri, Nov 25, 2011 at 9:55 PM, Antoine Pitrou  wrote:
> On Fri, 25 Nov 2011 08:38:48 +0200
> Eli Bendersky  wrote:
>>
>> Just to be clear, there were two separate issues raised here. One is the
>> speed regression of readinto() from 2.7 to 3.2, and the other is the
>> relative slowness of justread() in 3.3
>>
>> Regarding the second, I'm not sure it's an issue because I tried a larger
>> file (100MB and then also 300MB) and the speed of 3.3 is now on par with
>> 3.2 and 2.7
>>
>> However, the original question remains - on the 100MB file also, although
>> in 2.7 readinto is 35% faster than readandcopy(), on 3.2 it's about the
>> same speed (even a few % slower). That said, I now observe with Python 3.3
>> the same speed as with 2.7, including the readinto() speedup - so it
>> appears that the readinto() regression has been solved in 3.3? Any clue
>> about where it happened (i.e. which bug/changeset)?
>
> It would probably be http://hg.python.org/cpython/rev/a1d77c6f4ec1/
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

On Fri, Nov 25, 2011 at 10:04 PM, Antoine Pitrou  wrote:
> On Fri, 25 Nov 2011 20:34:21 +1100
> Matt Joiner  wrote:
>>
>> It's Python 3.2. I tried it for larger files and got some interesting 
>> results.
>>
>> readinto() for 10MB files, reading 10MB all at once:
>>
>> readinto/2.7 100 loops, best of 3: 8.6 msec per loop
>> readinto/3.2 10 loops, best of 3: 29.6 msec per loop
>> readinto/3.3 100 loops, best of 3: 19.5 msec per loop
>>
>> With 100KB chunks for the 10MB file (annotated with #):
>>
>> matt@stanley:~/Desktop$ for f in read bytearray_read readinto; do for
>> v in 2.7 3.2 3.3; do echo -n "$f/$v "; "python$v" -m timeit -s 'import
>> readinto' "readinto.$f()"; done; done
>> read/2.7 100 loops, best of 3: 7.86 msec per loop # this is actually
>> faster than the 10MB read
>> read/3.2 10 loops, best of 3: 253 msec per loop # wtf?
>> read/3.3 10 loops, best of 3: 747 msec per loop # wtf??
>
> No "wtf" here, the read() loop is quadratic since you're building a
> new, larger, bytes object every iteration.  Python 2 has a fragile
> optimization for concatenation of strings, which can avoid the
> quadratic behaviour on some systems (depends on realloc() being fast).

Is there any way to bring back that optimization? a 30 to 100x slow
down on probably one of the most common operations... string
contatenation, is very noticeable. In python3.3, this is representing
a 0.7s stall building a 10MB string. Python 2.7 did this in 0.007s.

>
>> readinto/2.7 100 loops, best of 3: 8.93 msec per loop
>> readinto/3.2 100 loops, best of 3: 10.3 msec per loop # suddenly 3.2
>> is performing well?
>> readinto/3.3 10 loops, best of 3: 20.4 msec per loop
>
> What if you allocate the bytearray outside of the timed function?

This change makes readinto() faster for 100K chunks than the other 2
methods and clears the differences between the versions.
readinto/2.7 100 loops, best of 3: 6.54 msec per loop
readinto/3.2 100 loops, best of 3: 7.64 msec per loop
readinto/3.3 100 loops, best of 3: 7.39 msec per loop

Updated test code: http://pastebin.com/8cEYG3BD

>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>

So as I think Eli suggested, the readinto() performance issue goes
away with large enough reads, I'd put down the differences to some
unrelated language changes.

However the performance drop on read(): Python 3.2 is 30x slower than
2.7, and 3.3 is 100x slower than 2.7.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Matt Joiner]

I was under the impression this is already in 3.3?

On Nov 25, 2011 10:58 PM, "Eli Bendersky"  wrote:
>
>
>> > However, the original question remains - on the 100MB file also,
although
>> > in 2.7 readinto is 35% faster than readandcopy(), on 3.2 it's about the
>> > same speed (even a few % slower). That said, I now observe with Python
3.3
>> > the same speed as with 2.7, including the readinto() speedup - so it
>> > appears that the readinto() regression has been solved in 3.3? Any clue
>> > about where it happened (i.e. which bug/changeset)?
>>
>> It would probably be http://hg.python.org/cpython/rev/a1d77c6f4ec1/
>
>
> Great, thanks. This is an important change, definitely something to wait
for in 3.3
> Eli
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecation policy

[Matt Joiner]

On Mon, Nov 28, 2011 at 11:14 PM, Steven D'Aprano  wrote:
> Xavier Morel wrote:
>
>> Not being too eager to kill APIs is good, but giving rise to this kind of
>> living-dead APIs is no better in my opinion, even more so since Python has
>> lost one of the few tools it had to manage them (as DeprecationWarning was
>> silenced by default). Both choices are harmful to users, but in the long
>> run I do think zombie APIs are worse.
>
> I would much rather have my code relying on "zombie" APIs and keep working,
> than to have that code suddenly stop working when the zombie is removed.
> Working code should stay working. Unless the zombie is actively harmful,
> what's the big deal if there is a newer, better way of doing something? If
> it works, and if it's fast enough, why force people to "fix" it?
>
> It is a good thing that code or tutorials from Python 1.5 still (mostly)
> work, even when there are newer, better ways of doing something. I see a lot
> of newbies, and the frustration they suffer when they accidentally
> (carelessly) try following 2.x instructions in Python3, or vice versa, is
> great. It's bad enough (probably unavoidable) that this happens during a
> major transition like 2 to 3, without it also happening during minor
> releases.
>
> Unless there is a good reason to actively remove an API, it should stay as
> long as possible. "I don't like this and it should go" is not a good reason,
> nor is "but there's a better way you should use". When in doubt, please
> don't break people's code.

This is a great argument. But people want to see new, bigger better
things in the standard library, and the #1 reason cited against this
is "we already have too much". I think that's where the issue lies:
Either lots of cool nice stuff is added and supported (we all want our
favourite things in the standard lib for this reason), and or the old
stuff lingers...

I'm sure a while ago there was mention of a "staging" area for
inclusion in the standard library. This attracts interest,
stabilization, and quality from potential modules for inclusion.
Better yet, the existing standard library ownership is somehow
detached from the CPython core, so that changes enabling easier
customization to fit other implementations (jpython, pypy etc.) are
possible.

tl;dr old stuff blocks new hotness. make room or separate standard
library concerns from cpython

>
>
>
> --
> Steven
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecation policy

[Matt Joiner]

I like this article on it:

http://semver.org/

The following snippets being relevant here:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards
compatible functionality is introduced to the public API. It MUST be
incremented if any public API functionality is marked as deprecated.

Major version X (X.y.z | X > 0) MUST be incremented if any backwards
incompatible changes are introduced to the public API.

With the exception of actually dropping stuff (however this only
occurs in terms of modules, which hardly count in special cases?),
Python already conforms to this standard very well.

On Wed, Nov 30, 2011 at 11:00 AM, Benjamin Peterson  wrote:
> 2011/11/29 Nick Coghlan :
>> On Wed, Nov 30, 2011 at 1:13 AM, Barry Warsaw  wrote:
>>> On Nov 29, 2011, at 01:59 PM, Antoine Pitrou wrote:
>>>
Well, that's why I think the version number components are not
correctly named. I don't think any of the 2.x or 3.x releases can be
called "minor" by any stretch of the word. A quick glance at
http://docs.python.org/dev/whatsnew/index.html should be enough.
>>>
>>> Agreed, but it's too late to change it.  I look at it as the attributes of 
>>> the
>>> namedtuple being evocative of the traditional names for the digit positions,
>>> not the assignment of those positions to Python's semantics.
>>
>> Hmm, I wonder about that. Perhaps we could add a second set of names
>> in parallel with the "major.minor.micro" names:
>> "series.feature.maint".
>>
>> That would, after all, reflect what is actually said in practice:
>> - release series: 2.x, 3.x  (usually used in a form like "In the 3.x
>> series, X is true. In 2.x, Y is true)
>> - feature release: 2.7, 3.2, etc
>> - maintenance release: 2.7.2, 3.2.1, etc
>>
>> I know I tend to call feature releases major releases and I'm far from
>> alone in that. The discrepancy in relation to sys.version_info is
>> confusing, but we can't make 'major' refer to a different field
>> without breaking existing programs. But we *can* change:
>>
> sys.version_info
>> sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0)
>>
>> to instead read:
>>
>> sys.version_info(series=2, feature=7, maint=2, releaselevel='final', 
>> serial=0)
>>
>> while allowing 'major' as an alias of 'series', 'minor' as an alias of
>> 'feature' and 'micro' as an alias of 'maint'. Nothing breaks, and we'd
>> have started down the path towards coherent terminology for the three
>> fields in the version numbers (by accepting that 'major' has now
>> become irredeemably ambiguous in the context of CPython releases).
>>
>> This idea of renaming all three fields has come up before, but I
>> believe we got stuck on the question of what to call the first number
>> (i.e. the one I'm calling the "series" here).
>
> Can we drop this now? Too much effort for very little benefit. We call
> releases what we call releases.
>
>
>
> --
> Regards,
> Benjamin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] LZMA support has landed

[Matt Joiner]

Congrats, this is an excellent feature.

On Wed, Nov 30, 2011 at 10:34 AM, Amaury Forgeot d'Arc
 wrote:
> 2011/11/29 Nadeem Vawda 
>>
>> I'm pleased to announce that as of changeset 74d182cf0187, the
>> standard library now includes support for the LZMA compression
>> algorithm
>
>
> Congratulations!
>
>>
>> I'd like to ask the owners of (non-Windows) buildbots to install the
>> XZ Utils development headers so that they can build the new module.
>
>
> And don't worry about Windows builbots, they will automatically download
> the XZ prebuilt binaries from the usual place.
> (svn export http://svn.python.org/projects/external/xz-5.0.3)
>
> Next step: add support for tar.xz files (issue5689)...
>
> --
> Amaury Forgeot d'Arc
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] STM and python

[Matt Joiner]

Given GCC's announcement that Intel's STM will be an extension for C
and C++ in GCC 4.7, what does this mean for Python, and the GIL?

I've seen efforts made to make STM available as a context, and for use
in user code. I've also read about the "old attempts way back" that
attempted to use finer grain locking. The understandably failed due to
the heavy costs involved in both the locking mechanisms used, and the
overhead of a reference counting garbage collection system.

However given advances in locking and garbage collection in the last
decade, what attempts have been made recently to try these new ideas
out? In particular, how unlikely is it that all the thread safe
primitives, global contexts, and reference counting functions be made
__transaction_atomic, and magical parallelism performance boosts
ensue?

I'm aware that C89, platforms without STM/GCC, and single threaded
performance are concerns. Please ignore these for the sake of
discussion about possibilities.

http://gcc.gnu.org/wiki/TransactionalMemory
http://linux.die.net/man/4/futex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] STM and python

[Matt Joiner]

I did see this, I'm not convinced it's only relevant to PyPy.

On Thu, Dec 1, 2011 at 2:25 AM, Benjamin Peterson  wrote:
> 2011/11/30 Matt Joiner :
>> Given GCC's announcement that Intel's STM will be an extension for C
>> and C++ in GCC 4.7, what does this mean for Python, and the GIL?
>>
>> I've seen efforts made to make STM available as a context, and for use
>> in user code. I've also read about the "old attempts way back" that
>> attempted to use finer grain locking. The understandably failed due to
>> the heavy costs involved in both the locking mechanisms used, and the
>> overhead of a reference counting garbage collection system.
>>
>> However given advances in locking and garbage collection in the last
>> decade, what attempts have been made recently to try these new ideas
>> out? In particular, how unlikely is it that all the thread safe
>> primitives, global contexts, and reference counting functions be made
>> __transaction_atomic, and magical parallelism performance boosts
>> ensue?
>
> Have you seen 
> http://morepypy.blogspot.com/2011/08/we-need-software-transactional-memory.html
> ?
>
>
> --
> Regards,
> Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] STM and python

[Matt Joiner]

I saw this, I believe it just exposes an STM primitive to user code.
It doesn't make use of STM for Python internals.

Explicit STM doesn't seem particularly useful for a language that
doesn't expose raw memory in its normal usage.

On Thu, Dec 1, 2011 at 4:41 PM, Nick Coghlan  wrote:
> On Thu, Dec 1, 2011 at 10:58 AM, Gregory P. Smith  wrote:
>> Azul has been using hardware transactional memory on their custom CPUs (and
>> likely STM in their current x86 virtual machine based products) to great
>> effect for their massively parallel Java VM (700+ cpu cores and gobs of ram)
>> for over 4 years.  I'll leave it to the reader to do the relevant searching
>> to read more on that.
>>
>> My point is: This is up to any given Python VM implementation to take
>> advantage of or not as it sees fit.  Shoe horning it into an existing VM may
>> not make much sense but anyone is welcome to try.
>
> There's a patch somewhere on the tracker to add an "Armin Rigo hook"
> to the CPython eval loop so he can play with STM in Python as well (at
> least, I think it was STM he wanted it for - it might have been
> something else).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] STM and python

[Matt Joiner]

Armin, thanks for weighing in on this. I'm keen to see a CPython
making use of STM, maybe I'll give it a try over Christmas break. I'm
willing to take the single threaded performance hit, as I have several
applications that degrade due to significant contention with the GIL.

The other benefits of STM you describe make it a lot more appealing. I
actually tried out Haskell recently to make use of many of the
advanced features but came crawling back.

If anyone else is keen to try this, I'm happy to receive patches for
testing and review.

On Thu, Dec 1, 2011 at 10:01 PM, Armin Rigo  wrote:
> Hi,
>
> On Thu, Dec 1, 2011 at 07:06, Matt Joiner  wrote:
>> I saw this, I believe it just exposes an STM primitive to user code.
>> It doesn't make use of STM for Python internals.
>
> That's correct.
>
>> Explicit STM doesn't seem particularly useful for a language that
>> doesn't expose raw memory in its normal usage.
>
> In my opinion, that sentence could not be more wrong.
>
> It is true that, as I discuss on the blog post cited a few times in
> this thread, the first goal I see is to use STM to replace the GIL as
> an internal way of keeping the state of the interpreter consistent.
> This could quite possibly be achieved using the new GCC
> __transaction_atomic keyword, although I see already annoying issues
> (e.g. the keyword can only protect a _syntactically nested_ piece of
> code as a transaction).
>
> However there is another aspect: user-exposed STM, which I didn't
> explore much.  While it is potentially even more important, it is a
> language design question, so I'm happy to delegate it to python-dev.
> In my opinion, explicit STM (like Clojure) is not only *a* way to
> write multithreaded Python programs, but it seems to be *the only* way
> that really makes sense in general, for more than small examples and
> more than examples where other hacks are enough (see
> http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations
> ).  In other words, locks are low-level and should not be used in a
> high-level language, like direct memory accesses, just because it
> forces the programmer to think about increasingly complicated
> situations.
>
> And of course there is the background idea that TM might be available
> in hardware someday.  My own guess is that it will occur, and I bet
> that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM.
> On such hardware, the performance penalty mostly disappears (which is
> also, I guess, the reasoning behind GCC 4.7, offering a future path to
> use Hybrid TM).
>
> If python-dev people are interested in exploring the language design
> space in that direction, I would be most happy to look in more detail
> at GCC 4.7.  If we manage to make use of it, then we could get a
> version of CPython using STM internally with a very minimal patch.  If
> it seems useful we can then turn that patch into #ifdefs into the
> normal CPython.  It would of course be off by default because of the
> performance hit; still, it would give an optional alternate
> "CPythonSTM" to play with in order to come up with good user-level
> abstractions.  (This is what I'm already trying to do with PyPy
> without using GCC 4.7, and it's progressing nicely.)  (My existing
> patch to CPython emulating user-level STM with the GIL is not really
> satisfying, also for the reason that it cannot emulate some other
> potentially useful user constructs, like abort_and_retry().)
>
>
> A bientôt,
>
> Armin.



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] STM and python

[Matt Joiner]

This is very interesting, cheers for the link.

On Tue, Dec 6, 2011 at 8:55 PM, Armin Rigo  wrote:
> Hi,
>
> Actually, not even one month ago, Intel announced that its processors
> will offer Hardware Transactional Memory in 2013:
>
> http://www.h-online.com/newsticker/news/item/Processor-Whispers-About-Haskell-and-Haswell-1389507.html
>
> So yes, obviously, it's going to happen.
>
>
> A bientôt,
>
> Armin.



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] readd u'' literal support in 3.3?

[Matt Joiner]

Nobody is using 3 yet ;)

Sure, I use it for some personal projects, and other people pretend to
support it. Not really.

The worst of the pain in porting to Python 3000 has yet to even begin!

On Thu, Dec 8, 2011 at 6:33 PM, Nick Coghlan  wrote:
> Such code still won't work on 3.2, hence restoring the redundant notation
> would be ultimately pointless.
>
> --
> Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
>
> On Dec 8, 2011 4:34 PM, "Chris McDonough"  wrote:
>>
>> On Thu, 2011-12-08 at 01:18 -0500, Benjamin Peterson wrote:
>> > 2011/12/8 Chris McDonough :
>> > > On Thu, 2011-12-08 at 01:02 -0500, Benjamin Peterson wrote:
>> > >> 2011/12/8 Chris McDonough :
>> > >> > On the heels of Armin's blog post about the troubles of making the
>> > >> > same
>> > >> > codebase run on both Python 2 and Python 3, I have a concrete
>> > >> > suggestion.
>> > >> >
>> > >> > It would help a lot for code that straddles both Py2 and Py3 to be
>> > >> > able
>> > >> > to make use of u'' literals.
>> > >>
>> > >> Helpful or not helpful, I think that ship has sailed. The earliest it
>> > >> could see the light of day is 3.3, which would leave people trying to
>> > >> support 3.1 and 3.2 in a bind.
>> > >
>> > > Right.. the title does say "readd ... support in 3.3".  Are you
>> > > suggesting "the ship has sailed" for eternity because it can't be
>> > > supported in Python < 3.3?
>> >
>> > I'm questioning the real utility of it.
>>
>> All I can really offer is my own experience here based on writing code
>> that needs to straddle Python 2.5, 2.6, 2.7 and 3.2 without use of 2to3.
>> Having u'' work across all of these would mean porting would not require
>> as much eyeballing as code modified via "from future import
>> unicode_literals", it would let more code work on 2.5 unchanged, and the
>> resulting code would execute faster than code that required us to use a
>> u() function.
>>
>> What's the case against?
>>
>> - C
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing the XML batteries

[Matt Joiner]

+1

On Sat, Dec 10, 2011 at 2:09 AM, Dirkjan Ochtman  wrote:
> On Fri, Dec 9, 2011 at 09:02, Stefan Behnel  wrote:
>> a) The stdlib documentation should help users to choose the right tool right
>> from the start.
>> b) cElementTree should finally loose it's "special" status as a separate
>> library and disappear as an accelerator module behind ElementTree.
>
> An at least somewhat informed +1 from me. The ElementTree API is a
> very good way to deal with XML from Python, and it deserves to be
> promoted over the included alternatives.
>
> Let's deprecate the NiCad batteries and try to guide users toward the
> Li-Ion ones.
>
> Cheers,
>
> Dirkjan
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PATCH] Adding braces to __future__

[Matt Joiner]

If braces were introduced I would switch to Haskell, I can't stand the
noise. If you want to see a language that allows both whitespace, semi
colons and braces take a look at it. Nails it.
On Dec 10, 2011 9:31 AM, "Cedric Sodhi"  wrote:

> On Fri, Dec 09, 2011 at 02:21:42PM -0800, Guido van Rossum wrote:
> >On Fri, Dec 9, 2011 at 12:26 PM, Cedric Sodhi <[1]man...@gmx.net>
> wrote:
> >
> >  IF YOU THINK YOU MUST REPLY SOMETHING WITTY, ITERATE THAT THIS HAD
> BEEN
> >  DISCUSSED BEFORE, REPLY THAT "IT'S SIMPLY NOT GO'NNA HAPPEN", THAT
> "WHO
> >  DOESN'T LIKE IT IS FREE TO CHOOSE ANOTHER LANGUAGE" OR SOMETHING
> >  SIMILAR, JUST DON'T.
> >
> >Every single response in this thread so far has ignored this request.
> The
> >correct response honoring this should have been deafening silence.
> >
> >For me, if I had to design a new language today, I would probably use
> >braces, not because they're better than whitespace, but because pretty
> >much every other lanugage uses them, and there are more interesting
> >concepts to distinguish a new language. That said, I don't regret that
> >Python uses indentation, and the rest I have to say about the topic
> would
> >violate the above request.
> >
>
> I think this deserves a reply. Thank you for contributing your opinion
> and respecting my request and therefore honoring the rules of a
> civilized debate.
>
> -- Cedric
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing the XML batteries

[Matt Joiner]

I second this. The doco is very bad.
On Dec 10, 2011 6:34 AM, "Bill Janssen"  wrote:

> Xavier Morel  wrote:
>
> > On 2011-12-09, at 19:15 , Bill Janssen wrote:
> > > I use ElementTree for parsing valid XML, but minidom for producing it.
> > Could you expand on your reasons to use minidom for producing XML?
>
> Inertia, I guess.  I tried that first, and it seems to work.
>
> I tend to use html5lib and/or BeautifulSoup instead of ElementTree, and
> that's mainly because I find the documentation for ElementTree is
> confusing and partial and inconsistent.  Having various undated but
> obsolete tutorials and documentation still up on effbot.org doesn't
> help.
>
>
> Bill
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PATCH] Adding braces to __future__

[Matt Joiner]

Ditch the colon too. Also you're a troll.
On Dec 10, 2011 9:58 AM, "Cedric Sodhi"  wrote:

> I reply to your contribution mainly because I see another, valid
> argument hidden in what you formulated as an opinion:
>
> Readability would be reduced by such "noise". To anticipate other people
> agreeing with that, let me say, that it would be exactly one more
> character, and the same amount of key presses. All that, assuming you
> use editor automatisms, which particularly the advocates of WSB tend to
> bring forth in defense of WSB and aforementioned problems associated
> with it.
>
> Only one more character and not more key presses? Yes, instead of
> opening a block with a colon, you open it with an opening bracket. And
> you close it with a closing one.
>
> Referring to "noise", I take it you are preferring naturally expressed
> languages (what Roff's PIC, for example, exemplifies to banality).
>
> How is a COLON, which, in natural language, PUNCTUATES a context, any
> more suited than braces, which naturally ENCLOSE a structure?
>
> Obviously, it by far is not, even from the standpoint of not
> intersparsing readable code with unnatural characters.
>
> On Sat, Dec 10, 2011 at 09:40:54AM +1100, Matt Joiner wrote:
> >If braces were introduced I would switch to Haskell, I can't stand the
> >noise. If you want to see a language that allows both whitespace, semi
> >colons and braces take a look at it. Nails it.
> >
> >On Dec 10, 2011 9:31 AM, "Cedric Sodhi" <[1]man...@gmx.net> wrote:
> >
> >  On Fri, Dec 09, 2011 at 02:21:42PM -0800, Guido van Rossum wrote:
> >  >On Fri, Dec 9, 2011 at 12:26 PM, Cedric Sodhi
> >  <[1][2]man...@gmx.net> wrote:
> >  >
> >  >  IF YOU THINK YOU MUST REPLY SOMETHING WITTY, ITERATE THAT
> THIS
> >  HAD BEEN
> >  >  DISCUSSED BEFORE, REPLY THAT "IT'S SIMPLY NOT GO'NNA HAPPEN",
> >  THAT "WHO
> >  >  DOESN'T LIKE IT IS FREE TO CHOOSE ANOTHER LANGUAGE" OR
> SOMETHING
> >  >  SIMILAR, JUST DON'T.
> >  >
> >  >Every single response in this thread so far has ignored this
> >  request. The
> >  >correct response honoring this should have been deafening
> silence.
> >  >
> >  >For me, if I had to design a new language today, I would
> probably
> >  use
> >  >braces, not because they're better than whitespace, but because
> >  pretty
> >  >much every other lanugage uses them, and there are more
> interesting
> >  >concepts to distinguish a new language. That said, I don't
> regret
> >  that
> >  >Python uses indentation, and the rest I have to say about the
> topic
> >  would
> >  >violate the above request.
> >  >
> >
> >  I think this deserves a reply. Thank you for contributing your
> opinion
> >  and respecting my request and therefore honoring the rules of a
> >  civilized debate.
> >
> >  -- Cedric
> >  ___
> >  Python-Dev mailing list
> >  [3]Python-Dev@python.org
> >  [4]http://mail.python.org/mailman/listinfo/python-dev
> >  Unsubscribe:
> >  [5]
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
> >
> > References
> >
> >Visible links
> >1. mailto:man...@gmx.net
> >2. mailto:man...@gmx.net
> >3. mailto:Python-Dev@python.org
> >4. http://mail.python.org/mailman/listinfo/python-dev
> >5.
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Potential NULL pointer dereference in descrobject.c

[Matt Joiner]

ಠ_ಠ

On Sat, Dec 17, 2011 at 8:55 PM, Michael Mueller
 wrote:
> Hi Guys,
>
> We've been analyzing CPython with our static analysis tool (Sentry)
> and a NULL pointer dereference popped up the other day, in
> Objects/descrobject.c:
>
>    if (descr != NULL) {
>        Py_XINCREF(type);
>        descr->d_type = type;
>        descr->d_name = PyUnicode_InternFromString(name);
>        if (descr->d_name == NULL) {
>            Py_DECREF(descr);
>            descr = NULL;
>        }
>        descr->d_qualname = NULL; // Possible NULL pointer dereference
>    }
>
> If the inner conditional block can be reached, descr will be set NULL
> and then dereferenced on the next line.  The commented line above was
> added in this commit: http://hg.python.org/cpython/rev/73948#l4.92
>
> Hopefully someone can take a look and determine the appropriate fix.
>
> Best,
> Mike
>
> --
> Mike Mueller
> Phone: (401) 405-1525
> Email: mmuel...@vigilantsw.com
>
> http://www.vigilantsw.com/
> Static Analysis for C and C++
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Anyone still using Python 2.5?

[Matt Joiner]

I'm paid to write Python3. I've also been writing Python3 for hobby
projects since mid 2010. I'm on the verge of going back to 2.7 due to
compatibility issues :(

On Thu, Dec 22, 2011 at 1:45 PM, Mike Meyer  wrote:
> On Thu, 22 Dec 2011 01:49:37 +
> Michael Foord  wrote:
>> These figures can't possibly be true. No-one is using Python 3 yet. ;-)
>
> Since you brought it up. Is anyone paying people (or trying to hire
> people) to write Python 3?
>
>        Thanks,
>         --
> Mike Meyer               http://www.mired.org/
> Independent Software developer/SCM consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 7 clarification request: braces

[Matt Joiner]

FWIW I'm against forcing braces to be used. Readability is the highest
concern, and this should be at the discretion of the contributor. A
code formatting tool, or compiler extension is the only proper handle
this, and neither are in use or available.

On Tue, Jan 3, 2012 at 7:44 PM, "Martin v. Löwis"  wrote:
>> He keeps leaving them out, I occasionally tell him they should always
>> be included (most recently this came up when we gave conflicting
>> advice to a patch contributor). He says what he's doing is OK, because
>> he doesn't consider the example in PEP 7 as explicitly disallowing it,
>> I think it's a recipe for future maintenance hassles when someone adds
>> a second statement to one of the clauses but doesn't add the braces.
>> (The only time I consider it reasonable to leave out the braces is for
>> one liner if statements, where there's no else clause at all)
>
> While this appears to be settled, I'd like to add that I sided with
> Benjamin here all along.
>
> With Python, I accepted a style of "minimal punctuation". Examples
> of extra punctuation are:
> - parens around expression in Python's if (and while):
>
>    if (x < 10):
>      foo ()
>
> - parens around return expression (C and Python)
>
>    return(*p);
>
> - braces around single-statement blocks in C
>
> In all these cases, punctuation can be left out without changing
> the meaning of the program.
>
> I personally think that a policy requiring braces would be (mildly)
> harmful, as it decreases readability of the code. When I read code,
> I read every character: not just the identifiers, but also every
> punctuation character. If there is extra punctuation, I stop and wonder
> what the motivation for the punctuation is - is there any hidden
> meaning that required the author to put the punctuation?
>
> There is a single case where I can accept extra punctuation in C:
> to make the operator precedence explicit. Many people (including
> myself) don't know how
>
>   a | b << *c * *d
>
> would group, so I readily accept extra parens as a clarification.
>
> Wrt. braces, I don't share the concern that there is a risk of
> somebody being confused when adding a second statement to a braceless
> block. An actual risk is stuff like
>
>   if (cond)
>     MACRO(argument);
>
> when MACRO expands to multiple statements. However, we should
> accept that this is a bug in MACRO (which should have used the
> do-while(0)-idiom), not in the application of the macro.
>
> Regards,
> Martin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] usefulness of Python version of threading.RLock

[Matt Joiner]

I'm pretty sure the Python version of RLock is in use in several
alternative implementations that provide an alternative _thread.lock. I
think gevent would fall into this camp, as well as a personal project of
mine in a similar vein that operates on python3.

2012/1/6 Charles-François Natali 

> Hi,
>
> Issue #13697 (http://bugs.python.org/issue13697) deals with a problem
> with the Python version of threading.RLock (a signal handler which
> tries to acquire the same RLock is called right at the wrong time)
> which doesn't affect the C version.
> Whether such a use case can be considered good practise or the best
> way to fix this is not settled yet, but the question that arose to me
> is: "why do we have both a C and Python version?".
> Here's Antoine answer (he suggested to me to bring this up on python-dev":
> """
> The C version is quite recent, and there's a school of thought that we
> should always provide fallback Python implementations.
> (also, arguably a Python implementation makes things easier to
> prototype, although I don't think it's the case for an RLock)
> """
>
> So, what do you guys think?
> Would it be okay to nuke the Python version?
> Do you have more details on this "school of thought"?
>
> Also, while we're at it, Victor created #13550 to try to rewrite the
> "logging hack" of the threading module: there again, I think we could
> just remove this logging altogether. What do you think?
>
> Cheers,
>
> cf
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
>



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] usefulness of Python version of threading.RLock

[Matt Joiner]

_PyRLock is not used directly. Instead, no _CRLock is provided, so the
threading.RLock function calls _PyRLock.

It's done this way because green threading libraries may only provide a
greened lock. _CRLock in these contexts would not work: It would block the
entire native thread.

I suspect that if you removed _PyRLock, these implementations would have to
expose their own RLock primitive which works the same way as the one just
removed from the standard library. I don't know if this is a good thing.

I would recommend checking with at least the gevent and eventlet developers.

2012/1/7 Charles-François Natali 

> Thanks for those precisions, but I must admit it doesn't help me much...
> Can we drop it? A yes/no answer will do it ;-)
>
> > I'm pretty sure the Python version of RLock is in use in several
> alternative
> > implementations that provide an alternative _thread.lock. I think gevent
> > would fall into this camp, as well as a personal project of mine in a
> > similar vein that operates on python3.
>
> Sorry, I'm not sure I understand. Do those projects use _PyRLock directly?
> If yes, then aliasing it to _CRLock should do the trick, no?
>



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] usefulness of Python version of threading.RLock

[Matt Joiner]

Nick did you mean to say "wrap python code around a reentrant lock to
create a non-reentrant lock"? Isn't that what PyRLock is doing?

FWIW having now read issues 13697 and 13550, I'm +1 for dropping Python
RLock, and all the logging machinery in threading.

2012/1/8 Nick Coghlan 

> 2012/1/7 Charles-François Natali :
> > Thanks for those precisions, but I must admit it doesn't help me much...
> > Can we drop it? A yes/no answer will do it ;-)
>
> The yes/no answer is "No, we can't drop it".
>
> Even though CPython no longer uses the Python version of RLock in
> normal operation, it's still the reference implementation for everyone
> else that has to perform the same task (i.e. wrap Python code around a
> non-reentrant lock to create a reentrant one).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python C API: Problem sending tuple to a method of a python Class

[Matt Joiner]

Perhaps the python-dev mailing list should be renamed to python-core.

On Tue, Jan 10, 2012 at 7:35 PM, Stefan Behnel  wrote:
> Hi,
>
> sorry for hooking into this off-topic thread.
>
> Amaury Forgeot d'Arc, 09.01.2012 19:09:
>> 2012/1/9 
>>> I am trying to send a tuple to a method of a python class and I got a Run
>>> failed from netbeans compiler
>>> when I want to send a tuple to a simple method in a module it works,when I
>>> want to send a simple parameter to a method of a clas it works also but not
>>> a tuple to a method of a class
>>
>> This mailing list is for the development *of* python.
>> For development *with* python, please ask your questions on
>> the comp.lang.python group or the python-l...@python.org mailing list.
>> There you will find friendly people willing to help.
>
> It's also worth mentioning the cython-users mailing list here, in case the
> OP cares about simplifying these kinds of issues from the complexity of
> C/C++ into Python. Cython is a really good and simple way to implement
> these kinds of language interactions, also for embedding Python.
>
>
>> [for your particular question: keep in mind that PyObject_Call takes
>> arguments as a tuple;
>> if you want to pass one tuple, you need to build a 1-tuple around your
>> tuple]
>
> The presented code also requires a whole lot of fixes (specifically in the
> error handling parts) that Cython would basically just handle for you already.
>
> Stefan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com



-- 
ಠ_ಠ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] devguide: Backporting is obsolete. Add details that I had to learn.

[Matt Joiner]

http://semver.org/

This has made sense since Gentoo days.

On Tue, Jan 10, 2012 at 11:57 PM, Antoine Pitrou  wrote:
> On Tue, 10 Jan 2012 08:49:04 +
> Rob Cliffe  wrote:
>> But "minor version" and "major version" are readily understandable to
>> the general reader, e.g. me, whereas "feature release" and "release
>> series" I find are not.  Couldn't the first two terms be defined once
>> and then used throughout?
>
> To me "minor" is a bugfix release, e.g. 2.7.2, and "major" is a feature
> release, e.g. 3.3.  I have a hard time considering 3.2 or 3.3 "minor".
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python C API: Problem sending tuple to a method of a python Class

[Matt Joiner]

I suspect it actually would fix the confusion. "dev" usually means
development, not "core implementation development". People float past
looking for dev help... python-dev. Python-list is a bit generic.

On Tue, Jan 10, 2012 at 11:17 PM, Stefan Behnel  wrote:
> Matt Joiner, 10.01.2012 09:40:
>> Perhaps the python-dev mailing list should be renamed to python-core.
>
> Well, there *is* a rather visible warning on the list subscription page
> that tells people that it's most likely not the list they actually want to
> use. If they manage to ignore that, I doubt that a different list name
> would fix it for them.
>
> Stefan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com