Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github

[Glyph]

> On Nov 30, 2014, at 11:17, Chris Angelico  wrote:
> 
> On Sun, Nov 30, 2014 at 8:54 PM, Nick Coghlan  wrote:
>> On 30 November 2014 at 15:23, Chris Angelico  wrote:
>>> Python is already using quite a bit of non-free software in its
>>> ecosystem. The Windows builds of CPython are made with Microsoft's
>>> compiler, and the recent discussion about shifting to Cygwin or MinGW
>>> basically boiled down to "but it ought to be free software", and that
>>> was considered not a sufficiently strong argument. In each case, the
>>> decision has impact on other people (using MSVC for the official
>>> python.org installers means extension writers need to use MSVC too;
>>> and using GitHub means that contributors are strongly encouraged,
>>> possibly required, to use GitHub); so why is it acceptable to use a
>>> non-free compiler, but not acceptable to use a non-free host?
>> 
>> Relying on non-free software to support users of a non-free platform
>> is rather different from *requiring* the use of non-free software to
>> participate in core Python community design processes.
> 
> But what non-free software is required to use the community design
> processes? The GitHub client is entirely optional; I don't use it, I
> just use git itself. Using a free client to access a proprietary
> server isn't the same as using non-free software.

Also keep in mind that unless you are using a very esoteric hardware setup and 
dedicated leased lines that you purchased yourself, you are likely to be using 
copyrighted, patented, proprietary software at a number of levels:

the microcode implementation in your CPU
the firmware in your GPU
the firmware in your network card
the boot code (e.g.: BIOS or EFI implementation) of your motherboard
the firmware in your router
or the firmware in your cable or DSL modem, if you thought to get a free router 
with OpenWRT or something in it
the firmware in your ISP's router
the firmware in the backbone's routers
the firmware in the PSF's ISP's routers

Does this sound like ridiculous nitpicking?  Of course it does!  If you refused 
to use all that stuff you just wouldn't be able to access the internet at all, 
regardless of your personal choices.  Most layers of this stack are _less_ 
constrained to open standards and open data access than Github, and can require 
layers and layers of additional proprietary software or reverse engineering 
(ask anyone who has tried to play a video game on a Linux computer what the 
experience is like without gobs of proprietary blobs from nvidia or ATI).

And as the story of BitKeeper shows, if a proprietary platform decides to do 
something bad, if the cost of migration is within your means, you can just 
leave.  This is far more true of Github than Bitkeeper - Linux had to create a 
totally new VCS to migrate off of that, we just have to install Trac or Roundup 
or something again. (Which, as per the presently-proposed PEP, we don't even 
have to install them again, just change some configuration to put a few 
repositories back.)

The monoculture about Github concerns me.  I also have concerns about the 
long-term consequences of not having an all-free-software stack.  But focusing 
on avoiding services like Github at this point in history is just a gigantic 
waste of time; it's resolving dependencies in the wrong order.

The only reason to avoid Github is ideological purity, and even then it's not 
even "purity" because you still have to accept this other ideological 
contamination.  Except, what about other ideological concepts that are 
important to the Python core developers, like equitable representation for 
women and minorities, and compassionate and polite interactions within the 
community?  Clearly we can't require the use of Linux.  If we treat these 
ideals as an equivalent priority as free software (even if we ignore the 
"invisible" dependencies like the list I just made above), then there is 
literally no software left that we can use to develop Python. Linux kernel and 
GNU low-level user-land development are a perpetual cesspit, and their leaders 
serve as a continuous embarrassment to our community.

And speaking of equitable representation, one proven technique we have learned 
for dealing with that problem is making it for newcomers of all stripes to 
access the community.  Like it or not, Github's popularity means that it's a 
place where most newcomers to programming are learning to use version control 
and bug tracking.  This is a network effect that can have a real impact on 
people's lives.  Requiring newcomers to learn our weird technology religion 
before they can contribute creates a real barrier to entry, which in turn makes 
our community more ins

Re: [Python-Dev] Yearly PyPI breakage

[Glyph]

On May 3, 2016, at 2:38 PM, Stefan Krah  wrote:
> 
> But making them completely unreachable does not increase reliability. :)

But it does increase security.

The other motivation, besides reliability, listed in this section 
<https://www.python.org/dev/peps/pep-0470/#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that>,
 is that:

"transparently including external links [is] a security hazard (given that in 
most cases it allowed a MITM to execute arbitrary Python code on the end users 
machine)".

And, indeed, the URL presently listed on PyPI for the cdecimal upload is an 
unverified http URL.  This means that any evil barista with access to a 
coffee-shop wifi router could instantly execute user-privileged code on any 
Python programmer's laptop if they were to `pip install´ this externally hosted 
package, which is one of the reasons why neither `pip´ nor `pypi´ allow such a 
thing any more.

Please believe me when I say I do not mean the following to be insulting - 
information security is incredibly confusing, difficult, and rapidly evolving, 
and I don't blame you for getting it wrong - but maintaining a popular package 
in this way is dangerously irresponsible.  There are solid social reasons to 
centralize the control of the default package repository in the hands of 
dedicated experts who can scale their security expertise to a large audience, 
so that package authors like you and I don't need to do this in order to 
prevent Python from gaining a reputation as a vector for malware; this package 
is a case in point.

Separately from the issue of how PyPI works, even if you have some reason you 
need to host it externally (which I seriously doubt), please take the trouble 
to set up a server with properly verified TLS, or use a '.github.io' hostname 
that can be verified that way.

In the meanwhile, just to demonstrate that it's a trivial amount of work to 
just host it on PyPI, I checked out this package via a verified mechanism ("git 
clone https://github.com/bytereef/bytereef.github.io";) and created a new 
pypi-cdecimal package <https://pypi.python.org/pypi/pypi-cdecimal 
<https://pypi.python.org/pypi/pypi-cdecimal>>, via editing the setup.py to 
change the name, 'python setup.py register', 'python setup.py sdist', 'pip 
wheel' (for some reason direct 'python setup.py bdist_wheel' didn't work), and 
'twine upload'.  `pip install pypi-cdecimal´ should now work and get you an 
importable `cdecimal´, and if you happen to be lucky enough to run the same OS 
version I am, you won't even need to build C code.  cdecimal users may wish to 
retrieve it via this mechanism until there's a secure way to get the proper 
upstream distribution.

If anyone wants package-index access to this name to upload Windows or 
manylinux wheels just let me know; however, as this is just a proof of concept, 
I do not intend to maintain it long-term.

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Yearly PyPI breakage

[Glyph]

On May 3, 2016, at 9:15 PM, Stefan Krah  wrote:
> 
>> [cut overlong post]
> 
> Glyph,
> 
> nice sneaky way to try to divert from the original issue.

The original issue, as I understood it, at the start of the thread was "which 
hoops I have to jump through this year in order to keep pip downloads working". 
 So I showed you the hoops.

Your question implied, to me, that you were not presently aware of how easy it 
is to simply build and upload your packages with sdist and twine.  After years 
and years of horrible setuptools bugs, it certainly seemed plausible to me that 
if you had long-standing experience with python packaging, you might have 
perhaps developed a well-deserved low opinion of them in the past.  Therefore, 
you might not be aware of relatively recent improvements to the packaging 
ecosystem which made this problem trivial to solve.

My intent was, therefore, simply to demonstrate that things have improved, and 
that this was not a hard thing for you to do and could be resolved with a 
minimum of fuss.

I confess that before posting I was made aware that you'd had some personality 
conflicts with some PyPI maintainers in the past.  But that sentence was about 
the extent and detail level of my understanding.  I was not aware to what 
extent, and the reason I jumped into this particular thread, when I rarely 
participate in python-dev, was that I hoped a simple explanation of the facts 
of the matter from someone you hadn't previously interacted with could address 
your concerns.

> Your whole post is invalidated by the simple fact that the URL was protected 
> by a hash (which I repeatedly asked to be upgraded to sha256).

Based only on previous discussion here, I had no way to know either of those 
things.  You didn't reference it in the post I was replying to, or in your 
original post.  And, as you say later, PyPI's download URL doesn't include the 
hash any more, so it wasn't there for me to observe.  (There were some manual 
instructions in your package description but no automated tooling will honor 
that.)  In any case, fragment hashes are not really a suitable general-purpose 
mechanism as they are only honored by specific tools (like pip) whereas HTTPS 
verification ought to be universally supported, so IMHO it is a good thing that 
PyPI is discouraging their use for this purpose.

> This was the official scheme promoted by PEP-438, which you should know.  But 
> of course your actual intention here is character assassination, pretending 
> to "rescue" cdecimal

In the "overlong" post that you elided, I specifically said I didn't intend to 
maintain it for long. If this wasn't clear, what I meant to say by that comment 
was that I would keep the index entry available until you had the opportunity 
to upload some sdists and wheels yourself to PyPI.  If you don't intend to, I 
am not the right person to "rescue" the package; someone else who is more 
invested in cdecimal should provide an alternate PyPI entry, or take over this 
one.

> and trying to divert from the fact that
> the transition to PEP 470 was handled suboptimally.

I don't see any need to divert attention from this fact, because you appear to 
be in a minority of one in considering it so.

> The very reason for this thread is that the security was silently disabled 
> WITHOUT me getting a notification.  What is on PyPI *now* is not what I 
> configured!

If that was the reason for the thread, you would have been better served by 
making that specific complaint rather than asking for information, and then 
yelling at the people who provided it to you.  You might also consider 
reporting these issues to an appropriate forum, since python-dev is not the 
bugtracker for PyPI.  You can find that here: 
<https://bitbucket.org/pypa/pypi/issues 
<https://bitbucket.org/pypa/pypi/issues>>.  You might also want to continue 
this thread on distutils-sig; I'm sorry for contributing to the noise on 
python-dev, but I thought getting a high-profile package such as cdecimal 
integrated into the modern packaging ecosystem would be worth the off-topic 
digression.

> [various spurious and irrelevant ad-hominem attacks redacted]


Perhaps naively, given the level of hostility on display here, I still hope 
that you might see the wisdom in simply uploading build artifacts to PyPI.  But 
I won't try to convince you further.

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: What is a public API?

[Glyph]

> On Jul 13, 2019, at 1:56 PM, Serhiy Storchaka  wrote:
> 
> I thought that the name in a module is in the public interface if:
> 
> * It doesn't start with an underscore and the module does not have __all__.
> * It is included in the module's __all__ list.
> * It is explicitly documented as a part of the public interface.
> 
> help() uses more complex rules, but it believes __all__ if it defined.
> 
> But seems there are different views on this.
> 
> * Raymond suggested to add an underscore the two dozens of names in the 
> calendar module not included in __all__.
> https://bugs.python.org/issue28292#msg347758
> 
> I do not like this idea, because it looks like a code churn and makes the 
> code less readable.
> 
> * Gregory suggests to document codecs.escape_decode() despite it is not 
> included in __all__.
> https://bugs.python.org/issue30588
> 
> I do not like this idea, because this function always was internal, its only 
> purpose was implementing the "string-escape" codec which was removed in 
> Python 3 (for reasons). In Python 3 it is only used for supporting the old 
> pickle protocol 0.
> 
> Could we strictly define what is considered a public module interface in 
> Python?

My apologies for not having read this very large thread before posting, but 
hopefully this small note won't be adding too much fuel to the fire:

Earlier this year I created an extremely small project called "publication" 
(https://pypi.org/project/publication/ <https://pypi.org/project/publication/>, 
https://github.com/glyph/publication <https://github.com/glyph/publication>) 
which attempts to harmonize the lofty ideal of "only the names explicitly 
mentioned in __all__ and in the documentation!" with the realpolitik of 
"anything I can reach where I don't have to type a leading underscore is fair 
game".  It simply removes the ability to externally invoke non-__all__-exported 
names without importing an explicitly named "._private" namespace.  It does not 
add any new syntactic idiom like a @public decorator (despite the aesthetic 
benefits of doing something like that) so that existing IDEs, type checkers, 
refactoring tools, code browsers etc can use the existing __all__ idiom and not 
break.  It intentionally doesn't try hard to hide the implementation; it's 
still Python and if you demonstrate that you know what you're doing you're 
welcome to all the fiddly internals, it just makes sure you know that that's 
what you're getting.

While I am perhaps infamously a stdlib contrarian ;-) this project is a single 
module with extremely straightforward, stable semantics which I would 
definitely not mind being copy/pasted into the stdlib wholesale, either under a 
different (private, experimental) name, or even under its current one if folks 
like it.  I'd be very pleased if this could solve the issue for the calendar 
module.

Thanks for reading,

-glyph___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O3BKNPBV5AW7TS2C5NLPEI2TZGJOKDGP/

Re: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as

[Glyph]

On Feb 25, 2013, at 12:32 PM, Barry Warsaw  wrote:

>> Dumb question, but are flufl.enums ordered?  That's also an important use
>> case.
> 
> Kind of.  Ordered comparisons are explicitly not supported, but iteration over
> the Enum is guaranteed to be returned in int-value order.

Sorry to jump in to a random leaf of this thread, but there is such a barrage 
here I cannot find the beginning :).

I can see in  that 
Twisted is mentioned; it should probably reference 
 
and  
since we actually implemented a thing as well.

(You can order constants by sorting them; off the top of my head, 
NamedConstant, ValueConstant, and FlagConstant all probably behave differently.)

-g___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as

[Glyph]

On Feb 26, 2013, at 5:25 AM, Eli Bendersky  wrote:

> Glyph, thanks for the input. I mentioned Twisted because in its code I found 
> a number of places with simple string enumerations used to represent state. I 
> was not aware of twisted.python.constants, but it doesn't appear that this 
> module is used  at least in the places I checked.

Quite so.  twisted.python.constants was created because we made the same 
observation that you did.  Hopefully, more of these protocols will be 
transitioned to make use of twisted.python.constants internally.

> In general, many protocols have some "state" instance var that's usually just 
> a string, using either predefined constants or direct string literals.

Indeed.  This idiom varies considerably.  Another thing that Twisted needs is a 
mechanism for explicitly building state-machine, but that's a discussion for 
another day.

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)

[Glyph]

On Mar 4, 2013, at 11:13 PM, Robert Collins  wrote:

> In principle maybe. Need to talk with the trial developers, nose
> developers, py.test developers etc - to get consensus on a number of
> internal API friction points.

Some of trial's lessons might be also useful for the stdlib going forward, 
given the hope of doing some event-loop stuff in the core.

But, I feel like this might be too much to cover at the language summit; there 
could be a test frameworks summit of its own, of about equivalent time and 
scope, and we'd still have a lot to discuss.

Is there a unit testing SIG someone from Twisted ought to be a member of, to 
represent Trial, and to get consensus on these points going forward?

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About issue 6560

[Glyph]

On Mar 14, 2013, at 3:48 PM, Martin v. Löwis  wrote:

> Am 14.03.13 15:15, schrieb Ani Sinha:
>> I was looking into a mechanism to get the aux fields from recvmsg() in
>> python and I came across this issue. Looks like this feature was added
>> in python 3.3. Is there any reason why this feature was not added for
>> python 2.7?
> 
> Most certainly: Python 2.7 (and thus Python 2) is feature-frozen; no
> new features can be added to it. People wanting new features need to
> port to Python 3.


Or you can use Twisted: 
<http://twistedmatrix.com/trac/browser/trunk/twisted/python/sendmsg.c>

That module ought to have no dependencies outside of Twisted.

We only use it for passing file descriptors between processes, but I believe it 
should be able to deal with whatever other types of auxiliary data that you 
need from recvmsg; if not, please file a bug (at <http://twistedmatrix.com/>).

-glyph

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Simple IDLE issues to commit before Python 2.7.4 release in two weeks on 4/6/2013

[Glyph]

On Mar 25, 2013, at 1:40 PM, Benjamin Peterson  wrote:

> ... Assuming PEP 343 becomes policy ...

Are you sure you got this PEP number right?  The 'with' statement?



-glyph___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] yield * (Re: Missing operator.call)

[glyph]

On 01:00 am, greg.ew...@canterbury.ac.nz wrote:

Guido van Rossum wrote:

We already have yield expressions and they mean something else...


They don't have a "*" in them, though, and I don't
think the existing meaning of yield as an expression
would carry over into the "yield *" variant, so there
shouldn't be any conflict.

But if you think there will be a conflict, or that the
similarity would be too confusing, maybe the new
construct should be called something else. I'm
open to suggestions.


I'm *already* regretting poking my head into this particular bike shed, 
but...


has anyone considered the syntax 'yield from iterable'?  i.e.

   def foo():
   yield 1
   yield 2

   def bar():
   yield from foo()
   yield from foo()

list(bar()) -> [1, 2, 1, 2]

I suggest this because (1) it's already what I say when I see the 'for' 
construct, i.e. "foo then *yield*s all results *from* bar", and (2) no 
new keywords are required.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Choosing a best practice solution for Python/extension modules

[glyph]

On 07:07 pm, br...@python.org wrote:
On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone 
wrote:


But there is another issue with this: the pure Python code will never 
call
the extension code because the globals will be bound to _pypickle and 
not

_pickle. So if you have something like::



 # _pypickle
 def A(): return _B()
 def _B(): return -13



 # _pickle
 def _B(): return 42



 # pickle
 from _pypickle import *
 try: from _pickle import *
 except ImportError: pass



This is really the same as any other high-level/low-level
library split.  It doesn't matter that in this case, one
low-level implementation is provided as an extension module.
Importing the low-level APIs from another module and then
using them to implement high-level APIs is a pretty common,
simple, well-understood technique which is quite applicable
here.


But that doesn't provide a clear way, short of screwing with 
sys.modules, to
get at just the pure Python implementation for testing when the 
extensions

are also present. The key point in trying to figure this out is to
facilitate testing since the standard library already uses the import *
trick in a couple of places.


You don't have to screw with sys.modules.  The way I would deal with 
testing this particular interaction would be a setUp that replaces 
pickle._A with _pypickle._A, and a tearDown that restores the original 
one.


Twisted's TestCase has specific support for this.  You would spell it 
like this:


   import _pypickle
   # ...
   testCase.patch(pickle, '_A', _pypickle._A)

You can read more about this method here:

http://python.net/crew/mwh/apidocs/twisted.trial.unittest.TestCase.html#patch
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 08:46 pm, gu...@python.org wrote:

This seems to be the crux of the problem with asyncore, ever since it
was added to the stdlib -- there's no real API, so every change
potentially breaks something. I wish we could start over with a proper
design under a new name.


Might I suggest "reactor"... or possibly "twisted", as that new name? 
;-)


(Sorry, I was trying to avoid this thread, but that was an opening I 
could drive a truck through).


In all seriousness, I seem to recall that Thomas Wouters was interested 
in doing integrating some portion of Twisted core into the standard 
library as of last PyCon.  I mention him specifically by name in the 
hopes that it will jog his memory.


At the very least, this might serve as a basis for an abstract API for 
asyncore:


http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.interfaces.IProtocol.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 3 Mar, 10:20 pm, gu...@python.org wrote:

On Tue, Mar 3, 2009 at 1:17 PM,   wrote:



At the very least, this might serve as a basis for an abstract API for
asyncore:

http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.interfaces.IProtocol.html



I hope we have learned from asyncore that a future-proof and maximally
flexible design of such an API is not easy. This is one of those cases
where I believe thinking hard up front will really pay off. In this
case I would recommend going through a full-blown PEP process before
committing to an API, to avoid making the same mistake twice. (I don't
mean this as a dig at Twisted, just as an observation about this
particular design problem. If Twisted solves all the problems already,
writing the PEP should be easy.)


I didn't see it as a dig at Twisted; you're right, this is a hard 
problem.  Twisted doesn't solve all the problems perfectly, either; much 
has been made of some rather unfortunate nuances of the IProducer and 
IConsumer interfaces.


I also regret that no Twisted people (myself included) seem to have the 
time to go through the PEP process and get something accepted.


Even if Twisted's interfaces aren't perfect, there's a lot of working 
application code which demonstrates that they are sufficient.  Perhaps 
more importantly, they are future-proof.  Every proposal we've come up 
with for addressing the more subtle shortcomings of these APIs has been 
completely backwards-compatible; there are no APIs with vague or broken 
semantics, just sub-optimal spellings.  The fact that there are already 
multiple reactor implementations which share very little code helps.


If someone who has PEP-writing skills and some free time is updating 
asyncore to be more modern, I'd be very happy to walk that person 
through Twisted's API design and explain how either portions of our 
implementation could be lifted for the stdlib, or how a "high level" 
layer could be written into asyncore so that the author of a particular 
chunk of low-level networking code could ignore whether they're using 
the stdlib mainloop or one of the Twisted ones.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 06:51 pm, exar...@divmod.com wrote:
On Wed, 4 Mar 2009 10:46:28 -0800, Guido van Rossum  
wrote:
On Wed, Mar 4, 2009 at 10:27 AM, Jean-Paul Calderone 
 wrote:



So, as a disinterested party in this specific case, I'd say revert to
the pre-2.6 behavior.



Sorry, but I really do think that we should maintain backward
compatibility *within* the 2.6 series as well.



But why?  The argument I made had the objective of minimizing developer
effort.  What's the objective of maintaining backward compatibility 
within
the 2.6 series in this case (sorry if it appeared earlier in this 
thread

and I missed it)?


Python's compatibility policy dictates that compatibility between micro- 
version releases be maintained *extremely* carefully.


As you all well know, normally I'd be strongly in favor of adhering to 
such a policy.  However, the ostensible purpose of the policy here is 
that users can *always* install micro-version upgrades to get bugfixes 
and security fixes, without worrying about compatibility upgrades.


Maintaining compatibility with the 2.6.x version of asyncore presupposes 
that *someone* has written some software against that version of 
asyncore and it might break if they installed an upgrade, though.  If 
that's the case - if there's even one person who has written or run any 
asyncore software against the version in 2.6 - then I think maintaining 
bug-for-bug compatibility is important.


However, my guess - and I'm assuming that JP was thinking the same thing 
- is that literally nobody has done that, or even *would* ever do that, 
so there's no software anywhere in the world that would break.  asyncore 
is relatively unpopular (thanks in part to the excellent alternatives 
:-)); its major users are the same people who have already complained.


Maybe it's too late to do something like this for 2.6, or even 3.0, but 
I've thought on a few occasions that projects (both Python and Twisted, 
at least) could use a special very-low-traffic policy-deviations mailing 
list for "We *REALLY* don't think this is going to ever break anything 
anyone has actually done, but just to be sure..." notifications for 
situations like this.  The implications being that (A) _everyone_ who 
uses the software should subscribe, and (B) if anyone ever responded to 
a message, the deviation wouldn't take place.


Anyway, I'm also a disinterested observer, so please don't take this as 
a strong endorsement of doing the fix in 2.6; the set-an-attribute-to- 
fix-it idea is fine by me too.  I thought I'd lay out the reasoning for 
violating an otherwise extremely reasonable policy, though.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 4 Mar, 09:14 pm, krs...@solarsail.hcs.harvard.edu wrote:
I spent about a half hour sometime in the last month talking this 
through with Itamar, though not in great detail. I'd be interested in 
sitting down with both of you and trying to establish more precisely 
how much work is necessary to get something to actually happen here. I 
won't outright promise a PEP, but I'll promise at the very least to 
write down elaborate notes that someone could turn into a PEP 
relatively straightforwardly. Deal?


Absolutely.  I really appreciate the offer.  As the other gentlemen 
suggested, PyCon would be an ideal venue for doing this.  Are you going 
to be there?


I'll hopefully be seeing your talk this evening, but I suspect that 
would be a pretty bad venue to try to get this done ;-).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 4 Mar, 08:28 pm, ba...@python.org wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mar 4, 2009, at 2:44 PM, gl...@divmod.com wrote:
Maintaining compatibility with the 2.6.x version of asyncore 
presupposes that *someone* has written some software against that 
version of asyncore and it might break if they installed an upgrade, 
though.


FWIW, I use smtpd.py and a few of the asyncore APIs  (.loop(), 
.socket_map.clear(), and .close_all()) in the Mailman 3 test  suite. 
That only works on Python 2.6 and I don't recall even a hiccup  when 
moving from 2.5 to 2.6.


Well, in that case, I withdraw even my disinterested suggestion :).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 4 Mar, 09:36 pm, dan...@stutzbachenterprises.com wrote:
Will any or all of you be at PyCon?  I'd be willing to put in the extra 
work

to turn your notes into a PEP.


I definitely will be.  I'll see you there!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 07:30 pm, n...@arctrix.com wrote:

Chris McDonough  wrote:

As far as I can tell, asyncore/asynchat is all "undocumented
internals".  Any use of asyncore in anger will use internals;
there never was any well-understood API to these modules.



The implementation requires some intricate and platform specific
code which is why it would be nice to be a standard library feature.

I'm sure that Twisted has the necessary parts but the problem, IMHO,
is that it does so much more else.


... which is exactly why I have volunteered to explain to someone how to 
separate the core event-loop bits (suitable for inclusion in the 
standard library) from the huge pile of protocol implementations which 
are not necessarily useful.


Despite the FUD to the contrary, Twisted's internal factoring is quite 
good; it's not a single, undifferentiated pile of code.  Plus, at this 
point, we're not even talking about actually putting any Twisted code 
into the standard library, just standardizing the "protocol" API, which 
basically boils down to:


 connectionMade() -> your connection has begun
 dataReceived(data) -> you got some bytes, handle them
 connectionLost(reason) -> your connection has gone away (with an object 
explaining why)


and the inverse, "transport", which is:

 write(data) -> deliver some data to the dataReceived on the other end 
of this connection (non-blocking, with buffering)

 loseConnection() -> goodbye

There are a few other minor details related to how you set these up to 
talk to each other and tell when the out-buffer is empty, but it's all 
pretty straightforward.  The main point is that you don't ever call 
recv() or send() and deal with buffering or handling weird errno values. 
For example, if your connection goes away, the notification you get is 
"your connection went away", not "oops you tried to read some bytes, but 
your connection was gone by the time you tried, even though I just told 
you it was ready for reading" or other similarly obtuse failure modes.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 10:01 am, greg.ew...@canterbury.ac.nz wrote:

Hrvoje Niksic wrote:

   Under Linux, select() may report a socket file descriptor
   as "ready for reading",  while  nevertheless
   a subsequent read blocks.


Blarg. Linux is broken, then. This should not happen.


You know what else is broken?  MacOS, FreeBSD, Solaris, and every 
version of Windows.  I haven't tried running Twisted on AmigaOS but I 
bet it has some problems too.


On occasion Linux has been so badly broken that Twisted has motivated a 
fix.  For example, 
http://lkml.indiana.edu/hypermail/linux/kernel/0502.3/1160.html


But even if we ignore difficulties at the OS level (which should, after 
all, be worked around rather than catered to in API design) there are 
other good reasons why the general async API should be fairly distant 
from both the select/poll wrapper and the questions of blocking vs. non- 
blocking sockets.  For another example, consider the issue of non- 
blocking SSL sockets.  Sometimes, in order to do a "read", you actually 
need to do a write and then another read.  Which means that application 
code, if it wants to be compatible with SSL, needs to deal with any 
failure that may come from a write as coming from a read, unless you 
abstract all this away somehow.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 02:28 am, s...@pobox.com wrote:
   Glyph> ... which is exactly why I have volunteered to explain to 
someone

   Glyph> how to separate the core event-loop bits



   Neil> This sounds great.



Anybody interested in working on this at a PyCon Sprint?  I won't be
attending the conference proper, but plan to spend a couple days 
sprinting,
one on Mailman/SpamBayes integration and one on Python core stuff. 
This
might fit well into my Python core "slot".  I will probably have little 
time
before the sprint to do much, but any brain dumps or Twisted pointers 
people

could give me in the interim would be appreciated.


I'll try to make sure we get those notes to you in advance of the 
sprints :).  I should be at the Twisted sprint the whole time - will you 
be at the physical sprint, or following along at home?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 01:50 am, n...@arctrix.com wrote:

gl...@divmod.com  wrote:



we're not even talking about actually putting any Twisted code
into the standard library, just standardizing the "protocol" API, 
which

basically boils down to:

[...]

This sounds great.  I'm interested on working on this since it
scratches an itch of mine but I don't know if I will have time.  Do
you think if this part of Twisted became part of the standard
library that it would be used by Twisted or would it continue to
have its own version?


As I said above, right now I'm just talking about the interface.  There 
are several implementations of the main loop that call those interfaces, 
as well as test implementations.


Of course I hope we can eliminate some redundancy, but that's step 2: 
before we refactor, we have to make the code that we're refactoring 
actually repetitive rather than just similar.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa

[glyph]

On 02:46 pm, dan...@stutzbachenterprises.com wrote:

On Sat, Mar 7, 2009 at 8:28 PM,  wrote:

Anybody interested in working on this at a PyCon Sprint?  I won't be
attending the conference proper, but plan to spend a couple days 
sprinting,



I'll be there and interested. :)


Great!  I plan to hold both of you to that :).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[glyph]

On 12:56 pm, ncogh...@gmail.com wrote:

PEP 377 is a proposal to allow context manager __enter__() methods to
skip the body of the with statement by raising a specific (new) flow
control exception.

Since there is a working reference implementation now, I thought it was
time to open it up for broader discussion.


Why not allow a context manager to implement some other method, for the 
sake of argument let's say "__start__", which was invoked with a 
callable object and could choose to evaluate or not evaluate the 
statement body by simply not calling that object (or perhaps iterable, 
in the case of a generator)?


This PEP proposes that we have two ways to deal with the body of a 
'with' statement: either the body is run or not.  I have always wanted 
to have another option: run the body later.


Passing around an object representing the body of the with statement 
would allow for this use-case, as well as removing the ugly protrusion 
of yet another control-flow exception (which, as has already been noted, 
creates difficulties for some other python implementations).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "setuptools has divided the Python community"

[glyph]

On 26 Mar, 07:22 pm, ba...@python.org wrote:
One thing that /would/ be helpful though is the ability to list all 
the resources under a specific package path.  This is (I think) one 
use case that pkg_resource fails to support and it's the one place 
that I've had to drop down to file system introspection.


Think: the package-y flavor of 
os.listdir(os.path.dirname(package.__file__))


FWIW, the Twisted spelling of this is:

from twisted.python.modules import getModule
getModule(__name__).filePath.parent().listdir()

(and yes, this works with zip files)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] splitting out bdist_* (was: interminable 'setuptools' thread)

[glyph]

On 07:59 pm, fdr...@acm.org wrote:
I'm actually in favor of removing the bdist_* from the standard 
library, and allowing 3rd-party tools to implement whatever they need 
for the distros.  But I don't think what you're presenting there 
supports it.


I do think that it's relevant that the respective operating system 
packagers don't find bdist_rpm, bdist_deb, et. al. useful.  It's not 
very useful to have a bdist_deb that nobody actually builds debs with. 
This has been a problem for me, personally, since debian has made 
various ad-hoc change to Twisted or Twisted-based packages to break our 
plugin system, since the distutils metadata has been insufficient for 
their purposes.  If the deb-generating stuff were in a separate project 
with a faster release cycle that were easier to contribute packages to, 
perhaps debian packagers could be convinced to contribute their build- 
hacks there (and bdist_deb could invoke debhelper, or vice-versa).


It would be great if someone could volunteer to maintain this stuff 
independently, put it in a Launchpad project or something.  IMHO it 
would be better for the community at large if this were spun as 
"increasing the release speed" of the bdist_* family, rather than 
"removing", which seems to me like it would generate another teacup- 
tempest on the blogowebs.  Of course I'm not volunteering, but I will be 
best friends forever with whoever does this PR/maintenance :).


Given that py2exe and py2app (which includes "bdist_mpkg") are both 
based on distutils, it seems like we're on the way to independent 
maintenance anyway.  Perhaps bdist_wininst/_msi could be donated to the 
py2exe project if they would be willing to maintain it, and the new 
project for _deb and _rpm could be called "py2packman" or something.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382: Namespace Packages

[glyph]

On 08:15 pm, mar...@v.loewis.de wrote:

Note that there is no such thing as a "defining namespace package" --
namespace package contents are symmetrical peers.


With the PEP, a "defining package" becomes possible - at most one
portion can define an __init__.py.


For what it's worth, this is a _super_ useful feature for Twisted.  We 
have one "defining package" for the "twisted" package (twisted core) and 
then a bunch of other things which want to put things into twisted.* 
(twisted.web, twisted.conch, et. al.).


For debian we already have separate packages, but such a definition of 
namespace packages would allow us to actually have things separated out 
on the cheeseshop as well.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] the email module, text, and bytes (was Re: Dropping bytes "support" in json)

[glyph]

On 02:26 am, ba...@python.org wrote:
There are really two ways to look at an email message.  It's either an 
unstructured blob of bytes, or it's a structured tree of objects. 
Those objects have headers and payload.  The payload can be of any 
type, though I think it generally breaks down into "strings" for text/ 
* types and bytes for anything else (not counting multiparts).


I think this is a problematic way to model bytes vs. text; it gives text 
a special relationship to bytes which should be avoided.


IMHO the right way to think about domains like this is a multi-level 
representation.  The "low level" representation is always bytes, whether 
your MIME type is text/whatever or application/x-i-dont-know.


The thing that's "special" about text is that it's a "high level" 
representation that the standard library can know about.  But the 
'email' package ought to support being extended to support other types 
just as well.  For example, I want to ask for image/png content as 
PIL.Image objects, not bags of bytes.  Of course this presupposes some 
way for PIL itself to get at some bytes, but then you need the email 
module itself to get at the bytes to convert to text in much the same 
way.  There also needs to be layering at the level of 
bytes->base64->some different bytes->PIL->Image.  There are mail clients 
that will base64-encode unusual encodings so you have to do that same 
layering for text sometimes.


I'm also being somewhat handwavy with talk of "low" and "high" level 
representations; of course there are actually multiple levels beyond 
that.  I might want text/x-python content to show up as an AST, but the 
intermediate DOM-parsing representation really wants to operate on 
characters.  Similarly for a DOM and text/html content.  (Modulo the 
usual encoding-detection weirdness present in parsers.)


So, as long as there's a crisp definition of what layer of the MIME 
stack one is operating on, I don't think that there's really any 
ambiguity at all about what type you should be getting.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping bytes "support" in json

[glyph]

On 02:38 am, ba...@python.org wrote:
So, what I'm really asking is this.  Let's say you agree that there 
are use cases for accessing a header value as either the raw encoded 
bytes or the decoded unicode.  What should this return:


>>> message['Subject']

The raw bytes or the decoded unicode?


My personal preference would be to just get deprecate this API, and get 
rid of it, replacing it with a slightly more explicit one.


   message.headers['Subject']
   message.bytes_headers['Subject']
Now, setting headers.  Sometimes you have some unicode thing and 
sometimes you have some bytes.  You need to end up with bytes in the 
ASCII range and you'd like to leave the header value unencoded if so. 
But in both cases, you might have bytes or characters outside that 
range, so you need an explicit encoding, defaulting to utf-8 probably.


   message.headers['Subject'] = 'Some text'

should be equivalent to

   message.headers['Subject'] = Header('Some text')

My preference would be that

   message.headers['Subject'] = b'Some Bytes'

would simply raise an exception.  If you've got some bytes, you should 
instead do


   message.bytes_headers['Subject'] = b'Some Bytes'

or

   message.headers['Subject'] = Header(bytes=b'Some Bytes', 
encoding='utf-8')


Explicit is better than implicit, right?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dropping bytes "support" in json

[glyph]

On 03:21 am, ncogh...@gmail.com wrote:

Barry Warsaw wrote:



I don't know whether the parameter thing will work or not, but you're
probably right that we need to get the bytes-everywhere API first.



Given that json is a wire protocol, that sounds like the right approach
for json as well. Once bytes-everywhere works, then a text API can be
built on top of it, but it is difficult to build a bytes API on top of 
a

text one.


I wish I could agree, but JSON isn't really a wire protocol.  According 
to http://www.ietf.org/rfc/rfc4627.txt JSON is "a text format for the 
serialization of structured data".  There are some notes about encoding, 
but it is very clearly described in terms of unicode code points.

So I guess the IO library *is* the right model: bytes at the bottom of
the stack, with text as a wrapper around it (mediated by codecs).


In email's case this is true, but in JSON's case it's not.  JSON is a 
format defined as a sequence of code points; MIME is defined as a 
sequence of octets.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382: Namespace Packages

[glyph]

On 15 Apr, 09:11 pm, p...@telecommunity.com wrote:
I think that there is some confusion here.  A "main" package or 
buildout that assembles a larger project from components is not the 
same thing as having a "base" package for a namespace package.


I'm certainly confused.

Twisted has its own system for "namespace" packages, and I'm not really 
sure where we fall in this discussion.  I haven't been able to follow 
the whole thread, but my original understanding was that the PEP 
supports "defining packages", which we now seem to be calling "base 
packages", just fine.  I don't understand the controversy over the 
counterproposal, since it seems roughly functionally equivalent to me.


I'd appreciate it if the PEP could also be extended cover Twisted's very 
similar mechanism for namespace packages, 
"twisted.plugin.pluginPackagePaths".  I know this is not quite as widely 
used as setuptools' namespace package support, but its existence belies 
a need for standardization.


The PEP also seems a bit vague with regard to the treatment of other 
directories containing __init__.py and *.pkg files.  The concept of a 
"defining package" seems important to avoid conflicts like this one:


   http://twistedmatrix.com/trac/ticket/2339

More specifically I don't quite understand the PEP's intentions towards 
hierarchical packages.  It says that all of sys.path will be searched, 
but what about this case?


In Twisted, the suggested idiom to structure a project which wants to 
provide Twisted plugins is to have a directory structure like this:


 MyProject/
   myproject/
 __init__.py
   twisted/
 plugins/
   myproject_plugin.py

If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins will 
be picked up automatically by the plugin machinery.  However, as 
"twisted" is *not* a "namespace" package in the same way, .py files in 
MyProject/twisted/ would not be picked up - this is very much 
intentional, since the "twisted" namespace is intended to be reserved 
for packages that we actually produce.  If either MyProject/twisted or 
MyProject/twisted/plugins/ had an __init__.py, then no modules in 
MyProject/twisted/plugins/ would be picked up, because it would be 
considered a conflicting package.


This is important so that users can choose not to load the system- 
installed Twisted's plugins when they have both a system-installed 
version of Twisted and a non-installed development version of Twisted 
found first on their PYTHONPATH, and switch between them to indicate 
which version they want to be the "base" or "defining" package for the 
twisted/plugins/ namespace.


Developers might also want to have a system-installed Twisted, but a 
non-installed development version of MyProject on PYTHONPATH.


I hope this all makes sense.  As I understand it, both setuptools and 
the proposed standard would either still have the bug described by 
ticket 2339 above, or would ignore twisted/plugins/ as a namespace 
package because its parent isn't a namespace package.  I apologize for 
not testing with current setuptools before asking, but I'm not sure my 
experiments would be valid given that my environment is set up with 
assumptions from Twisted's system.


P.S.: vendor packaging systems *ARE* a major use case for just about any 
aspect of Python's package structure.  I really liked MvL's coverage of 
"vendor packages", in the PEP, since this could equally well apply to 
MSIs, python libraries distributed in bundles on OS X, debs, or RPMs. 
If this use-case were to be ignored, as one particular fellow seems to 
be advocating, then the broken packages and user confusion that has been 
going on for the last 5 years or so is just going to get worse.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue5434: datetime.monthdelta

[glyph]

On 16 Apr, 11:11 pm, f...@fuhm.net wrote:

On Apr 16, 2009, at 5:47 PM, Antoine Pitrou wrote:


It's a human-interface operation, and as such, everyone (ahem) "knows 
what it means" to say "2 months from now", but the details don't 
usually have to be thought about too much. Of course when you have a 
computer program, you actually need to tell it what you really mean.


I do a fair amount of date calculating, and use two different kinds of 
"add-month":


Option 1)
Add n to the month number, truncate day number to fit the month you 
end up in.


Option 2)
As above, but with the additional caveat that if the original date is 
the last day of its month, the new day should also be the last day of 
the new month. That is:

April 30th + 1 month = May 31st, instead of May 30th.

They're both useful behaviors, in different circumstances.


I don't have a third option, but something that would be useful to 
mention in the documentation for "monthdelta": frequently users will 
want a recurring "monthly" event.  It's important to note that you need 
to keep your original date around if you want these rules to be 
consistently applied.  For example, if you have a monthly billing cycle 
that starts on May 31, you need to keep the original May 31 around to 
add monthdelta(X) if you want it to be May 31 when it rolls around next 
year; otherwise every time February rolls around all of your end-of- 
month dates get clamped to the 28th of every month.  (Unless you're 
following James's option 2, of course, in which case things which are 
normally on the 28th will get clamped to the 31st of following months.)


My experience with month-calculating software suggests that this is 
something very easy to screw up.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 382: Namespace Packages

[glyph]

On 16 Apr, 03:36 pm, p...@telecommunity.com wrote:

At 03:46 AM 4/16/2009 +, gl...@divmod.com wrote:

On 15 Apr, 09:11 pm, p...@telecommunity.com wrote:


Twisted has its own system for "namespace" packages, and I'm not 
really sure where we fall in this discussion.  I haven't been able to 
follow the whole thread, but my original understanding was that the 
PEP supports "defining packages", which we now seem to be calling 
"base packages", just fine.


Yes, it does.  The discussion since the original proposal, however, has 
been dominated by MAL's counterproposal, which *requires* a defining 
package.


[snip clarifications]

Does that all make sense now?


Yes.  Thank you very much for the detailed explanation.  It was more 
than I was due :-).
MAL's proposal requires a defining package, which is counterproductive 
if you have a pure package with no base, since it now requires you to 
create an additional project on PyPI just to hold your defining 
package.


Just as a use-case: would the Java "com.*" namespace be an example of a 
"pure package with no base"?  i.e. lots of projects are in it, but no 
project owns it?
I'd appreciate it if the PEP could also be extended cover Twisted's 
very similar mechanism for namespace packages,


"twisted.plugin.pluginPackagePaths".  I know this is not quite as 
widely used as setuptools' namespace package support, but its 
existence belies a need for standardization.


The PEP also seems a bit vague with regard to the treatment of other 
directories containing __init__.py and *.pkg files.


Do you have a clarification to suggest?  My understanding (probably a 
projection) is that to be a nested namespace package, you have to have 
a parent namespace package.


Just to clarify things on my end: "namespace package" to *me* means 
"package with modules provided from multiple distributions (the 
distutils term)".  The definition provided by the PEP, that a package is 
spread over multiple directories on disk, seems like an implementation 
detail.


Entries on __path__ slow down import, so my understanding of the 
platonic ideal of a system python installation is one which has a single 
directory where all packages reside, and a set of metadata off to the 
side explaining which files belong to which distributions so they can be 
uninstalled by a package manager.


Of course, for a development installation, easy uninstallation and quick 
swapping between different versions of relevant dependencies is more 
important than good import performance.  So in that case, you would want 
to optimize differently by having all of your distributions installed 
into separate directories, with a long PYTHONPATH or lots of .pth files 
to point at them.


And of course you may want a hybrid of the two.

So another clarification I'd like in the PEP is an explanation of 
motivation.  For example, it comes as a complete surprise to me that the 
expectation of namespace packages was to provide only single-source 
namespaces like zope.*, peak.*, twisted.*.  As I mentioned above, I 
implicitly thought this was more for com.*, twisted.plugins.*.


Right now it just says that it's a package which resides in multiple 
directories, and it's not made clear why that's a desirable feature.
  The concept of a "defining package" seems important to avoid 
conflicts like this one:


   http://twistedmatrix.com/trac/ticket/2339


[snip some stuff about plugins and package layout]
Namespaces are not plugins and vice versa.  The purpose of a namespace 
package is to allow projects managed by the same entity to share a 
namespace (ala Java "package" names) and avoid naming conflicts with 
other authors.


I think this is missing a key word: *separate* projects managed by the 
same entity.


Hmm.  I thought I could illustrate that the same problem actually occurs 
without using a plugin system, but I can actually only come up with an 
example if an application implements multi-library-version compatibility 
by doing


   try:
   from bad_old_name import bad_old_feature as feature
   except ImportError:
   from good_new_name import good_new_feature as feature

rather than the other way around; and that's a terrible idea for other 
reasons.  Other than that, you'd have to use 
pkg_resources.resource_listdir or somesuch, at which point you pretty 
much are implementing a plugin system.


So I started this reply disagreeing but I think I've convinced myself 
that you're right.
Precisely.  Note, however, that neither is twisted.plugins a namespace 
package, and it should not contain any .pkg files.  I don't think it's 
reasonable to abuse PEP 382 namespace packages as a plugin system.  In 
setuptools' case, a different mechanism is provided for locating plugin 
code, and of course Twisted already has its own system for the same 
thing. It would be nice to have a standardized way of locating plugins 
in the stdlib, but that will need to be a different PEP.


Okay.  So what I'm hearing is that Twisted sh

Re: [Python-Dev] PEP 382: Namespace Packages

[glyph]

On 04:56 am, p...@telecommunity.com wrote:

At 03:58 AM 4/17/2009 +, gl...@divmod.com wrote:
Just as a use-case: would the Java "com.*" namespace be an example of 
a "pure package with no base"?  i.e. lots of projects are in it, but 
no project owns it?


Er, I suppose.  I was thinking more of the various 'com.foo' and 
'org.bar' packages as being the pure namespaces in question.  For 
Python, a "flat is better than nested" approach seems fine at the 
moment.


Sure.  I wasn't saying we should go down the domain-names-are-package- 
names road for Python itself, just that "com.*" is a very broad example 
of a multi-"vendor" namespace :).
Entries on __path__ slow down import, so my understanding of the 
platonic ideal of a system python installation is one which has a 
single directory where all packages reside, and a set of metadata off 
to the side explaining which files belong to which distributions so 
they can be uninstalled by a package manager.


True... except that part of the function of the PEP is to ensure that 
if you install those separately-distributed modules to the same 
directory, it still needs to work as a package and not have any inter- 
package file conflicts.


Are you just referring to anything other than the problem of multiple 
packages overwriting __init__.py here?  It's phrased in a very general 
way that makes me think maybe there's something else going on.
So another clarification I'd like in the PEP is an explanation of 
motivation.  For example, it comes as a complete surprise to me that 
the expectation of namespace packages was to provide only single- 
source namespaces like zope.*, peak.*, twisted.*.  As I mentioned 
above, I implicitly thought this was more for com.*, 
twisted.plugins.*.


Well, aside from twisted.plugins, I wasn't aware of anybody in Python 
doing that...  and as I described, I never really interpreted that 
through the lens of "namespace package" vs. "plugin finding".


There is some overlap.  In particular, in the "vendor distribution" 
case, I would like there to be one nice, declarative Python way to say 
"please put these modules into the same package".  In the past, Debian 
in particular has produced some badly broken Twisted packages in the 
past because there was no standard Python way to say "I have some 
modules here that go into an existing package".  Since every 
distribution has its own funny ideas about what the filesystem should 
look like, this has come up for us in a variety of ways.


I'd like it if we could use the "official" way of declaring a namespace 
package for that.
Right now it just says that it's a package which resides in multiple 
directories, and it's not made clear why that's a desirable feature.


Good point; perhaps you can suggest some wording on these matters to 
Martin?


I think the thing I said in my previous message about "multiple 
distributions" is a good start.  That might not be everything, but I 
think it's clearly the biggest motivation.
Okay.  So what I'm hearing is that Twisted should happily continue 
using our own wacky __path__-calculation logic for twisted.plugins, 
but that *twisted* should be a namespace package so that our separate 
distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be 
installed into separate directories.


Yes.


I'm fairly happy with that, except the aforementioned communication- 
channel-with-packagers feature of namespace packages; they unambiguously 
say "multiple OS packages may contribute modules to this Python 
package".
Thanks for taking the time to participate in this and add another 
viewpoint to the mix, not to mention clarifying some areas where the 
PEP could be clearer.


My pleasure.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces

[glyph]

On 06:50 am, mar...@v.loewis.de wrote:

I'm proposing the following PEP for inclusion into Python 3.1.
Please comment.



To convert non-decodable bytes, a new error handler "python-escape" is
introduced, which decodes non-decodable bytes using into a private-use
character U+F01xx, which is believed to not conflict with private-use
characters that currently exist in Python codecs.


-1.  On UNIX, character data is not sufficient to represent paths.  We 
must, must, must continue to have a simple bytes interface to these 
APIs.  Covering it up in layers of obscure encoding hacks will not make 
the problem go away, it will just make it harder to understand.


To make matters worse, Linux and GNOME use the PUA for some printable 
characters.  If you open up charmap on an ubuntu system and select "view 
by unicode character block", then click on "private use area", you'll 
see many of these.  I know that Apple uses at least a few PUA codepoints 
for the apple logo and the propeller/option icons as well.


I am still -1 on any turn-non-decodable-bytes-into-text, because it 
makes life harder for those of us trying to keep bytes and text 
straight, but if you absolutely must represent POSIX filenames as 
mojibake rather than bytes, the only workable solution is to use NUL as 
your escape character.  That's the only code point which _actually_ 
can't show up in a filename somehow.  As we discussed last time, this is 
what Mono does with System.IO.Path.  As a bonus, it's _much_ easier to 
detect a NUL from random application code than to try to figure out if a 
string has any half-surrogates or magic PUA characters which shouldn't 
be interpreted according to platform PUA rules.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces

[glyph]

On 07:17 pm, mar...@v.loewis.de wrote:

-1.  On UNIX, character data is not sufficient to represent paths.  We
must, must, must continue to have a simple bytes interface to these
APIs.



I'd like to respond to this concern in three ways:

1. The PEP doesn't remove any of the existing interfaces. So if the
  interfaces for byte-oriented file names in 3.0 work fine for you,
  feel free to continue to use them.


It's good to know this.  It would be good if the PEP made it clear that 
it is proposing an additional way to work with undecodable bytes, not 
replacing the existing one.


For me, this PEP isn't an acceptable substitute for direct bytes-based 
access to command-line arguments and environment variables on UNIX.  To 
my knowledge *those* APIs still don't exist yet.  I would like it if 
this PEP were not used as an excuse to avoid adding them.

2. Even if they were taken away (which the PEP does not propose to do),
  it would be easy to emulate them for applications that want them.


I think this is a pretty clear abstraction inversion.  Luckily nobody is 
proposing it :).

3. I still disagree that we must, must, must continue to provide these
  interfaces.


You do have a point; if there is a clean, defined mapping between str 
and bytes in terms of all path/argv/environ APIs, then we don't *need* 
those APIs, since we can just implement them in terms of characters. 
But I still think that's a bad idea, since mixing the returned strings 
with *other* APIs remains problematic.  However, I still think the 
mapping you propose is problematic...

  I don't understand from the rest of your message what
  would *actually* break if people would use the proposed interfaces.


As far as more concrete problems: the utf-8 codec currently in python 
2.5 and 2.6, and 3.0 will happily encode half-surrogates, at least in 
the builds I have.


   >>> '\udc81'.encode('utf-8').decode('utf-8')
   '\udc81'

So there's an ambiguity when passing U+DC81 to this codec: do you mean 
\xed\xb2\x81 or do you just mean \x81?  Of course it would be possible 
to make UTF-8B consistent in this regard, but it is still going to 
interact with code that thinks in terms of actual UTF-8, and the failure 
mode here is very difficult to inspect.


A major problem here is that it's very difficult to puzzle out whether 
anything *will* actually break.  I might be wrong about the above for 
some subtlety of unicode that I don't quite understand, but I don't want 
to spend all day experimenting with every possible set of build options, 
python versions, and unicode specifications.  Neither, I wager, do most 
people who want to call listdir().


Another specific problem: looking at the Character Map application on my 
desktop, U+F0126 and U+F0127 are considered printable characters.  I'm 
not sure what they're supposed to be, exactly, but there are glyphs 
there.  This is running Ubuntu 8.04; there may be more of these in use 
in more recent version of GNOME.


There is nothing "private" about the "private use" area; Python can 
never use any of these characters for *anything*, except possibly 
internally in ways which are never exposed to application code, because 
the operating system (or window system, or libraries) might use them. 
If I pass a string with those printable PUA/A characters in it to 
listdir(), what happens?  Do they get turned into bytes, do they only 
get turned into bytes if my filesystem encoding happens to be something 
other than UTF-8...?


The PEP seems a bit ambiguous to me as far as how the PUA hack and the 
half-surrogate hack interact.  I could be wrong, but it seems to me to 
be an either-or proposition, in which case there would be *four* bytes 
types in python 3.1: bytes, bytearray, str-with-PUA/A-junk, str-with- 
half-surrogate-junk.  Detecting the difference would be an expensive and 
subtle affair; the simplest solution I could think of would be to use an 
error-prone regex.  If the encoding hack used were simply NULL, then the 
detection would be straightforward: "if '\u' in thingy:".


Ultimately I think I'm only -0 on all of this now, as long as we get 
bytes versions of environ and argv.  Even if these corner-case issues 
aren't fixed, those of us who want to have correct handling of 
undecodable filenames can do so.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a suggestion ... Re: PEP 383 (again)

[glyph]

On 08:25 am, mar...@v.loewis.de wrote:

Why did you choose an incompatible approach for PEP 383?


Because in Python, we want to be able to access all files on disk.
Neither Java nor Mono are capable of doing that.


Java is not capable of doing that.  Mono, as I keep pointing out, is. 
It uses NULLs to escape invalid UNIX filenames.  Please see:


http://go-mono.com/docs/index.aspx?link=T%3AMono.Unix.UnixEncoding

"The upshot to all this is that Mono.Unix and Mono.Unix.Native can list, 
access, and open all files on your filesystem, regardless of encoding."

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a suggestion ... Re: PEP 383 (again)

[glyph]

On 02:42 pm, tmb...@gmail.com wrote:

So, why do you prefer half surrogate coding to U+ quoting?


I have also been eagerly waiting for an answer to this question.  I am 
afraid I have lost it somewhere in the storm of this thread :).


Martin, if you're going to stick with the half-surrogate trick, would 
you mind adding a section to the PEP on "alternate encoding strategies", 
explaining why the NULL method was not selected?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a suggestion ... Re: PEP 383 (again)

[glyph]

On 03:35 pm, mar...@v.loewis.de wrote:

So, why do you prefer half surrogate coding to U+ quoting?


If I pass a string with an embedded U+ to gtk, gtk will truncate
the string, and stop rendering it at this character. This is worse than
what it does for invalid UTF-8 sequences. Chances are fairly high that
other C libraries will fail in the same way, in particular if they
expect char* (which is very common in C).


Hmm.  I believe the intended failure mode here, for PyGTK at least, is 
actually this:


   TypeError: GtkLabel.set_text() argument 1 must be string without null 
bytes, not unicode


APIs in PyGTK which accept NULLs and silently trucate are probably 
broken.  Although perhaps I've just made your point even more strongly; 
one because the behavior is inconsistent, and two because it sometimes 
raises an exception if a NULL is present, and apparently the goal here 
is to prevent exceptions from being raised anywhere in the process.


For this idiom to be of any use to GTK programs, 
gtk.FileChooser.get_filename() will probably need to be changed, since 
(in py2) it currently returns a str, not unicode.


The PEP should say something about how GUI libraries should handle file 
choosers, so that they'll be consistent and compatible with the standard 
library.  Perhaps only that file choosers need to take this PEP into 
account, and the rest is obvious.  Or maybe the right thing for GTK to 
do would be to continue to use bytes on POSIX and convert to text on 
Windows, since open(), listdir() et. al. will continue to accept bytes 
for filenames?

So I prefer the half surrogate because its failure mode is better th


Heh heh heh.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] a suggestion ... Re: PEP 383 (again)

[glyph]

On 04:07 pm, mar...@v.loewis.de wrote:

Martin, if you're going to stick with the half-surrogate trick, would
you mind adding a section to the PEP on "alternate encoding 
strategies",

explaining why the NULL method was not selected?


In the PEP process, it isn't my job to criticize competing proposals.
Instead, proponents of competing proposals should write alternative
PEPs, which then get criticized on their own. As the PEP author, I 
would

have to collect the objections to the PEP in the PEP, which I did;
I'm not convinced that I would have to also collect all alternative
proposals that people come up with in the PEP (except when they are in
fact amendments that I accept).


Fair enough.  I have probably misunderstood the process.  I dimly 
recalled reading some PEPs which addressed alternate approaches in this 
way and I thought it was part of the process.


Anyway, congratulations on getting the PEP accepted, good luck with the 
implementation.  Thanks for addressing my question.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] question about docstring formatting

[glyph]

On 01:06 pm, jer...@alum.mit.edu wrote:

It says that the summary line may be used by automatic indexing tools,
but is there any evidence that such a tool actually exists?  Or was
there once upon a time?  If there are no such tools, do we still think
that it is important that it fits on line line?


For what it's worth, https://launchpad.net/pydoctor appears to do this, 
as you can see from the numerous truncated sentences on 
.


I suspect a more reasonable approach for automatic documentation 
generators would be to try to identify the first complete sentence, 
rather than the first line... but, this is at least an accurate 
description of the status quo for some tools :).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Issues with process and discussions (Re: Issues with Py3.1's new ipaddr)

[glyph]

On 02:39 am, gu...@python.org wrote:

I'm disappointed in the process -- it's as if nobody really reviewed
the API until it was released with rc1, and this despite there being a
significant discussion about its inclusion and alternatives months
ago. (Don't look at me -- I wouldn't recognize a netmask if it bit me
in the behind, and I can honestly say that I don't know whether /8
means to look only at the first 8 bits or whether it means to mask off
the last 8 bits.)

I hope we can learn from this.


As he pointed out to Martin, Jean-Paul voiced objections several months 
ago which are similar to the ones which are now being discussed.  To be 
fair, he didn't unambiguously say "... and therefore don't include this 
library"; he simply suggested that netaddr was superior in some ways and 
that perhaps some documentation could illuminate why ipaddr was better.


I've been frustrated with similar aspects of Python's development 
process in the past.  The biggest problem I can see is that it's too 
hard to follow the discussion, and catch oneself up on the discussion 
thus far.


It's also difficult to refer back to posts much earlier in the history 
of an email discussion, and that frequently needs to be done when 
someone jumps into the middle of a discussion.


The way Twisted dealt with this particular issue was to move *all* 
discussions relevant to a particular feature into the ticket for that 
feature.  If discussion starts up on the mailing list, within a few 
messages of it starting, someone on the dev team will pipe up and say 
"Hey, here's the ticket for this, could you add a link to this 
discussion and a summary".


Once on a ticket, the phraseology and typesetting used by our core team 
has reached a very precise consensus.  Like the feature?  "Merge this 
patch" or "Land this branch".  Don't like it?  "Thanks for your patch, 
but:", followed by a list of enumerated feedback.  The required 
reactions to such feedback are equally well understood.  Even if a 
comment isn't a full, formal code review, it still follows a similar 
style.


This system is possibly too simplistic for the more wide-ranging 
development of Python; although Twisted has its share of enthusiastic 
discussions, there is rarely the level of controversy I see here on 
python-dev, so I can't recommend it as such.  I can say that keeping 
ticket discussions on tickets is generally a good idea, though, 
especially in a system like roundup or trac where it's easy to say "see 
message 7" rather than repeating yourself.


It seems that there is a habit of occasionally using ASF-style 
+1/+0/-0/-1 markers, but it's inconsistently applied.  More importantly, 
nobody ever actually adds them up, so it's not entirely clear when they 
should be used.


To go back to JP's original comments though: what was the right thing 
for him to do, back in January, when he had these concerns?  Should he 
have said "I am therefore -1 on this inclusion"?  Should he have been 
discussing this on the mailing list rather than the tracker?  Should he 
have kept coming back to the ticket and answering every single message 
reinforcing his original conclusions?  I honestly don't think it's very 
clear what one is "officially" supposed to do.  Without a clear 
expectation that one should say "No" to features that are problematic, 
it seems gratuitously mean to do so; so, it's nicer to just say "here's 
what I found" with the hopes that someone will evaluate that feedback.


Unfortunately it seems like the winning strategy here is just to keep 
flogging a dead horse until it's a dead horse hamburger; reply and reply 
and reply until everybody gets sick of talking about it.  Repeat your 
original points in every post so that nobody will miss them.  I think 
everyone is ill-served by this discussion format.  Certainly when I 
voice my own objections or support for something, I'd like to just stop 
by, add a note for the committers to take into account when considering 
the issue, and then go away.


So, here are my recommendations:

 1. Use the tracker for discussing tickets, so that it's easy to refer 
back to a previous point in the discussion, and so that people working 
on those tickets can easily find your commentary.
 2. Use the mailing list for drawing attention to these discussions if 
they are of general interest, especially if the discussion is time- 
critical.  In this case, an announcement "You have six weeks to review 
ipaddr now until its inclusion is permanent, anyone interested please 
look at issue 3959."
 3. If you have an opinion, put your +1/+0/-0/-1 on a line by itself at 
the top of your message, so that it's easy for newcomers to the 
discussion to get a general feel.


Of course, this won't prevent all meandering discussions, or discussions 
that are too late to the party, but I do think it will help.


However, I think before everyone just starts doing this, even *more* 
important is my meta-suggestion: let's agree on how and when feedba

Re: [Python-Dev] Issues with Py3.1's new ipaddr

[glyph]

On 02:44 am, a...@pythoncraft.com wrote:

On Tue, Jun 02, 2009, Guido van Rossum wrote:


I hope we can learn from this.


I'm not thrilled with adding more process just because we had a problem
here, and the only obvious solution I see is to require a PEP every 
time
a module is added.  Based on what I've seen of this discussion so far, 
I

think that cure would at this time be worse than the disease.


I thought the solution that I just proposed was pretty obvious ;-).

But in all seriousness, even if an improvement looks nothing like what I 
just proposed, it seems like a simple failure of imagination to say that 
nothing could make this situation better.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with Py3.1's new ipaddr

[glyph]

On 07:51 am, p.f.mo...@gmail.com wrote:

2009/6/3 Stephen J. Turnbull :



One thing I would recommend is that while inclusion is not a matter of
voting, people who are recognized as domain experts on Python-Dev
*should* try to add their "+1" or "-1" early. �Especially if they
don't expect to have time to participate actively in discussion.



2. Encouraging a clear +1/-1 from people, in addition to discussion on
specific points, would clarify things. I believe Martin commented that
he hadn't realised that one of the opposing comments was a strong
enough objection to count as a -1.


+1 :-)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with Py3.1's new ipaddr

[glyph]

On 05:19 pm, gu...@python.org wrote:
On Wed, Jun 3, 2009 at 10:13 AM, Neil Schemenauer  
wrote:

Barry Warsaw  wrote:

It would be really nice if say the Cheeseshop had a voting feature.
Use PEP 10 voting to get a rough estimate of a module's popularity
(download counts alone might not tell you everything). �Then at least
you can get a rough idea of how generally popular a module is in the
wider community. �Also, a module should have to live on its own two
feet for while on Cheeseshop before being considered for inclusion in
the stdlib.


Better yet would be something like Debian's popularity-contest
mechanism (it provides opt-in voting of packages based on what is
installed on your machine). �popularity-contest runs from a cron
job. �Maybe when Python is installed it could ask if you want to
submit package statistics. �If so, installing a package with
distutils would submit the name and version number to a central
server.

If we knew which batteries were most popular we could make sure they
are included in the package. ;-)


Whoa. Are you all suddenly trying to turn Python into a democracy? I'm
outta here if that ever happens (even if I were voted BDFL by a
majority :-).


I'm sure that what Barry and Neil are recommending is the same as what I 
did later: a way to give our most beloved regent accurate data on the 
populace's current mood.


Certainly we wouldn't be discussing our plans for a democratic coup out 
in the open like this!  Clearly, that would be foolish ;-).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with process and discussions (Re: Issues with Py3.1's new ipaddr)

[glyph]

On 05:42 pm, p.f.mo...@gmail.com wrote:

2009/6/3  :

So, here are my recommendations:

 1. Use the tracker for discussing tickets, so that it's easy to refer 
back
to a previous point in the discussion, and so that people working on 
those

tickets can easily find your commentary.
 2. Use the mailing list for drawing attention to these discussions if 
they
are of general interest, especially if the discussion is time- 
critical.  In
this case, an announcement "You have six weeks to review ipaddr now 
until
its inclusion is permanent, anyone interested please look at issue 
3959."
 3. If you have an opinion, put your +1/+0/-0/-1 on a line by itself 
at the
top of your message, so that it's easy for newcomers to the discussion 
to

get a general feel.


Mostly, I agree, but I definitely disagree, I'm afraid, on the use of
the tracker for discussions. To keep track of discussions on a ticket,
I have to personally keep a list of the tickets I'm interested in,
check back regularly to see if there's anything new, and keep a mental
note of where I've read up to so I know what's new. RSS would make
this simpler, certainly, but I'm not sure about how I'd use it (it's
not how I currently use RSS, so I'd have to mess round with my current
setup to make it appropriate).

Email is delivered to me by default - I get anything new in my
python-dev folder, and I can skip or read the discussion as I choose.
I don't have to take action just to monitor things. (In other words,
the default is for people to see the discussions, rather than the
other way around.


A good point, but there are a couple of technical solutions to this 
problem, which, according to http://wiki.python.org/moin/TrackerDocs/, 
have already been implemented.


If you want to get email about new issues, subscribe to new-bugs- 
annou...@mail.python.org.  If you want to know about every message on 
every issue, subscribe to python-bugs-l...@mail.python.org.


But, frankly, I think it's a bad idea to subscribe to python-bugs-list 
for announcements.  The whole point here is that there is simply too 
much going on in python development for anyone to reasonably keep track 
of at a low level.  Guido himself has complained on numerous occasions 
of being too busy to monitor things closely.  A better model is to 
subscribe to new-bugs-announce and selectively pay attention to the bugs 
which are interesting to you; and, when a discussion you're involved in 
gets interesting and becomes of more general interest, raise it on 
python-dev.


(On the other hand, if you want to subscribe to get your own personal 
searchable archive, then by all means.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with process and discussions (Re: Issues with Py3.1's new ipaddr)

[glyph]

On 3 Jun, 07:08 pm, mar...@v.loewis.de wrote:

To go back to JP's original comments though: what was the right thing
for him to do, back in January, when he had these concerns?


To me, it's fairly clear: what the committer needs to get is guidance 
in

any action to take. In most cases, the set of possible actions comes
down to three:
a) reject-as-is
b) commit-as-is
c) commit-with-changes (specify changes to make)
[d) take no action at this point, until certain preconditions are met]

For d), it is common to request, to the submitter,
resubmit-with-changes, then the code needs to be reevaluated when the
submitter claims to have implemented the requested changes.


Is there a document which lists these things, and explains how it is 
desirable to communicate them?  I recently updated Twisted's equivalent 
document, adding minutae like which buttons to click on in our issue 
tracker, since that seems obvious to me but apparently wasn't obvious to 
a lot of new contributors.

In the specific case, JP didn't propose an action to take, hence it
wasn't clear (to me) whom his comment was directed to; I understood
it as "the module has these minor flaws, they should be fixed at some
point", which means "commit, then change later". This is what happened.


My reading of it suggests that he was saying "netaddr appears to be 
superior in every way, so python should include that instead.  But, if 
someone is insisting on ipaddr here are the things that could change 
about it".  The important thing here is that interpretation of the 
comment is required, so I can definitely see how you saw it the way you 
did.  There is no "-1" in his comment, and there's no documentation 
(that I'm aware of) which says that a "-1" is required, or how it will 
be used or interpreted.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 02:17 am, benja...@python.org wrote:

Backwards compatibility seems to be an issue that arises a lot here. I
think we all have an idea of it is, but we need some hard place to
point to. So here's my attempt:



PEP: 387
Title: Backwards Compatibility Policy


Thanks for getting started on this.  This is indeed a bikeshed that 
there would be less discussion around if there were a clearer PEP to 
point to.  However, this draft PEP points to the problem rather than the 
solution.  In order to really be useful this needs to answer a whole 
slew of very very specific questions, because the real problem is that 
everybody thinks they know what they mean but in fact we all have 
slightly different definitions of these words.


I've taken a stab at this myself in the past while defining a policy for 
Twisted, which is here: 
.  I think we 
might be a bit stricter than Python, but I believe our attempt to 
outline these terms specifically is worth taking a look at.


The questions that follow might seem satirical or parodic but I am 
completely serious and each of these terms really does have a variable 
definition.
* The behavior of an API *must* not change between any two consecutive 
releases.


What is "behavior"?  Software is composed of behavior.  If no behavior 
changes, then nothing can ever happen.


What is an "API"?  Are you talking about a public name in a Python 
module?  A name covered in documentation?  A name covered by a specific 
piece of documentation?  Is "subclassing" considered an API, or just 
invoking?  What about isinstance()?  What about repr()?  Are string 
formats allowed to change?  For example, certain warnings have caused 
the Twisted test suite to fail because with the introduction of warnings 
at the C level it becomes progressively harder for a Python process to 
control its own stderr stream.  I can't remember the specifics right 
now, but on more than one occasion a python upgrade caused our test 
suite to fail because the expectation was that a process would be able 
to successfully terminate without putting anything on any FD but stdout.


In order for any changes to be possible, there needs to be a clearly 
labeled mine-field: don't touch or depend on these things in your Python 
application or it *will* break every time Python is released.


What are "consecutive" releases?  If we have releases 1, 2, 3, 4, and 
behavior can't change 1->2, 2->3, or 3->4, then no change may ever be 
made.


What does "must not" mean here?  My preference would be "once there is 
agreement that an incompatible change has occurred, there should be an 
immediate revert with no discussion".  Unless of course the change has 
already been released.
* A feature cannot be removed without notice between any two 
consecutive

 releases.


Presumably removal of a feature is a change in behavior, so isn't this 
redundant with the previous point?
* Addition of a feature which breaks 3rd party libraries or 
applications should
 have a large benefit to breakage ratio, and/or the incompatibility 
should be

 trival to fix in broken code.


How do you propose to measure the benefit to breakage ratio?  How do you 
propose to define "trivial" to fix?  Most projects, including Python, 
Twisted, Django, and Zope, have an ever-increasing bug count, which 
means that trivial issues fall off the radar pretty easily.


One of the big problems here in detecting and fixing "trivial" changes 
is that they can occur in test code as well.  So if the answer is "run 
your tests", that's not much help if you can't call the actual APIs 
whose behavior has changed.  The standard library would need to start 
providing a lot more verified test stubs for things like I/O in order 
for this to be feasible.

Making Incompatible Changes
===

It's a fact: design mistakes happen.  Thus it is important to be able 
to change
APIs or remove misguided features.  This is accomplished through a 
gradual

process over several releases:

1. Discuss the change.  Depending on the size of the incompatibility, 
this could
  be on the bug tracker, python-dev, python-list, or the appropriate 
SIG.  A
  PEP or similar document may be written.  Hopefully users of the 
affected API

  will pipe up to comment.


There are a very large number of users of Python, the large percentage 
of whom do not read python-dev.  A posting on python-list is likely to 
provoke an unproductive pile-on.  I suggest a dedicated list, which 
should ideally be low traffic, "python-incompatible-announce", or 
something like that, so that *everyone* who maintains Python software 
can subscribe to it, even if they're not otherwise interested in Python 
development.
2. Add a warning [#warnings_]_.  If behavior is changing, a the API may 
gain a
  new function or method to perform the new behavior; old usage should 
raise
  the warning.  If an API is being removed, simply warn whenever it is 
enter

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 09:21 am, solip...@pitrou.net wrote:

 divmod.com> writes:


In order for any changes to be possible, there needs to be a clearly
labeled mine-field: don't touch or depend on these things in your 
Python

application or it *will* break every time Python is released.


I think the "frozen area" should be defined positively (explicit public 
APIs and
explicitly guaranteed behaviour) rather than negatively (an explicit 
"mine
field"). Otherwise, we will forget to put some important things in the 
minefield

and get bitten later when we need to change those things in a
backwards-incompatible way.


This is a false dichotomy; for core developers, the list needs to be 
exhaustive.  Everything that can change needs to be described as either 
compatible or incompatible.


However, the reason I say that the list needs to be phrased as a 
whitelist is that we have to assume *all* people writing Python code, 
who ever want to be able to upgrade Python, need to *completely* 
understand the list of things which they should not depend on.  Every 
piece of documentation they read and every API they find, the assumption 
is that it's going to be compatible and it's not something they need to 
worry about.


It's important to remember that this PEP has a "public" (python 
application programmers) and "private" (python core developer) 
interfaces, too ;-).
For example, I think it was wrong to change the name of a test-skipping 
unittest

method just so that it wouldn't clash with a method created by Twisted
subclassing unittest (besides, self.skip() was much nicer than 
self.skipTest()
;-)). Just because some class is public shouldn't prevent us to add new 
public

methods or attributes to it.


I think it would be wrong to have a blanket prohibition on such changes, 
by which I mean additions of names to public namespaces.  Twisted's own 
compatibility possibility would not forbid a similar change.  But in 
that specific case I think it was the right thing to do.  Like it or 
not, a lot of people use Twisted, a lot of people run tests with Trial, 
and we beat stdlib unittest to the punch on the 'skip' testing feature 
by a good 5 years.  We caught the change well before the release, we 
reported it and discussed it.


IMHO this is the best way to deal with incompatible changes, especially 
in the case of superclasses, given Python's subtle and complex 
inheritance semantics.  It's not feasible to provide a policy that 
somehow prohibits subclasses from adding names which may eventually be 
used on a superclass.


Projects which notice test failures with new versions of Python should 
report them so that the features can be adjusted to be compatible, 
assuming the project in question hasn't done anything in egregious 
violation of the compatibility policy (like invoking a private method). 
This lets users, system administrators, and application authors upgrade 
components individually, without worrying about the components further 
down the stack flaking out on them.  It also provides a feedback loop 
for the compatibility policy: if there are things that initially seem 
reasonable, but projects report compatibility issues when they are 
changed, they might need to be added later.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 11:11 am, solip...@pitrou.net wrote:

 divmod.com> writes:


This is a false dichotomy; for core developers, the list needs to be
exhaustive.  Everything that can change needs to be described as 
either

compatible or incompatible.


How do you enumerate "everything that can change"? It does not look 
like a
finite set to me (but perhaps I'm wrong); and certainly not like a set 
of a size

reasonable enough to be enumerated in a human-readable way :)


Sorry.  You're right.  To be clear, I mean, "everything that can change" 
as classified by some arbitrary classification system yet to be defined 
:-).


For example, the number of different programmatic entities in Python is 
pretty small: you've got classes, methods, modules, and constants.  The 
ways you can change them is also not too huge: you can add to them, 
remove them, or rename them.  I realize that given Python's flexibility 
when loading code, any system of such classification is going to have 
edge cases, but I doubt those are going to matter that much.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 11:13 am, fuzzy...@voidspace.org.uk wrote:
Just to note that Twisted (along with Bazaar) are one of the few 'good 
citizens' in the Python community running their tests on Python trunk. 
Both projects have caught incompatibilities *before* release of new 
versions which is greatly preferable to discovering them after a 
release. Thanks for this.


And thank *you* for being so responsive to the issues we've reported as 
a result of that.  It's definitely an incentive to remain engaged in 
core development :).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 02:09 pm, benja...@python.org wrote:

2009/6/19  :


On 02:17 am, benja...@python.org wrote:


I've taken a stab at this myself in the past while defining a policy 
for

Twisted



Yes, that's helpful. Thanks.


Glad you found it useful.

The questions that follow might seem satirical or parodic but I am
completely serious and each of these terms really does have a variable
definition.


And will always no matter what we do. It's how natural language works.


Well, one cannot proceed from the informal to the formal by formal 
means.  I'm pretty sure we can nail down the definitions of these terms 
for the scope of Python core development.

What is "behavior"? �Software is composed of behavior. �If no behavior
changes, then nothing can ever happen.



I mean that if you pass X and Y into a function and get Z in 2.6, then
you should be able to get Z from passing X and Y in 2.7 even if
there's a new argument that returns Z' if you pass True to it.
(Obviously, this is somewhat simplified, but I hope it helps.)


This definition only makes sense if the interesting thing about a 
function is its return value, and if the only sort of thing you have are 
functions.  What about side-effects, or exceptional conditions?  What 
about interactions with subclasses?  (Changing a class in a library from 
old-style to new-style has serious repercussions, as MRO conflicts can 
result in applications that subclass it.)
How do you propose to measure the benefit to breakage ratio?  How do 
you

propose to define "trivial" to fix? �Most projects, including Python,
Twisted, Django, and Zope, have an ever-increasing bug count, which 
means

that trivial issues fall off the radar pretty easily.


Well, if you're tests aren't failing from it, is it an incompatibility?


Well, let's say the tests do fail from it.  Right now, even Twisted 
trunk still doesn't *quite* support Python 2.6, but only on Windows, due 
to stricter checking of the 'mode' argument for files.  But this failing 
test is just not that high priority, so our recommendation is "don't use 
python 2.6 yet on Windows, 2.5 works fine".  My point is that triviality 
is a matter of perspective :).  Eventually somebody will get around to 
it, but 2.6 has been out for a while now.
There are a very large number of users of Python, the large percentage 
of
whom do not read python-dev.  A posting on python-list is likely to 
provoke
an unproductive pile-on.  I suggest a dedicated list, which should 
ideally
be low traffic, "python-incompatible-announce", or something like 
that, so
that *everyone* who maintains Python software can subscribe to it, 
even if

they're not otherwise interested in Python development.


And that won't generate a pile-on?


Well, the etiquette for that specific list could be "keep this low- 
traffic unless you have a serious problem with this change".  Or, better 
yet, make it an announce-only list: the pile-on can still happen on 
python-list, but only the results of the discussion will be announced on 
the incompatible-announce list.


The point is, the notifications about compatibility are really 
important, and sometimes people need to offer feedback about them, but 
not everyone who needs to know about the changes needs to hear about the 
feedback.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 07:06 pm, pyt...@rcn.com wrote:
Not sure why we need yet another pep on the subject.  Just update PEP 5 
if needed.


Hmm.  This is a good point; it might make sense to have a more specific 
PEP for library compatibility as opposed to language compatibility 
though, since language compatibility is necessarily a vaguer concept.
Also, I think there is a certain amount of wishful thinking here. 
Ideally, we could approve a tiny PEP with just a few bullet points and 
it would eliminate the need for all of the complicated decision making 
that we usually go through.  Ideally, we could make all decisions in 
advance of seeing the facts for a given situation.  ISTM, this pep is 
wishing away the actual complexity of making software design decisions.


This is not about making design decisions.  This is about how to treat 
the *output* of design decisions.


A really important part of this PEP, as I alluded to previously, is the 
part that tells developers what *they* should be doing if they want 
their python code to function on the next release of the interpreter.


Right now, the rule to write software that will not break with the next 
Python release is "read the minds of the python core committers and hope 
that you do not do anything with the stdlib that they don't like". 
Along with this there are several rules you can infer that are probably 
true most of the time: don't call stuff starting with "_", don't monkey- 
patch anything, don't use dynamic class replacement on objects from 
classes other than your own, don't depend on the depth of inheritance 
hierarchies (for example, no ".__bases__[0].__bases__[0]"), make sure 
your tests run without producing any DeprecationWarnings, be mindful of 
potential namespace conflicts when adding attributes to classes that 
inherit from other libraries.  I don't think all these things are 
written down in one place though.
Another sticking point about preserving features across releases arises 
if the feature itself is hazardous in some way (like have a security 
hole or some such).  The contextlib.nested() function was an example. 
(...)
We resolved the question after a long and thoughtful discussion; I 
don't think that decision making process could have been solved in 
advance by a bullet-point in a general purpose process PEP.


You are correct that nothing in Python's policy could have dictated how 
this problem could be resolved.  However, the policy can most definitely 
state how to deal with code using contextlib.nested in the interim 
before it has been changed to use the new syntax: to wit, that 
contextlib.nested has to stick around, continue to do the (broken) thing 
that it did before, and emit a DeprecationWarning indicating the new 
syntax.  The existing policy in PEP 5 already does this, but doesn't 
offer guidelines on *how* to do this for lots of different types of 
changes.  For example, how do you issue a deprecation warning for a new 
parameter you want to require application code to accept?  How do you 
change the name of a parameter, to wit, do you need to expect that all 
arguments can validly be used as keyword arguments?  How do you 
determine an appropriate stack-depth, etc?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft pep: backwards compatibility

[glyph]

On 19 Jun, 09:08 pm, benja...@python.org wrote:

2009/6/19  :

On 02:09 pm, benja...@python.org wrote:

2009/6/19 �:



�What about side-effects, or exceptional conditions? �What about
interactions with subclasses?  (Changing a class in a library from 
old-style

to new-style has serious repercussions, as MRO conflicts can result in
applications that subclass it.)



I've added a little more about this to the PEP. See what you think.


Finally had a chance to take a look.  It's a big improvement, certainly: 
much more specific.  Although I think I have a few quibbles with the 
wording.


For one thing, I don't like the use of the word "reasonable".  Everybody 
thinks *their* exception to the rules is reasonable, but nobody else's 
is.  Besides, if the BDFL thinks a change is really reasonable, do you 
think a PEP is going to stop him? :)


For another, I think it's actually a bit too strict, as worded.  The 
side-effects of a function includes the warnings that it emits; any 
method-call can have side-effects, so you can't change an algorithm *at 
all*.  The only side-effects that I think are important are the ones 
that get invoked on objects that an application gets to inject 
somewhere, or inspect later via a public API.


The word "releases" also needs to be qualified.  Most importantly, 
minor-version and micro-version releases need to be described 
distinctly.


Finally, the PEP should mention its participation in the universe of 
compatibility process PEPs.  It should describe its relationship to at 
least some of PEP 3002, 291, 5, 6, 2, 3001, and 384.

My point is that triviality is a matter of perspective :).



I understand. I think we will just have to provide guidelines for
triviality and decide on a case by case basis.


A simple litmus test, or some examples, of triviality would be helpful.

the pile-on can still happen on python-list, but
only the results of the discussion will be announced on the
incompatible-announce list.


I think that's a fine idea, but the PEP is dealing with policy as our
mailing list infrastructure is now.


Hmm... well, maybe everybody should just run their tests against Python 
trunk.  The commits list is a reliable notification mechanism for 
potentially incompatible changes ;-).  Perhaps it would be good to 
mention this specifically in the PEP?  For example, "third party 
projects may request that an incompatible change be reverted, by 
providing evidence of tests failing on x.y+1 that passed on x.y, where 
the code in question does not rely on any details not specified as 
'public' in the section above"?


If more projects did this when there *was* a problem, then it would 
actually be a lot easier to break the policy with confidence. With an 
incompatible change, you could know, "we checked it in, and nobody 
complained".  Whereas right now is nobody complains it's more likely 
that nobody is paying attention.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[glyph]

On 02:55 am, e...@trueblade.com wrote:
I really don't get this use case of multiple installers trying to 
install the same package. There's just no way that running "yum install 
twisted" and "apt-get install twisted" and "pip install twisted" are 
going to coexist with each other. The best they can do is say "a file 
I'm trying to install already exists". Why try for anything else?


Here are some use-cases.

You do "apt-get install twisted", or, rather, Twisted comes preinstalled 
on your OS.  Later, you do "pip install 
py_awesome_network_thing_system", which depends on twisted.


You've configured pip to install files into ~/.local, though, and it's 
not sure if Twisted is installed or not.  So thanks to automatic 
dependency resolution, it downloads and installs another copy.  But 
that's not what you want: you want it to use the system-installed 
Twisted (and zope.interface, and pycrypto, and pytz, etc etc) and just 
install PANTS into your .local environment.  No files will conflict, but 
it would be helpful for the package installation tool to have an API to 
inspect both the system package installation and any other places where 
packages have been installed in order to make a decision about how it 
needs to satisfy its dependencies.


If you're not as interested as I am in per-user installation of 
packages, then consider the fact that Ubuntu stores Python packages in 
/usr/lib/pythonX.Y/dist-packages now.  Python package installers should 
be able to query that before putting things into site-packages. (which I 
*believe* is now reserved for non-OS-installed packages, but it's very 
hard to find documentation on the motivation behind debian's 
manipulations of distutils.  In any event, I digress.)


You're correct that worrying about 'yum install twisted' and 'apt-get 
install twisted' is way out of scope for any Python installation system; 
unfortunately, yum and apt are likely as not to have giant, opaque 
makefiles or shell scripts attached to these packages for obscure 
linux-y reasons that (hopefully) we don't need to care about, but would 
fight with each other if you tried to combine them, totally out of scope 
with distutils.


One hopes that regardless of whatever else they're doing, these packages 
*are* providing correct installation metadata.  (Unfortunately, I can 
bet you a nickel they will screw it up in the first few python packages 
that include it; see, for example, 
.  This 
spec should try to gracefully address what to do if system packagers 
have done something unusual or wrong.)


So, the filesystem and the OS package database and the Python package 
"database" (such as it is) are definitely three distinct things, but 
they have some overlap, and there should be APIs for querying each.  If 
for no other reason so you know when Python packages shouldn't stomp on 
OS packages, and vice versa.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - Open questions

[glyph]

On 03:28 am, e...@trueblade.com wrote:
Eventually, I'd like PEP 376 to support system packagers too. So for 
example, if you did "apt-get install python-pyqt4", then running "pip 
install python-pyqt4" should return without installing anything .. as 
RECORD will be part of the .deb previously installed. As for 
generating the RECORD file, I vote for generating it during install 
time (w/ absolute paths).


I think we should explicitly not support this. What if pip and apt-get 
(or rpm, or others) install the same package in different places 
because of system conventions (/usr vs. /usr/local vs. /opt, say)? 
There's no way we're ever going to get this right, and it's not worth 
complicating our lives over it.


As I understand it, the point of the PEP is just to *provide the 
information* that the package installers need to get it right, not to 
actually implement the interaction with system package managers.
Seriously: Is there some real world use case I'm missing? Does any 
existing install system support this?


System package managers already provide .egg-info metadata, so this is 
hardly unprecedented.  The antecedent of your "this" is somewhat 
ambiguous, so I'm not sure if any package manager provides the exact 
feature that you have in mind.  If you're talking about different 
package managers installing stuff on the same system, you can look a bit 
farther afield: 'alien' allows you to install RPM and debian packages on 
the same system, with some reconciliation of dependencies and other 
metadata.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Community buildbots (was Re: User's complaints)

[glyph]

On Wed, 12 Jul 2006 10:30:17 +1000, Michael Ellerman <[EMAIL PROTECTED]> wrote:

>Well here's one I stumbled across the other day. I don't know if it's
>legit, but it's still bad PR:
>
>http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html

Having been exhorted (or maybe I mean "excoriated") by your friendly release 
manager earlier this week to post my comments and criticisms about Python here
rather than vent in random IRC chatter, I feel morally compelled to comment.

I see some responses to that post which indicate that the specific bug will be
fixed, and that's good, but there is definitely a pattern he's talking about
here, not just one issue.  I think there is a general pattern of small,
difficult to detect breakages in Python.  Twisted (and the various other
Divmod projects that I maintain) are thoroughly unit-tested, but there are
still significant issues in each Python release that require changes.

Unlike the jerk who posted that "python sucks" rant, I'm not leaving Python
because one function changed in a major release; I do expect to have to
maintain my projects myself, and major releases are "major" for a reason.  I
only wish that upgrading all my other dependencies were as easy as upgrading
Python!  I do see that the developers are working with some caution to avoid
breaking code, and attempting to consider the cost of each change.

However, although I've seen lots of discussions of what "average" code might
break when exposed to new versions of Python, these discussions tend to be
entirely hypothetical.  Do the core Python developers typically run the test
suites of major Python applications when considering major changes, such as 
Zope, Plone, TurboGears, and of course Twisted?  Not that breakages would be
considered unacceptable -- some gain is surely worth the cost -- but to
establish some empirical level of burden on the community?

I would like to propose, although I certainly don't have time to implement,
a program by which Python-using projects could contribute buildslaves which
would run their projects' tests with the latest Python trunk.  This would
provide two useful incentives: Python code would gain a reputation as
generally well-tested (since there is a direct incentive to write tests for
your project: get notified when core python changes might break it), and the
core developers would have instant feedback when a "small" change breaks more
code than it was expected to.

I can see that certain Python developers expect that some of this work is the
responsibility of the user community, and to some extent that's true, but at
least half of the work needs to be done _before_ the changes are made.  If
some Python change breaks half of Twisted, I would like to know about it in
time to complain about the implementation, rather than flailing around once
the Python feature-freeze has set in and hoping that it's nothing too serious.
For example, did anyone here know that the new-style exceptions stuff in 2.5
caused hundreds of unit-test failures in Twisted?  I am glad the change was
made, and one of our users did catch it, so the process isn't fatally broken,
but it is still worrying.

Another problem is simply that the Python developers don't have the same
very public voice that other language developers do.  It doesn't necessarily
have to be a blog, but python-dev is fast-paced and intimidating, and a
vehicle for discussion among those in the know, rather than dissimenation to
the community at large.  It's a constant source of anxiety to me that I might
miss some key feature addition to Python which breaks or distorts some key bit
of Twisted functionality (as the new-style exceptions, or recent ImportWarning
almost did) because I don't have enough time to follow all the threads here.
I really appreciate the work that Steve Bethard et. al. are doing on the
python-dev summaries, but they're still pretty dry and low level.

While the new python.org is very nice, I do note that there's no "blogs" entry
on the front page, something which has become a fixture on almost every other 
website I visit regularly.  The news page is not very personal, mainly a 
listing of releases and events.  There's no forum for getting the community 
_excited_ about new features (especially new features which are explicitly 
enabled by potential breakages), and selling them on the cool uses.  Who 
knows, maybe I'll even start using decorators syntax all over the place if I 
see them presented somewhere by someone who is very excited about the feature 
and thinks it's worthwhile, rather than as a defense on a mailing list 
against a criticism, or a simple announcement of the feature's existence.

I've seen the other side of this problem as well, so I can appreciate that it
is quite difficult to get this kind of thing right: lots of applications using
Twisted break when we change broken or deprecated APIs.  Twisted is lucky
though; it has numerous subprojects, and I maintain a half-dozen unrelated
projec

Re: [Python-Dev] Community buildbots

[glyph]

On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson <[EMAIL PROTECTED]> wrote:
>[EMAIL PROTECTED] writes:

>> For example, did anyone here know that the new-style exceptions stuff in 2.5
>> caused hundreds of unit-test failures in Twisted?  I am glad the change was
>> made, and one of our users did catch it, so the process isn't fatally broken,
>> but it is still worrying.
>
>When implementing this stuff, I could have (... snip ...)

To be clear, I agree with the decision you made in this particular case.  I
just would have appreciated the opportunity to participate in the
discussion before the betas were out and the featureset frozen.  (Of course I
*can* always do that, and some other Twisted devs watch python-dev a bit more
closely than I do, but the point is that the amount of effort required to do
this is prohibitive for the average Python hacker, whereas the time to set up an
individual buildbot might not be.)

(Aside: IMHO, the sooner we can drop old-style classes entirely, the better.
That is one bumpy Python upgrade process that I will be _very_ happy to do.
There's no way to have documentation that expresses the requirement that an 
implementation of an interface be new-style or old-style without reference to 
numerous historical accidents, which are bewildering and upsetting to people
reading documentation for the first time.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[glyph]

On Thu, 13 Jul 2006 11:29:16 -0700, Aahz <[EMAIL PROTECTED]> wrote:

>There's been some recent discussion in the PSF wondering where it would
>make sense to throw some money to remove grit in the wheels; do you think
>this is a case where that would help?

Most likely yes.  It's not a huge undertaking, and there are a lot of people out
there in the community with the knowledge of Buildbot to make this happen.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots

[glyph]

On Thu, 13 Jul 2006 23:27:56 -0500, [EMAIL PROTECTED] wrote:

>The buildbot idea sounds excellent.

Thanks.  If someone can set this up, it pretty much addresses my concerns.

>If you're concerned about noticing when a new release train is pulling out
>of the station I think it would be sufficient to subscribe to the low-volume
>python-announce mailing list.  You will see announcements about alphas and
>betas there.  Even lower volume would be a subscription to the RSS feed of
>Python announcements on the python.org front page (scroll to the bottom).

I am aware of when new releases come out :).  What I'm not aware of is what
features (may) have broken my code, and why.  As long as Python's [EMAIL 
PROTECTED]
continues to run the test suites cleanly, I am mostly unconcerned.  When it
breaks, though, I want a chance to look at the cause of the breakage, *before*
there is an alpha or beta Python release out and people are starting to write
code that depends on its new features.  Most importantly, python-dev is
extremely context-sensitive.  I want to be able to participate in the discussion
when my concerns are relevant and still reasonable to act upon.

Additionally I would like to know about these changes so that I can modify code
to support Python releases and release a compatible version of Twisted _in
advance_ of the Python release that's going to break them, so assuming users are
keeping relatively current, there won't be a window where their most recent
Twisted release will not work with the most recent Python release.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[glyph]

On Fri, 14 Jul 2006 06:46:55 -0500, [EMAIL PROTECTED] wrote:
>
>Neal> How often is the python build broken or otherwise unusable?
>
>Not very often.

I have to agree.  The effort I'm talking about is not in fixing large numbers
of problems, but simply gearing up to properly test to see if there *are*
problems.  Keep in mind though: just because the problems are small or easy
to fix doesn't mean they're not severe.  One tiny bug can prevent a program
from even starting up.

>Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I
>suspect that my usage of the language isn't all that different from most
>Python developers out there.

A huge percentage of Python developers are working with Zope, which means that
although *their* code might not be terribly "sophisticated", it is
nevertheless sitting on top of a rather large and intricate pile of implicit
dependencies on interpreter behavior.

>Neal> Is part of your point that these developers only care about
>Neal> something called "release" and they won't start testing before
>Neal> then?  If that's the case why don't we start making
>Neal> (semi-)automated alpha releases every month?

>How would that be any easier than a user setting up a read-only repository
>and svn-up-ing it once a month then using that as the default interpreter on
>that person's development machine?  I maintain interpreters for 2.3, 2.4 and
>bleeding edge at the moment.  If I need to it's fairly trivial (a symlink
>change) to fall back to the latest stable release.

>Glyph, would that sort of scheme work for you?

No.

I really think you're underestimating the sort of effort required to upgrade
Python.

First of all, I do have a job :) and it'd be very hard to make the case to an
investor that it was worth tracking down every potential bug I had to
determine whether it was a problem because I was working with an unreleased
version of Python.  This is the same reason I don't use beta versions of the
kernel or libc for development.

For that matter, I've had to avoid posting to this mailing list because even
*this* is stretching my already overburdened schedule :).

Secondly, I have to work with a few extension modules.  To name a few:
  * ctypes
  * PyPAM
  * pysqlite2
  * zope.interface (C optimizations)
  * xapian
  * pylucene
  * dspam
  * PyOpenSSL
  * PIL
  * PyCrypto
  * pyexpat
  * pygtk
  * pyvte

Recompiling all of these is a project that takes a day.  PyLucene, in fact,
I've never managed to build myself, and I can only run because there happen
to be debian packages which work (with some fiddling) on Ubuntu.  There's no
interactive warning during 'svn up' that it's time to recompile everything,
either, so I don't even know when the ABIs are going to have changed.

Even if everything works perfectly, and were perfectly automated, the compile 
itself would take a few hours.

I also test work on Windows on occasion and recompiling these things _there_
is the work of a week and a half, not to mention it requires that I be sitting
at the one machine where I have my Microsoft™ Developer™ Tools™ installed.

I made the buildbot recommendation specifically because it would centralize
the not inconsiderable effort of integrating these numerous dependencies
(presuming that I could submit a Divmod buildbot as well as a Twisted one).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots (was Re: User's complaints)

[glyph]

On Thu, 13 Jul 2006 23:39:06 -0700, Neal Norwitz <[EMAIL PROTECTED]> wrote:
>On 7/13/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:

>> a longer beta period gives *external* developers more time to catch up,
>> and results in less work for the end users.

>This is the part I don't get.  For the external developers, if they
>care about compatibility, why aren't they testing periodically,
>regardless of alpha/beta releases?  How often is the python build
>broken or otherwise unusable?

How often do you test new builds of Python against the most recent alpha of,
e.g. the Linux kernel?  This isn't just a hypothetical question: Twisted has
broken because of changes to Linux as often as it has broken due to changes in
Python :).  In Linux's case we're all lucky because *any* regressions with
existing software are considered bugs, whereas in Python's case, some breakagaes
are considered acceptable since it's more feasible to have multiple builds of
Python installed more than multiple kernels for different applications.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots

[glyph]

On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>Christopher Armstrong wrote:
>> python -U is a failure for obvious reasons and a
>> __future__ import is clearly better.
>
>I disagree.

I am surprised that you do, since I thought that Chris's conclusion was 
pretty obvious.  Python -U doesn't work, even on the standard library.

For example,

[EMAIL PROTECTED]:~% python -S -U -c 'import pickletools'
Traceback (most recent call last):
  File "", line 1, in ?
  File "/usr/lib/python2.4/pickletools.py", line 219, in ?
doc="One-byte unsigned integer.")
  File "/usr/lib/python2.4/pickletools.py", line 187, in __init__
assert isinstance(name, str)
AssertionError

This was just the first convenient example.  There are others.

A __future__ import would allow these behaviors to be upgraded module-by-module.
Right now, all -U provides is an option that can't be used on any realistically
sized program, so I don't see what the utility is.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots

[glyph]

On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy <[EMAIL PROTECTED]> wrote:

>Is the following something like what you are suggesting?

Something like it, but...

>A Python Application Testing (PAT) machine is set up with buildbot and any
>needed custom scripts.  Sometime after that and after 2.5 is released, when
>you have a version of, for instance, Twisted that passes its automated test
>suite when run on 2.5, you send it (or a URL) and an email address to PAT.
>Other developers do the same.  Periodically (once a week?), when PAT is
  ^
 "once per checkin to Python trunk"
>free and a new green development version of either the 2.5.x or 2.6
>branches is available, PAT runs the test suites against that version.  An
>email is sent for any that fail, perhaps accompanied by the concatenation
>of the relevant checkin message.  Some possible options are to select just
>one of the branches for testing, to have more than one stable version being
>tested, and to receive pass emails.

Sending email also isn't really necessary; I would just like a web page I can
look at (and draw the attention of the python core developers to).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots

[glyph]

On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:
>[EMAIL PROTECTED] wrote:
>>A __future__ import would allow these behaviors to be upgraded module-by- 
>>module.
>
>No it wouldn't.

Yes it would! :)

>__future__ works solely on the semantics of different pieces of syntax, 
>because any syntax changes are purely local to the current module.
...
>Changing all the literals in a module to be unicode instances instead of str 
>instances is merely scratching the surface of the problem - such a module 
>would still cause severe problems for any non-Unicode aware applications 
>that expected it to return strings.

A module with the given __future__ import could be written to expect that
literals are unicode instances instead of str, and encode them appropriately
when passing to modules that expect str.  This doesn't solve the problem, but
unlike -U, you can make fixes which will work persistently without having to
treat the type of every string literal as unknown.

The obvious way to write code that works under -U and still works in normal
Python is to .encode('charmap') every value intended to be an octet, and put
'u' in front of every string intended to be unicode.  That would seem to
defeat the purpose of changing the default literal type.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Community buildbots

[glyph]

On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:

>People can use [-U] to improve the Unicode support in the Python standard
>library. When they find that something doesn't work, they can study the
>problem, and ponder possible solutions. Then, they can contribute
>patches. -U has worked fine for me in the past, I contributed various
>patches to make it work better. It hasn't failed for me at all.

I guess it makes more sense as a development tool to work on zero-dependency
tools like the standard library.  Still, -Q has both a __future__ import and
a command-line option, why not -U?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dynamic module namspaces

[glyph]

On Mon, 17 Jul 2006 10:29:22 -0300, Johan Dahlin <[EMAIL PROTECTED]> wrote:

>I consider __getattribute__ a hack, being able to override __dict__ is less
>hackish, IMHO.

Why do you feel one is more "hackish" than the other?  In my experience the
opposite is true: certain C APIs expect __dict__ to be a "real" dictionary,
and if you monkey with it they won't call the overridden functions you expect,
whereas things accessing attributes will generally call through all the
appropriate Python-level APIs.

This makes sense to me for efficiency reasons and for clarity as well; if you're
trawling around in a module's __dict__ then you'd better be ready for what
you're going to get - *especially* if the module is actually a package.  Even in
"normal" python code, packages can have names which would be bound if they were
imported, but aren't yet.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem with super() usage

[glyph]

On Tue, 18 Jul 2006 09:24:57 -0400, Jean-Paul Calderone <[EMAIL PROTECTED]> 
wrote:
>On Tue, 18 Jul 2006 09:10:11 -0400, Scott Dial <[EMAIL PROTECTED]> wrote:
>>Greg Ewing wrote:
>>> Guido van Rossum wrote:
 In the world where cooperative multiple inheritance
 originated (C++), this would be a static error.

>>> I wasn't aware that C++ had anything resembling super().
>>> Is it a recent addition to the language?

>>C++ has no concept of MRO, so "super()" would be completely ambiguous.

>I think this was Greg's point.  Talking about C++ and super() is
>nonsensical.

C++ originally specified multiple inheritance, but it wasn't "cooperative" in
the sense that super is.  In Lisp, though, where cooperative method dispatch
originated, call-next-method does basically the same thing in the case where
there's no next method: it calls "no-next-method" which signals a generic
error.

http://www.lisp.org/HyperSpec/Body/locfun_call-next-method.html

However, you can write methods for no-next-method, so you can override that
behavior as appropriate.  In Python you might achieve a similar effect using a
hack like the one Greg suggested, but in a slightly more systematic way; using
Python's regular inheritance-based method ordering, of course, not bothering 
with multimethods.  Stand-alone it looks like an awful hack, but with a bit
of scaffolding I think it looks nice enough; at least, it looks like the Lisp
solution, which while potentially ugly, is complete :).

This is just implemented as a function for brevity; you could obviously use a
proxy object with all the features of 'super', including optional self,
method getters, etc.  For cooperative classes you could implement noNextMethod
to always be OK, or to provide an appropriate "null" value for a type map of
a method's indicated return value ('' for str, 0 for int, None for object, 
etc).

# ---cut here---

def callNextMethod(cls, self, methodName, *a, **k):
sup = super(cls, self)
method = getattr(sup, methodName, None)
if method is not None:
return method(*a, **k)
else:
return self.noNextMethod(methodName, *a, **k)

class NextMethodHelper(object):
def noNextMethod(self, methodName, *a, **k):
return getattr(self, "noNext_"+methodName)(*a, **k)

class A(object):
def m(self):
print "A.m"

class B(NextMethodHelper):
def m(self):
print "B.m"
return callNextMethod(B, self, "m")

def noNext_m(self):
# it's ok not to have an 'm'!
print "No next M, but that's OK!"
return None

class C(B, A):
def m(self):
print "C.m"
return callNextMethod(C, self, "m")


#>>> c = C()
#>>> c.m()
#C.m
#B.m
#A.m
#>>> b = B()
#>>> b.m()
#B.m
#No next M, but that's OK!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint

[glyph]

On Thu, 20 Jul 2006 14:57:07 -0400, "Phillip J. Eby" <[EMAIL PROTECTED]> wrote:
>While investigating the need to apply http://python.org/sf/1525766 I found
>that there was a modification to pkgutil during the need-for-speed sprint
>that affects the PEP 302 protocol in a backwards incompatible way.

It just so happens that the bug that is reported was probably reported because
I'm working on some controversial new functionality in Twisted - controversial
because it replicates the functionality that bug is about in pkgutil.  This
functionality does make some use of PEP 302 functionality :).

See 

>Specifically, PEP 302 documents that path_importer_cache always contains
>either importer objects or None.  Any code written to obtain importer
>objects is therefore now broken, because import.c is slapping False in for
>non-existent filesystem paths.

Oddly, for once I'm going to say I don't care about this change.  The code
I've written so far doesn't depend on this, and I was pretty careful to be
conservative about depending too much on the stuff described in PEP 302.  It
documents several features which don't exist (get_data, and methods in the 
"imp" module which don't exist in python2.3 or python2.4, where it was 
nominally accepted).

>There are several options as to how to proceed:

>2. Document the breakage, update PEP 302, and make everybody update their code

Personally I'd prefer it if PEP 302 were updated for a variety of reasons.
It's very hard to use as a reference for writing actual code because so many
features are "optional" or "open issues", and there's no description in the 
PEP of what their status is.

Better yet, this breakage (and other things) should be documented in the
Python reference, and the PEP should link to the documentation for different
versions, which can each describe the PEP's implementation status.  The
"importing modules" section of the library reference seems like a natural
place to put it.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] uuid tests failing on Windows

[glyph]

On Thu, 17 Aug 2006 22:06:24 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:
>Georg Brandl schrieb:
>>> Can somebody please fix that? If not, should we remove the uuid module
>>> as being immature?
>>
>> Patch #1541863 supposedly solves this.
>
>Ah, good. I think it should go in.

Uh, I may be misunderstanding here, but that patch looks like it changes that 
part of the test for test_uuid4 to stop calling uuid4 and call uuid1 instead?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] uuid tests failing on Windows

[glyph]

On Thu, 17 Aug 2006 23:58:27 +0200, "\"Martin v. Löwis\"" <[EMAIL PROTECTED]> 
wrote:

>You misunderstand indeed: the chunk reads (...)

>it currently calls uuid1, and will call uuid4 when patched.
>test_uuid4 should have never failed, except that it uses uuid1
>as the uniqueness test.

Whooops.  I must have hit the "reverse diff" button in Emacs before reading it.

Thanks for the correction.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggestion for a new built-in - flatten

[glyph]

On Fri, 22 Sep 2006 18:43:42 +0100, Michael Foord <[EMAIL PROTECTED]> wrote:

>I have a suggestion for a new Python built in function: 'flatten'.

This seems superficially like a good idea, but I think adding it to Python 
anywhere would do a lot more harm than good.  I can see that consensus is 
already strongly against a builtin, but I think it would be bad to add to 
itertools too.

Flattening always *seems* to be a trivial and obvious operation.  "I just need 
something that takes a group of deeply structured data and turns it into a 
group of shallowly structured data.".  Everyone that has this requirement 
assumes that their list of implicit requirements for "flattening" is the 
obviously correct one.

This wouldn't be a problem except that everyone has a different idea of those 
requirements:).

Here are a few issues.

What do you do when you encounter a dict?  You can treat it as its keys(), its 
values(), or its items().

What do you do when you encounter an iterable object?

What order do you flatten set()s in?  (and, ha ha, do you Set the same?)

How are user-defined flattening behaviors registered?  Is it a new special 
method, a registration API?

How do you pass information about the flattening in progress to the 
user-defined behaviors?

If you do something special to iterables, do you special-case strings?  Why or 
why not?

What do you do if you encounter a function?  This is kind of a trick question, 
since Nevow's "flattener" *calls* functions as it encounters them, then treats 
the *result* of calling them as further input.

If you don't think that functions are special, what about *generator* 
functions?  How do you tell the difference?  What about functions that return 
generators but aren't themselves generators?  What about functions that return 
non-generator iterators?  What about pre-generated generator objects (if you 
don't want to treat iterables as special, are generators special?).

Do you produce the output as a structured list or an iterator that works 
incrementally?

Also, at least Nevow uses "flatten" to mean "serialize to bytes", not "produce 
a flat list", and I imagine at least a few other web frameworks do as well.  
That starts to get into encoding issues.

If you make a decision one way or another on any of these questions of policy, 
you are going to make flatten() useless to a significant portion of its 
potential userbase.  The only difference between having it in the standard 
library and not is that if it's there, they'll spend an hour being confused by 
the weird way that it's dealing with  
rather than just doing the "obvious" thing, and they'll take a minute to write 
the 10-line function that they need.  Without the standard library, they'll 
skip to step 2 and save a lot of time.

I would love to see a unified API that figured out all of these problems, and 
put them together into a (non-stdlib) library that anyone interested could use 
for a few years to work the kinks out.  Although it might be nice to have a 
simple "flatten" interface, I don't think that it would ever be simple enough 
to stick into a builtin; it would just be the default instance of the 
IncrementalDestructuringProcess class with the most popular (as determined by 
polling users of the library after a year or so) 
IncrementalDestructuringTypePolicy.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggestion for a new built-in - flatten

[glyph]

On Fri, 22 Sep 2006 20:55:18 +0100, Michael Foord <[EMAIL PROTECTED]> wrote:

>[EMAIL PROTECTED] wrote:
>>On Fri, 22 Sep 2006 18:43:42 +0100, Michael Foord 
>><[EMAIL PROTECTED]> wrote:

>>This wouldn't be a problem except that everyone has a different idea of 
>>those requirements:).

You didn't really address this, and it was my main point.  In fact, you more or 
less made my point for me.  You just assume that the type of application you 
have in mind right now is the only one that wants to use a flatten function, 
and dismiss out of hand any uses that I might have in mind.

>If you consume iterables, and only special case strings - then none of the 
>issues you raise above seem to be a problem.

You have just made two major policy decisions about the flattener without 
presenting a specific use case or set of use cases it is meant to be restricted 
to.

For example, you suggest special casing strings.  Why?  Your guideline 
otherwise is to follow what the iter() or list() functions do.  What about 
user-defined classes which subclass str and implement __iter__?

>Sets and dictionaries are both iterable.
>
>If it's not iterable it's an element.
>
>I'd prefer to see this as a built-in, lots of people seem to want it. IMHO

Can you give specific examples?  The only significant use of a flattener I'm 
intimately familiar with (Nevow) works absolutely nothing like what you 
described.

>Having it in itertools is a good compromise.

No need to compromise with me.  I am not in a position to reject your change.  
No particular reason for me to make any concessions either: I'm simply trying 
to communicate the fact that I think this is a terrible idea, not come to an 
agreement with you about how progress might be made.  Absolutely no changes on 
this front are A-OK by me :).

You have made a case for the fact that, perhaps, you should have a utility 
library which you use in all your projects could use for consistency and to 
avoid repeating yourself, since you have a clearly defined need for what a 
flattener should do.  I haven't read anything that indicates there's a good 
reason for this function to be in the standard library.  What are the use cases?

It's definitely better for the core language to define lots of basic types so 
that you can say something in a library like "returns a dict mapping strings to 
ints" without having a huge argument about what "dict" and "string" and "int" 
mean.  What's the benefit to having everyone flatten things the same way, 
though?  Flattening really isn't that common of an operation, and in the cases 
where it's needed, a unified approach would only help if you had two 
flattenable data-structures from different libraries which needed to be 
combined.  I can't say I've ever seen a case where that would happen, let alone 
for it to be common enough that there should be something in the core language 
to support it.

>>What do you do if you encounter a function?  This is kind of a trick 
>>question, since Nevow's "flattener" *calls* functions as it encounters 
>>them, then treats the *result* of calling them as further input.
>>
>Sounds like not what anyone would normally expect.

Of course not.  My point is that there is nothing that anyone would "normally" 
expect from a flattener except a few basic common features.  Bob's use-case is 
completely different from yours, for example: he's talking about flattening to 
support high-performance I/O.

>What does the list constructor do with these ? Do the same.

>>> list('hello')
['h', 'e', 'l', 'l', 'o']

What more can I say?

>>Do you produce the output as a structured list or an iterator that works 
>>incrementally?

>Either would be fine. I had in mind a list, but converting an iterator into 
>a list is trivial.

There are applications where this makes a big difference.  Bob, for example, 
suggested that this should only work on structures that support the 
PySequence_Fast operations.

>>Also, at least Nevow uses "flatten" to mean "serialize to bytes", not 
>>"produce a flat list", and I imagine at least a few other web frameworks do 
>>as well.  That starts to get into encoding issues.

>Not a use of the term I've come across. On the other hand I've heard of 
>flatten in the context of nested data-structures many times.

Nevertheless the only respondent even mildly in favor of your proposal so far 
also mentions flattening sequences of bytes, although not quite as directly.

>I think that you're over complicating it and that the term flatten is really 
>fairly straightforward. Especially if it's clearly documented in terms of 
>consuming iterables.

And I think that you're over-simplifying.  If you can demonstrate that there is 
really a broad consensus that this sort of thing is useful in a wide variety of 
applications, then sure, I wouldn't complain too much.  But I've spent a LOT of 
time thinking about what "flattening" is, and several applications that I've 
worked on have very different ideas about how it should w

Re: [Python-Dev] PEP 355 status

[glyph]

On Fri, 29 Sep 2006 12:38:22 -0700, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>I would recommend not using it. IMO it's an amalgam of unrelated
>functionality (much like the Java equivalent BTW) and the existing os
>and os.path modules work just fine. Those who disagree with me haven't
>done a very good job of convincing me, so I expect this PEP to remain
>in limbo indefinitely, until it is eventually withdrawn or rejected.

Personally I don't like the path module in question either, and I think that 
PEP 355 presents an exceptionally weak case, but I do believe that there are 
several serious use-cases for "object oriented" filesystem access.  Twisted has 
a module for doing this:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/filepath.py

I hope to one day propose this module as a replacement, or update, for PEP 355, 
but I have neither the time nor the motivation to do it currently.  I wouldn't 
propose it now; it is, for example, mostly undocumented, missing some useful 
functionality, and has some weird warts (for example, the name of the 
path-as-string attribute is "path").

However, since it's come up I thought I'd share a few of the use-cases for the 
general feature, and the things that Twisted has done with it.

1: Testing.  If you want to provide filesystem stubs to test code which 
interacts with the filesystem, it is fragile and extremely complex to 
temporarily replace the 'os' module; you have to provide a replacement which 
knows about all the hairy string manipulations one can perform on paths, and 
you'll almost always forget some weird platform feature.  If you have an object 
with a narrow interface to duck-type instead; for example, a "walk" method 
which returns similar objects, or an "open" method which returns a file-like 
object, mocking the appropriate parts of it in a test is a lot easier.  The 
proposed PEP 355 module can be used for this, but its interface is pretty wide 
and implicit (and portions of it are platform-specific), and because it is also 
a string you may still have to deal with platform-specific features in tests 
(or even mixed os.path manipulations, on the same object).

This is especially helpful when writing tests for error conditions that are 
difficult to reproduce on an actual filesystem, such as a network filesystem 
becoming unavailable.

2: Fast failure, or for lack of a better phrase, "type correctness".  PEP 355 
gets close to this idea when it talks about datetimes and sockets not being 
strings.  In many cases, code that manipulates filesystems is passing around 
'str' or 'unicode' objects, and may be accidentally passed the contents of a 
file rather than its name, leading to a bizarre failure further down the line.  
FilePath fails immediately with an "unsupported operand types" TypeError in 
that case.  It also provides nice, immediate feedback at the prompt that the 
object you're dealing with is supposed to be a filesystem path, with no 
confusion as to whether it represents a relative or absolute path, or a path 
relative to a particular directory.  Again, the PEP 355 module's subclassing of 
strings creates problems, because you don't get an immediate and obvious 
exception if you try to interpolate it with a non-path-name string, it silently 
"succeeds".

3: Safety.  Almost every web server ever written (yes, including twisted.web) 
has been bitten by the "/../../../" bug at least once.  The default child(name) 
method of Twisted's file path class will only let you go "down" (to go "up" you 
have to call the parent() method), and will trap obscure platform features like 
the "NUL" and "CON" files on Windows so that you can't trick a program into 
manipulating something that isn't actually a file.  You can take strings you've 
read from an untrusted source and pass them to FilePath.child and get something 
relatively safe out.  PEP 355 doesn't mention this at all.

4: last, but certainly not least: filesystem polymorphism.  For an example of 
what I mean, take a look at this in-development module:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/zippath.py

It's currently far too informal, and incomplete, and there's no specified 
interface.  However, this module shows that by being objects and not 
module-methods, FilePath objects can also provide a sort of virtual filesystem 
for Python programs.  With FilePath plus ZipPath, You can write Python programs 
which can operate on a filesystem directory or a directory within a Zip 
archive, depending on what object they are passed.

On a more subjective note, I've been gradually moving over personal utility 
scripts from os.path manipulations to twisted.python.filepath for years.  I 
can't say that this will be everyone's experience, but in the same way that 
Python scripts avoid the class of errors present in most shell scripts 
(quoting), t.p.f scripts avoid the class of errors present in most Python 
scripts (off-by-one errors when looking at separators or e

Re: [Python-Dev] PEP 355 status

[glyph]

On Sun, 01 Oct 2006 13:56:53 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:

>Things the PEP 355 path object lumps together:
>   - string manipulation operations
>   - abstract path manipulation operations (work for non-existent filesystems)
>   - read-only traversal of a concrete filesystem (dir, stat, glob, etc)
>   - addition & removal of files/directories/links within a concrete filesystem
>
>Dumping all of these into a single class is certainly practical from a utility
>point of view, but it's about as far away from beautiful as you can get, which
>creates problems from a learnability point of view, and from a
>capability-based security point of view. PEP 355 itself splits the methods up
>into 11 distinct categories when listing the interface.
>
>At the very least, I would want to split the interface into separate abstract
>and concrete interfaces. The abstract object wouldn't care whether or not the
>path actually existed on the current filesystem (and hence could be relied on
>to never raise IOError), whereas the concrete object would include the many
>operations that might need to touch the real IO device. (the PEP has already
>made a step in the right direction here by removing the methods that accessed
>a file's contents, leaving that job to the file object where it belongs).
>
>There's a case to be made for the abstract object inheriting from str or
>unicode for compatiblity with existing code,

I think that compatibility can be achieved by having a "pathname" string 
attribute or similar to convert to a string when appropriate.  It's not like 
datetime inherits from str to facilitate formatting or anything like that.

>but an alternative would be to
>enhance the standard library to better support the use of non-basestring
>objects to describe filesystem paths. A PEP should at least look into what
>would have to change at the Python API level and the C API level to go that
>route rather than the inheritance route.

In C, this is going to be really difficult.  Existing C APIs want to use C 
functions to deal with pathnames, and many libraries are not going to support 
arbitrary VFS I/O operations.  For some libraries, like GNOME or KDE, you'd 
have to use the appropriate VFS object for their platform.

>For the concrete interface, the behaviour is very dependent on whether the
>path refers to a file, directory or symlink on the current filesystem. For an
>OO filesystem interface, does it really make sense to leave them all lumped
>into the one class with a bunch of isdir() and islink() style methods? Or does
>it make more sense to have a method on the abstract object that will return
>the appropriate kind of filesystem info object? 

I don't think returning different types of objects makes sense.  This sort of 
typing is inherently prone to race conditions.  If you get a "DirectoryPath" 
object in Python, and then the underlying filesystem changes so that the name 
that used to be a directory is now a file (or a device, or UNIX socket, or 
whatever), how do you change the underlying type?

>If the latter, then how would
>you deal with the issue of state coherency (i.e. it was a file when you last
>touched it on the filesystem, but someone else has since changed it to a
>link)? (that last question actually lends strong support to the idea of a
>*single* concrete interface that dynamically responds to changes in the
>underlying filesystem).

In non-filesystem cases, for example the "zip path" case, there are inherent 
failure modes that you can't really do anything about (what if the zip file is 
removed while you're in the middle of manipulating it?) but there are actual 
applications which depend on the precise atomic semantics and error conditions 
associated with moving, renaming, and deleting directories and files, at least 
on POSIX systems.

The way Twisted does this is that FilePath objects explicitly cache the results 
of "stat" and then have an explicit "restat" method for resychronizing with the 
current state of the filesystem.  None of their methods for *manipulating* the 
filesystem look at this state, since it is almost guaranteed to be out of date 
:).

>Another key difference between the two is that the abstract objects would be
>hashable and serialisable, as their state is immutable and independent of the
>filesystem. For the concrete objects, the only immutable part of their state
>is the path name - the rest would reflect the state of the filesystem at the
>current point in time.

It doesn't really make sense to separate these to me; whenever you're 
serializing or hashing that information, the "mutable" parts should just be 
discarded.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Path object design

[glyph]

On 03:14 am, [EMAIL PROTECTED] wrote:>One thing is sure -- we urgently need something better than os.path.>It functions well but it makes hard-to-read and unpythonic code.I'm not so sure.  The need is not any more "urgent" today than it was 5 years ago, when os.path was equally "unpythonic" and unreadable.  The problem is real but there is absolutely no reason to hurry to a premature solution.I've already recommended Twisted's twisted.python.filepath module as a possible basis for the implementation of this feature.  I'm sorry I don't have the time to pursue that.  I'm also sad that nobody else seems to have noticed.  Twisted's implemenation has an advantage that it doesn't seem that these new proposals do, an advantage I would really like to see in whatever gets seriously considered for adoption:*It is already used in a large body of real, working code, and therefore its limitations are known.*If I'm wrong about this, and I can't claim to really know about the relative levels of usage of all of these various projects when they're not mentioned, please cite actual experiences using them vs. using os.path.Proposals for extending the language are contentious and it is very difficult to do experimentation with non-trivial projects because nobody wants to do that and then end up with a bunch of code written in a language that is no longer supported when the experiment fails.  I understand, therefore, that language-change proposals are therefore going to be very contentious no matter what.However, there is no reason that library changes need to follow this same path.  It is perfectly feasible to write a library, develop some substantial applications with it, tweak it based on that experience, and *THEN* propose it for inclusion in the standard library.  Users of the library can happily continue using the library, whether it is accepted or not, and users of the language and standard library get a new feature for free.  For example, I plan to continue using FilePath regardless of the outcome of this discussion, although perhaps some conversion methods or adapters will be in order if a new path object makes it into the standard library.I specifically say "library" and not "recipie".  This is not a useful exercise if every user of the library has a subtly incompatible and manually tweaked version for their particular application.Path representation is a bike shed.  Nobody would have proposed writing an entirely new embedded database engine for Python: python 2.5 simply included SQLite because its utility was already proven.I also believe it is important to get this issue right.  It might be a bike shed, but it's a *very important* bike shed.  Google for "web server url filesystem path vulnerability" and you'll see what I mean.  Getting it wrong (or passing strings around everywhere) means potential security gotchas lurking around every corner.  Even Twisted, with no C code at all, got its only known arbitrary-code-execution vulnerability from a path manipulation bug.  That was even after we'd switched to an OO path-manipulation layer specifically to avoid bugs like this!I am not addressing this message to the py3k list because its general message of extreme conservatism on new features is more applicable to python-dev.  However, py3k designers might also take note: if py3k is going to do something in this area and drop support for the "legacy" os.path, it would be good to choose something that is known to work and have few gotchas, rather than just choosing the devil we don't know over the devil we do.  The weaknesses of os.path are at least well-understood.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Path object design

[glyph]

On 10:06 am, [EMAIL PROTECTED] wrote:>What a successor to os.path needs is not security, it's a better (more pythonic,>if you like) interface to the old functionality.Why?I assert that it needs a better[1] interface because the current interface can lead to a variety of bugs through idiomatic, apparently correct usage.  All the more because many of those bugs are related to critical errors such as security and data integrity.If I felt the current interface did a good job at doing the right thing in the right situation, but was cumbersome to use, I would strenuously object to _any_ work taking place to change it.  This is a hard API to get right.[1]: I am rather explicitly avoiding the word "pythonic" here.  It seems to have grown into a shibboleth (and its counterpart, "unpythonic", into an expletive).  I have the impression it used to mean something a bit more specific, maybe adherence to Tim Peters' "Zen" (although that was certainly vague enough by itself and not always as self-evidently true as some seem to believe).  More and more, now, though, I hear it used to mean 'stuff should be more betterer!' and then everyone nods sagely because we know that no filthy *java* programmer wants things to be more betterer; *we* know *they* want everything to be horrible.  Words like this are a pet peeve of mine though, so perhaps I am overstating the case.  Anyway, moving on... as long as I brought up the Zen, perhaps a particular couplet is appropriate here:  Now is better than never.  Although never is often better than *right* now.Rushing to a solution to a non-problem, e.g. the "pythonicness" of the interface, could exacerbate a very real problem, e.g. the security and data-integrity implications of idiomatic usage.  Granted, it would be hard to do worse than os.path, but it is by no means impossible (just look at any C program!), and I can think of a couple of kinds of API which would initially appear more convenient but actually prove more problematic over time.That brings me back to my original point: the underlying issue here is too important a problem to get wrong *again* on the basis of a superficial "need" for an API that is "better" in some unspecified way.  os.path is at least possible to get right if you know what you're doing, which is no mean feat; there are many path-manipulation libraries in many languages which cannot make that claim (especially portably).  Its replacement might not be.  Getting this wrong outside the standard library might create problems for some people, but making it worse _in_ the standard library could create a total disaster for everyone.I do believe that this wouldn't get past the dev team (least of all the release manager) but it would waste a lot less of everyone's time if we focused the inevitable continuing bike-shed discussion along the lines of discussing the known merits of widely deployed alternative path libraries, or at least an approach to *get* that data on some new code if there is consensus that existing alternatives are in some way inadequate.If for some reason it _is_ deemed necessary to go with an untried approach, I can appreciate the benefits that /F has proposed of trying to base the new interface entirely and explicitly off the old one.  At least that way it will still definitely be possible to get right.  There are problems with that too, but they are less severe.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Path object design

[glyph]

On 06:14 pm, [EMAIL PROTECTED] wrote:>[EMAIL PROTECTED] wrote:>>> I assert that it needs a better[1] interface because the current>> interface can lead to a variety of bugs through idiomatic, apparently>> correct usage.  All the more because many of those bugs are related to>> critical errors such as security and data integrity.>instead of referring to some esoteric knowledge about file systems that>us non-twisted-using mere mortals may not be evolved enough to under->stand,On the contrary, twisted users understand even less, because (A) we've been demonstrated to get it wrong on numerous occasions in highly public and embarrassing ways and (B) we already have this class that does it all for us and we can't remember how it works :-).>maybe you could just make a list of common bugs that may arise>due to idiomatic use of the existing primitives?Here are some common gotchas that I can think of off the top of my head.  Not all of these are resolved by Twisted's path class:Path manipulation: * This is confusing as heck:   >>> os.path.join("hello", "/world")   '/world'   >>> os.path.join("hello", "slash/world")   'hello/slash/world'   >>> os.path.join("hello", "slash//world")   'hello/slash//world'   Trying to formulate a general rule for what the arguments to os.path.join are supposed to be is really hard.  I can't really figure out what it would be like on a non-POSIX/non-win32 platform. * it seems like slashes should be more aggressively converted to backslashes on windows, because it's near impossible to do anything with os.sep in the current situation. * "C:blah" does not mean what you think it means on Windows.  Regardless of what you think it means, it is not that.  I thought I understood it once as the current process having a current directory on every mapped drive, but then I had to learn about UNC paths of network mapped drives and it stopped making sense again. * There are special files on windows such as "CON" and "NUL" which exist in _every_ directory.  Twisted does get around this, by looking at the result of abspath:   >>> os.path.abspath("c:/foo/bar/nul")   'nul' * Sometimes a path isn't a path; the zip "paths" in sys.path are a good example.  This is why I'm a big fan of including a polymorphic interface of some kind: this information is *already* being persisted in an ad-hoc and broken way now, so it needs to be represented; it would be good if it were actually represented properly.  URL manipulation-as-path-manipulation is another; the recent perforce use-case mentioned here is a special case of that, I think. * paths can have spaces in them and there's no convenient, correct way to quote them if you want to pass them to some gross function like os.system - and a lot of the code that manipulates paths is shell-script-replacement crud which wants to call gross functions like os.system.  Maybe this isn't really the path manipulation code's fault, but it's where people start looking when they want properly quoted path arguments. * you have to care about unicode sometimes.  rarely enough that none of your tests will ever account for it, but often enough that _some_ users will notice breakage if your code is ever widely distributed.  this is an even more obscure example, but pygtk always reports pathnames in utf8-encoded *byte* strings, regardless of your filesystem encoding.  If you forget to decode/encode it, hilarity ensues.  There's no consistent error reporting (as far as I can tell, I have encountered this rarely) and no real way to detect this until you have an actual insanely-configured system with an insanely-named file on it to test with.  (Polymorphic interfaces might help a *bit* here.  At worst, they would at least make it possible to develop a canonical "insanely encoded filesystem" test-case backend.  At best, you'd absolutely have to work in terms of unicode all the time, and no implicit encoding issues would leak through to application code.)  Twisted's thing doesn't deal with this at all, and it really should. * also *sort* of an encoding issue, although basically only for webservers or other network-accessible paths: thanks to some of these earlier issues as well as %2e%2e, there are effectively multiple ways to spell "..".  Checking for all of them is impossible, you need to use the os.path APIs to determine if the paths you've got really relate in the ways you think they do. * os.pathsep can be, and actually sometimes is, embedded in a path.  (again, more  of a general path problem, not really python's fault) * relative path manipulation is difficult.  ever tried to write the function to iterate two separate trees of files in parallel?  shutil re-implements this twice completely differently via recursion, and it's harder to do with a generator (which is what you really want).  you can't really split on os.sep and have it be correct due to the aforementioned windows-path issue, but that's what everybody does anyway. * os.path.split doesn't work anything like str.split.FS ma

Re: [Python-Dev] Path object design

[glyph]

On 08:14 pm, [EMAIL PROTECTED] wrote:>Argh, it's difficult to respond to one topic that's now spiraling into>two conversations on two lists.>[EMAIL PROTECTED] wrote:>(...) people have had to spend five years putting hard-to-read>os.path functions in the code, or reinventing the wheel with their own>libraries that they're not sure they can trust.  I started to use>path.py last year when it looked like it was emerging as the basis of>a new standard, but yanked it out again when it was clear the API>would be different by the time it's accepted.  I've gone back to>os.path for now until something stable emerges but I really wish I>didn't have to.You *don't* have to.  This is a weird attitude I've encountered over and over again in the Python community, although sometimes it masquerades as resistance to Twisted or Zope or whatever.  It's OK to use libraries.  It's OK even to use libraries that Guido doesn't like!  I'm pretty sure the first person to tell you that would be Guido himself.  (Well, second, since I just told you.)  If you like path.py and it solves your problems, use path.py.  You don't have to cram it into the standard library to do that.  It won't be any harder to migrate from an old path object to a new path object than from os.path to a new path object, and in fact it would likely be considerably easier.>> *It is already used in a large body of real, working code, and>> therefore its limitations are known.*>>This is an important consideration.However, to me a clean API is more>important.It's not that I don't think a "clean" API is important.  It's that I think that "clean" is a subjective assessment that is hard to back up, and it helps to have some data saying "we think this is clean because there are very few bugs in this 100,000 line program written using it".  Any code that is really easy to use right will tend to have *some* aesthetic appeal.>I took a quick look at filepath.  It looks similar in concept to PEP>355.  Four concerns:>    - unfamiliar method names (createDirectory vs mkdir, child vs join)Fair enough, but "child" really means child, not join.  It is explicitly for joining one additional segment, with no slashes in it.>    - basename/dirname/parent are methods rather than properties:>leads to () overproliferation in user code.The () is there because every invocation returns a _new_ object.  I think that this is correct behavior but I also would prefer that it remain explicit.>    - the "secure" features may not be necessary.  If they are, this>should be a separate discussion, and perhaps implemented as a>subclass.The main "secure" feature is "child" and it is, in my opinion, the best part about the whole class.  Some of the other stuff (rummaging around for siblings with extensions, for example) is probably extraneous.  child, however, lets you take a string from arbitrary user input and map it into a path segment, both securely and quietly.  Here's a good example (and this actually happened, this is how I know about that crazy windows 'special files' thing I wrote in my other recent message): you have a decision-making program that makes two files to store information about a process: "pro" and "con".  It turns out that "con" is shorthand for "fall in a well and die" in win32-ese.  A "secure" path manipulation library would alert you to this problem with a traceback rather than having it inexplicably freeze.  Obscure, sure, but less obscure would be getting deterministic errors from a user entering slashes into a text field that shouldn't accept them.>    - stylistic objection to verbose camelCase names like createDirectoryThere is no accounting for taste, I suppose.  Obviously if it violates the stlib's naming conventions it would have to be adjusted.>> Path representation is a bike shed.  Nobody would have proposed>> writing an entirely new embedded database engine for Python: python>> 2.5 simply included SQLite because its utility was already proven.>>There's a quantum level of difference between path/file manipulation>-- which has long been considered a requirement for any full-featured>programming language -- and a database engine which is much more>complex."quantum" means "the smallest possible amount", although I don't think you're using like that, so I think I agree with you.  No, it's not as hard as writing a database engine.  Nevertheless it is a non-trivial problem, one worthy of having its own library and clearly capable of generating a fair amount of its own discussion.>Fredrik has convinced me that it's more urgent to OOize the pathname>conversions than the filesystem operations.I agree in the relative values.  I am still unconvinced that either is "urgent" in the sense that it needs to be in the standard library.>Where have all the proponents of non-OO or limited-OO strategies been?This continuum doesn't make any sense to me.  Where would you place Twisted's solution on it?___
Python-Dev mailing list
Python-Dev@python.org

Re: [Python-Dev] Path object design

[glyph]

On 01:46 am, [EMAIL PROTECTED] wrote:>On 11/1/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:>This is ironic coming from one of Python's celebrity geniuses.  "We>made this class but we don't know how it works."  Actually, it's>downright alarming coming from someone who knows Twisted inside and>out yet still can't make sense of path patform oddities.Man, it is going to be hard being ironically self-deprecating if people keep going around calling me a "celebrity genius".  My ego doesn't need any help, you know? :)In some sense I was being serious; part of the point of abstraction is embedding some of your knowledge in your code so you don't have to keep it around in your brain all the time.  I'm sure that my analysis of path-based problems wasn't exhaustive because I don't really use os.path for path manipulation.  I use static.File and it _works_, I only remember these os.path flaws from the process of writing it, not daily use.>>  * This is confusing as heck:>>    >>> os.path.join("hello", "/world")>>    '/world'>>That's in the documentation.  I'm not sure it's "wrong".  What should>it do in this situation?  Pretend the slash isn't there?You can document anything.  That doesn't really make it a good idea.The point I was trying to make wasn't really that os.path is *wrong*.  Far from it, in fact, it defines some useful operations and they are basically always correct.  I didn't even say "wrong", I said "confusing".  FilePath is implemented strictly in terms of os.path because it _does_ do the right thing with its inputs.  The question is, how hard is it to remember what its inputs should be?>>    >>> os.path.join("hello", "slash/world")>>    'hello/slash/world'>>That has always been a loophole in the function, and many programs>depend on it.If you ever think I'm suggesting breaking something in Python, you're misinterpreting me ;).  I am as cagey as they come about this.  No matter what else happens, the behavior of os.path should not really change.>The user didn't call normpath, so should we normalize it anyway?That's really the main point here.What is a path that hasn't been "normalized"?  Is it a path at all, or is it some random garbage with slashes (or maybe other things) in it?  os.path performs correct path algebra on correct inputs, and it's correct (as far as one can be correct) on inputs that have weird junk in them.In the strings-and-functions model of paths, this all makes perfect sense, and there's no particular sensibility associated with defining ideas like "equivalency" for paths, unless that's yet another function you pass some strings to.  I definitely prefer this:    path1 == path2to this:    os.path.abspath(pathstr1) == os.path.abspath(pathstr2)though.You'll notice I used abspath instead of normpath.  As a side note, I've found interpreting relative paths as always relative to the current directory is a bad idea.  You can see this when you have a daemon that daemonizes and then opens files: the user thinks they're specifying relative paths from wherever they were when they ran the program, the program thinks they're relative paths from /var/run/whatever.  Relative paths, if they should exist at all, should have to be explicitly linked as relative to something *else* (e.g. made absolute) before they can be used.  I think that sequences of strings might be sufficient though.>Good point, but exactly what functionality do you want to see for zip>files and URLs?  Just pathname manipulation?  Or the ability to see>whether a file exists and extract it, copy it, etc?The latter.  See http://twistedmatrix.com/trac/browser/trunk/twisted/python/zippath.pyThis is still _really_ raw functionality though.  I can't claim that it has the same "it's been used in real code" endorsement as the rest of the FilePath stuff I've been talking about.  I've never even tried to hook this up to a Twisted webserver, and I've only used it in one environment.>>  * you have to care about unicode sometimes.>This is a Python-wide problem.I completely agree, and this isn't the thread to try to solve it.  The absence of a path object, however, and the path module's reliance on strings, exacerbates the problem.  The fact that FilePath doesn't deal with this either, however, is a fairly good indication that the problem is deeper than that.>>  * the documentation really can't emphasize enough how bad using>> 'os.path.exists/isfile/isdir', and then assuming the file continues to exist>> when it is a contended resource, is.  It can be handy, but it is _always_ a>> race condition.>>What else can you do?  It's either os.path.exists()/os.remove() or "do>it anyway and catch the exception".  And sometimes you have to check>the filetype in order to determine *what* to do.You have to catch the exception anyway in many cases.  I probably shouldn't have mentioned it though, it's starting to get a bit far afield of even this ridiculously far-ranging discussion.  A more accurate criticism might be that "the absence of a file locking syste

Re: [Python-Dev] Path object design

[glyph]

On 01:04 am, [EMAIL PROTECTED] wrote:>[EMAIL PROTECTED] wrote:>If you're serious about writing platform-agnostic>pathname code, you don't put slashes in the arguments>at all. Instead you do>>   os.path.join("hello", "slash", "world")>>Many of the other things you mention are also a>result of not treating pathnames as properly opaque>objects.Of course nobody who cares about these issues is going to put constant forward slashes into pathnames.  The point is not that you'll forget you're supposed to be dealing with pathnames; the point is that you're going to get input from some source that you've got very little control over, and *especially* if that source is untrusted (although sometimes just due to mistakes) there are all kinds of ways it can trip you up.  Did you accidentally pass it through something that doubles or undoubles all backslashes, etc.  Sometimes these will result in harmless errors anyway, sometimes it's a critical error that will end up trying to delete /usr instead of /home/user/installer-build/ROOT/usr.  If you have the path library catching these problems for you then a far greater percentage fall into the former category.>If you're saying that the fact they're strings makes>it easy to forget that you're supposed to be treating>them opaquely,That's exactly what I'm saying.>>  * although individual operations are atomic, shutil.copytree and friends >>aren't.  I've often seen python programs confused by partially-copied trees >>of files.>I can't see how this can be even remotely regarded>as a pathname issue, or even a filesystem interface>issue. It's no different to any other situation>where a piece of code can fall over and leave a>partial result behind.It is a bit of a stretch, I'll admit, but I included it because it is a weakness of the path library that it is difficult to do the kind of parallel iteration required to implement tree-copying yourself.  If that were trivial, then you could write your own file-copying loop and cope with errors yourself.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 11:45 pm, [EMAIL PROTECTED] wrote:

>I keep thinking I'd like to treat the OS as just another application,
>so that there's nothing special about it and the same infrastructure
>could be used for other applications with lots of entry level scripts.

I agree.  The motivation here is that the "OS" application keeps itself 
separate so that incorrect changes to configuration or installation of 
incompatible versions of dependencies don't break it.  There are other 
applications which also don't want to break.

This is a general problem with Python, one that should be solved with a 
comprehensive parallel installation or "linker" which explicitly describes 
dependencies and allows for different versions of packages.  I definitely don't 
think that this sort of problem should be solved during the *standardization* 
process - that should just describe the existing conventions for packaging 
Python stuff, and the OS can insulate itself in terms of that.  Definitely it 
shouldn't be changed as part of standardization unless the distributors are 
asking for it loudly.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 09:34 am, [EMAIL PROTECTED] wrote:

>There's another standard place that is searched on MacOS: a per-user
>package directory ~/Library/Python/2.5/site-packages (the name "site-
>packages" is a misnomer, really). Standardising something here is
>less important than for vendor-packages (as the effect can easily be
>gotten by adding things to PYTHONPATH) but it has one advantage:
>distutils and such could be taught about it and provide an option to
>install either systemwide or for the current user only.

Yes, let's do that, please.  I've long been annoyed that site.py sets up a 
local user installation directory, a very useful feature, but _only_ on OS X.  
I've long since promoted my personal hack to add a local user installation 
directory into a public project -- divmod's "Combinator" -- but it would 
definitely be preferable for Python to do something sane by default (and have 
setuptools et. al. support it).

I'd suggest using "~/.local/lib/pythonX.X/site-packages" for the "official" 
UNIX installation location, since it's what we're already using, and ~/.local 
seems like a convention being slowly adopted by GNOME and the like.  I don't 
know the cultural equivalent in Windows - "%USERPROFILE%\Application 
Data\PythonXX" maybe?

It would be nice if site.py would do this in the same place as it sets up the 
"darwin"-specific path, and to set that path as a module global, so packaging 
tools could use "site.userinstdir" or something.  Right now, if it's present, 
it's just some random entry on sys.path.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 29 Nov, 11:49 pm, [EMAIL PROTECTED] wrote:

>On Nov 29, 2006, at 5:18 AM, [EMAIL PROTECTED] wrote:

>> I'd suggest using "~/.local/lib/pythonX.X/site-packages" for the
>> "official" UNIX installation location, ...

>+1 from me also for the concept.  I'm not sure I like ~/.local though
>- -- it seems counter to the app-specific dot-file approach old
>schoolers like me are used to.  OTOH, if that's a convention being
>promoted by GNOME and other frameworks, then I don't have too much
>objection.

Thanks.  I just had a look at the code in Combinator which sets this up and it 
turns out it's horribly inconsistent and buggy.  It doesn't really work on any 
platform other than Linux.  I'll try to clean it up in the next few days so it 
can serve as an example.

GNOME et. al. aren't promoting the concept too hard.  It's just the first 
convention I came across.  (Pardon the lack of references here, but it's very 
hard to google for "~/.local" - I just know that I was looking for a convention 
when I wrote combinator, and this is the one I found.)

The major advantage ~/.local has for *nix systems is the ability to have a 
parallel *bin* directory, which provides the user one location to set their 
$PATH to, so that installed scripts work as expected, rather than having to 
edit a bunch of .foorc files to add to your environment with each additional 
package.  After all, what's the point of a per-user "install" if the software 
isn't actually installed in any meaningful way, and you have to manually edit 
your shell startup scripts, log out and log in again anyway?  Another nice 
feature there is that it uses a pre-existing layout convention (bin lib share 
etc ...) rather than attempting to build a new one, so the only thing that has 
to change about the package installation is the root.

Finally, I know there are quite a few Python developers out there already using 
Combinator, so at least there it's an established convention :).

>I also think that setuptools has the potential to be a big
>improvement here because it's much easier to install and use egg
>files than it is to get distutils to DTRT with setup.py.  (I still
>detest the command name 'easy_install' but hey that's still fixable
>right? :).  What might be nice would be to build a little more
>infrastructure into Python to support eggs, by say adding a default
>PEP 302 style importer that knows how to search for eggs in
>'nests' (a directory containing a bunch of eggs).

One of the things that combinator hacks is where distutils thinks it should 
install to - when *I* type "python setup.py install" nothing tries to insert 
itself into system directories (those are for Ubuntu, not me) - ~/.local is the 
*default* install location.  I haven't managed to make this feature work with 
eggs yet, but I haven't done a lot of work with setuptools.

On the "easy_install" naming front, how about "layegg"?

>What if then that importer were general enough (...)

These all sound like interesting ideas, but they're starting to get pretty far 
afield - I wish I had more time to share ideas about packaging, but I know too 
well that I'm not going to be able to back them up with any implementation 
effort.

I'd really like Python to use the ~/.local/bin / ~/.local/lib convention for 
installing packages, though.___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 12:34 am, [EMAIL PROTECTED] wrote:

>The whole concept of "hidden" files seems ill-
>considered to me, anyway. It's too easy to forget
>that they're there. Putting infrequently-referenced
>stuff in a non-hidden location such as ~/local
>seems just as good and less magical to me.

Something like "~/.local" is an implementation detail, not something that 
should be exposed to non-savvy users.  It's easy enough for an expert to "show" 
it if they want to - "ln -s .local local" - but impossible for someone more 
naive to hide if they don't understand what it is or what it's for.  (And if 
they try, by clicking a checkbox in Nautilus or somesuch, *all* their installed 
software breaks.)  This approach doesn't really work unless you have good 
support from the OS, so it can warn you you're about to do something crazy.

UI designers tend to get adamant about this sort of thing, but I'll admit they 
go both ways, some saying that everything should be exposed to the user, some 
saying that all details should be hidden by default.  Still, in the more recent 
UNIX desktops, the "let's hide the things that the user shouldn't see and just 
work really hard to make them work right all the time" camp seems to be winning.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 04:11 am, [EMAIL PROTECTED] wrote:
>On Wednesday 29 November 2006 22:20, [EMAIL PROTECTED] wrote:
> > GNOME et. al. aren't promoting the concept too hard.  It's just the first
> > convention I came across.  (Pardon the lack of references here, but it's
> > very hard to google for "~/.local" - I just know that I was looking for a
> > convention when I wrote combinator, and this is the one I found.)
>
>~/.local/ is described in the "XDG Base Directory Specification":
>
>http://standards.freedesktop.org/basedir-spec/latest/

Thanks for digging that up!  Not a whole lot of meat there, but at least it 
gives me some env vars to set / check...

> > On the "easy_install" naming front, how about "layegg"?
>
>Actually, why not just "egg"?

That works for me.  I assumed there was some other reason the obvious answer 
hadn't been chosen :).___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 04:36 am, [EMAIL PROTECTED] wrote:

>easy_install uses the standard distutils configuration system, which means 
>that you can do e.g.

Hmm.  I thought I knew quite a lot about distutils, but this particular nugget 
had evaded me.  Thanks!  I see that it's mentioned in the documentation, but I 
never thought to look in that section.  I have an aversion to .ini files; I 
tend to assume there's always an equivalent Python expression, and it's better. 
 Is there an equivalent Python API in this case?

I don't know if this is a personal quirk of mine, or a reinforcement of Talin's 
point about the audience for documentation documentation.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python and the Linux Standard Base (LSB)

[glyph]

On 05:37 pm, [EMAIL PROTECTED] wrote:
>Perhaps "pyinstall"?

Keep in mind that Python packages will still generally be *system*-installed 
with other tools, like dpkg (or apt) and rpm, on systems which have them.  The 
name of the packaging system we're talking about is called either "eggs" or 
"setuptools" depending on the context.  "pyinstall" invites confusion with "the 
Python installer", which is a different program, used to install Python itself 
on Windows.

It's just a brand.  If users can understand that "Excel" means "Spreadsheet", 
"Outlook" means "E-Mail", and "GIMP" means "Image Editor", then I think we 
should give them some credit on being able to figure out what the installer 
program is called.

(I don't really care that much in this particular case, but this was one of my 
pet peeves with GNOME a while back.  There was a brief change to the names of 
everything in the menus to remove all brand-names: "Firefox" became "Web 
Browser", "Evolution" became "E-Mail", "Rhythmbox" became "Music Player".  I 
remember looking at my applications menu and wondering which of the 3 "music 
players" that I had installed the menu would run.  Thankfully this nonsense 
stopped and they compromised on names like "Firefox Web Browser" and "GIMP 
Image Editor".)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?!

[glyph]

On 11:17 pm, [EMAIL PROTECTED] wrote:
>On 12/11/06, Jim Jewett <[EMAIL PROTECTED]> wrote:
>> On 12/8/06, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>> > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971
>> > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz
>> > (47,898 hits). Are these numbers real?
>>
>> Why wouldn't it be?
>
>Just because in the past the ratio of downloads for a particular
>version was always about 70% Windows vs. 30% source. Now it seems
>closer to 90/10.

Personally speaking, since switching to Ubuntu, I've been so happy with the 
speed of releases and the quality of packaged Python that I haven't downloaded 
a source release from python.org in over a year.  If I need packages, they're 
already installed.  If I need source from a release, I 'apt-get source' to 
conveniently install it from a (very fast) ubuntu mirror.  When I need 
something outside the Ubuntu release structure, it's typically an SVN trunk 
checkout, not a release tarball.

I don't know what Ubuntu's impact in the general user community has been, but 
it seems that the vast majority of python developers I interact with on a 
regular basis have switched.  I wouldn't be surprised if this were a major part 
of the impact.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 07:42 pm, [EMAIL PROTECTED] wrote:
>On 1/10/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote:
>>
>><"Anthony Baxter">
>> > Comments? What else should get warnings?
>>
>>It is my strong preference that we not go down this path.
>>Instead, the 2.6 vs 3.0 difference analysis should go in an
>>external lint utility.

>Having Python 2.6 optionally warn for
>3.0-compatibility is a lot easier for the average developer than having a
>separate tool or a separately compiled Python.

I could not possibly agree more.

Given the highly dynamic nature of Python, such a tool will *at best* catch 
only the most egregious uses of deprecated features.  Backticks are easy enough 
to find, but the *only* way that I can reasonably imagine migrating a body of 
code like Twisted (or any non-trivial Python library) would be having a way to 
examine the warning output of its test suite.

I am sper +1 on the warnings for 2.6, as well as forward-compatibility at 
some point in the 2.x series for new syntax.  Without the ability to bridge 
2.x->3.0 during some interim period, I can say for sure that Twisted _will not_ 
migrate to 3.0, ever.  We are really a small project and just don't have the 
manpower to maintain two overlapping but mutually incompatible codebases.

I've been assuming for some time that the only hope for Py3k compatibility 
within Twisted would be using PyPy as a translation layer.  With the addition 
of runtime compatibility warnings, it might be feasible that we could run on 
the "bare metal" (ha ha) of Python3's VM.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 10 Jan, 11:10 pm, [EMAIL PROTECTED] wrote:
>On 10/01/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>>I've been assuming for some time that the only hope for Py3k compatibility
>>within Twisted would be using PyPy as a translation layer.

>Does this ring as many warning bells for me as it does for others? I
>know very little about the current state of PyPy, but I read your
>comment as implying that you expect Twisted to be unavailable (at some
>level or other) for users of Py3K. Is that right?

This entirely depends on the migration path available.  My understanding of the 
situation is that Py3K is a different, incompatible language.  In that sense, 
Twisted will only be as unavailable to users of Py3K as it is unavailable to 
users of Ruby, Perl, or any other dynamic-but-not-Python language.

If the 2.x series grows deprecation warnings and gradually has all of py3k's 
features backported, such that there is a smooth transitional period of a few 
years where we can support one or two versions of 2.x and a version of 3.x, 
then eventually perhaps it will be worthwhile.

Alternately, if they really aren't compatible *but* there is such a compelling 
featureset drawing other developers that there's really a critical mass of 
projects that migrate to 3.x and stop maintaining their 2.x counterparts, it 
might be worthwhile to jump the chasm to follow them.  At a minimum, for it to 
be worthwhile for Twisted to attempt this if the following packages are all 
moved to 3.x:

 - pyopenssl
 - pygtk
 - pysqlite
 - PyObjC
 - win32all
 - zope interface (and therefore, probably, all of Zope)
 - pycrypto
 - pypam
 - pygame
 - pylucene

Those projects that do move I'm sure will take the "opportunity" to 
incompatibly break all of their APIs as well, since no existing code will 
actually use them, exacerbating the problems involved in porting.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 11 Jan, 08:22 pm, [EMAIL PROTECTED] wrote:
>On 1/11/07, James Y Knight <[EMAIL PROTECTED]> wrote:
>> If the goal is really to have Py 3.0 be released later this year,

>There will certainly be demand for an asynchronous server in 3.0,

To flip the question around: there might be a demand for Twisted in 3.0, but 
will there be a demand for 3.0 in Twisted?  It might just be easier for 
everyone concerned to just continue maintaining 2.x forever.  I have yet to see 
a reason why, other than continued maintenance, 3.0 would be a preferable 
development platform.

>So the two projects will operate independently, and the 3.0 one may be
>smaller and less ambitious than Twisted.  But if the need is there it
>will be written.

It is quite likely that someone else will write some completely different code 
for python 3.0 that calls select().  I hadn't considered that the goal of 3.0 
was to *discover* these people by alienating existing Python developers - 
that's crafty!  If so, though, you'll have to figure out a way to stop Anthony 
from providing all this compatibility stuff.  He might make it too attractive 
for us to continue development on future versions :).

>How did Perl 4 and Perl 5 handle the situation?  I basically waited
>2-3 years after Perl 5 came out, then started programming the new way.
> If it mattered (it didn't), I would have tied my applications
>specifically to Perl 4.

I handled the Perl 4 to 5 transition by dropping Perl and moving to Python, 
because if I was going to port all my code to another language I wanted to at 
least port to a better language.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 01:12 am, [EMAIL PROTECTED] wrote:
>On Friday 12 January 2007 06:09, James Y Knight wrote:
>> On Jan 10, 2007, at 6:46 PM, Benji York wrote:
>> > Paul Moore wrote:
>> >> How many other projects/packages anticipate *not* migrating to
>> >> Py3K, I wonder?
>> >
>> > I certainly can't speak for the project as a whole, but I
>> > anticipate a fair bit of work to port Zope 3 (100+ KLOC) to
>> > Python 3.0.
>>
>> I (another Twisted developer, among other hats I wear) am also
>> very worried about the Python 3.0 transition.
>
>I'm plan to try and make the transition as painless as possible. A
>goal I have is to try and make it possible to write code that works
>in both 2.6 and 3.0. Obviously 2.6 will be backwards-compatible
>with previous versions, but I'd like to see it get either a command
>line option or a from __future__ statement to enable compatibility
>with 3.0-isms, like the dict.items change.

This is all very encouraging.

Please stick with a __future__ statement if at all possible though.  The 
biggest challenge in migration is to reduce the impact so that it can be done 
partially in a real system.

If you have a module X wants to be "3.0 safe" which imports a module Y from a 
different developer that is not, command line options for compatibility might 
not help at all.  A __future__ statment in X but not in Y, though, would allow 
for a smooth transition.

I can see how that would be tricky for things like dictionary methods, but it 
does seem doable.  For example, have a has_key descriptor which can raise an 
AttributeError if the globals() of the calling stack frame has been marked in 
some way.

>> Basically: my plea is: please don't remove the old way of doing
>> things in Py 3.0 at the same time as you add the new way of doing
>> things.
>
>If there was a way to make 2.6 as compatible as possible with 3.0,
>would this make life less painful? Obviously there'd have to be
>breakages in a backwards direction, but I'd hope it would make it
>easier to go forward. Some things should also be OK to backport to
>2.6 - for instance, I can't see an obvious reason why 2.6 can't
>support "except FooError as oopsie" as an alternate spelling of
>"except FooError, oopsie".

You happen to have hit my favorite hot-button 3.0 issue right there :).  I 
don't care about backticks but sometimes I _do_ have to catch exceptions.

>Similarly, where the stdlib has been shuffled around, there should
>be shims in 2.6 that allow people to work with the new names.

This part I wouldn't even mind having to write myself.  It would certainly be 
good to have somewhere more official though.

>I don't think waiting for 2.7 to make the compatibility work is a
>workable approach - we're something like 2.5-3 years away from a
>2.7 release, and (optimistically) 12-18 months from a 3.0 final.
>That leaves a window of 1.5-2 years where we are missing an
>important tool for people.

It would be nice if the versioning would actually happen in order.

>> [1] Unless of course there's a perfect automated conversion
>> script that can generate the 3.X compatible source code from the
>> 2.X compatible source code.
>
>I doubt that the 2to3 script will be perfect, but hopefully it can
>get most things. I can't see it easily fixing up things like
>
>check = mydict.has_key
>...
>if check(foo):

Currently a common idiom, by the way, used throughout the lower levels of 
Twisted.

>This is why I also want to add Py3kDeprecationWarning to 2.6.
>
>On that note, I'd like to see changes like the mass-change to the
>stdlib in the 3.0 branch that changed raise A, B into raise A(B)
>applied to the trunk. This makes it much easier to apply patches to
>both the 3.0 branch and the trunk. Similar changes should be
>applied to remove, for instance, use of <> and dict.has_key from
>the stdlib. Simply put, I'd like the stdlib between 2 and 3 to be
>as similar as possible.

It would be nice if the stdlib could be used as a case study - if the 3 stdlib 
tests can pass on some version of 2 (or vice versa) that should be a minimum 
bar for application portability.

Maybe a better way to handle the similarity is that the reorganized stdlib 
should simply be available as a separate piece of code on 2.x?  That would 
allow 2.x to use any new features added and distinguish between code which had 
been moved to use new stdlib APIs and that which hadn't.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 07:56 am, [EMAIL PROTECTED] wrote:

>Additionally, without a 2.x<->3.x upgrade path 3.x is essentially a
>new language, having to build a new userbase from scratch. Worse
>yet, 2.x will suffer as people have the perception "Python 2?
>That's a dead/abandoned language"

It's worse than that.  This perception has _already_ been created.  I already 
have heard a few folks looking for new languages to learn choose Ruby over 
Python and give Py3K as a reason.  "Isn't Python going to be totally different 
in a few years anyway?  I'll just wait until then, seems like a waste of time 
to learn it now."

Given Ruby's own checkered history of compatibility, I don't think this is an 
_accurate_ perception, but it is the nature of perception to be inaccurate.

If the plan is to provide a smooth transition, it would help a lot to have this 
plan of foward and backward compatibility documented somewhere very public.  
It's hard to find information on Py3K right now, even if you know your way 
around the universe of PEPs.

FWIW, I also agree with James that Python 3 shouldn't even be released until 
the 2.x series has reached parity with its feature set.  However, if there's 
continuity in the version numbers instead of the release dates, I can at least 
explain to Twisted users that we will _pretend_ they are released in the order 
of their versions.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 10:12 am, [EMAIL PROTECTED] wrote:

>For practical reasons (we have enough work to be getting on with) PyPy
>is more-or-less ignoring Python 2.5 at the moment.  After funding and
>so on, when there's less pressure, maybe it will seem worth it.  Not
>soon though.

I think I know what you mean from previous conversations, but the context of 
the question makes the answer ambiguous.  Are you ignoring 2.5 in favor of 2.4, 
or 3.0?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 11:22 am, [EMAIL PROTECTED] wrote:

>> FWIW, I also agree with James that Python 3 shouldn't even be
>> released until the 2.x series has reached parity with its feature
>> set.  However, if there's continuity in the version numbers
>> instead of the release dates, I can at least explain to Twisted
>> users that we will _pretend_ they are released in the order of
>> their versions.
>
>I'm not sure what "parity with it's feature set" means.

I don't either!  "Parity with it is feature set"?  I can't even parse that!  ;-)

By "parity with *its* feature set", though, I meant what you said here:

>I do hope that it's _possible_ to work in a version of the language that works 
>in both 2.6+ and 3.0+, even if under the hood there are differences.

In order to do this, everything that has been changed in 3.0 has to have some 
mechanism for working both ways in some 2.x release.  I phrased this as its 
"feature set" because I am not aware of any new functionality in 3.0 that 
simply isn't available 2.x - everything I've seen are "cleanups", which expose 
basically the same features as 2.5.

If there are some real new features, then I am in error.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The bytes type

[glyph]

On 06:49 pm, [EMAIL PROTECTED] wrote:

>I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
>The whole Py3.0 project is about eliminating cruft and being free of the
>bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
>goes against that philosophy.

Emotionally charged like "cruft" and "garbage" are obscuring the issue.

Let's replace them with equivalents charged in the opposite direction:

"I think we should draw a line in the sand and resolve not to compatibility-up 
Py2.6.  The whole Py3.0 project is about eliminating useful libraries and being 
free of the bonds of working software.  Adding non-essential 
forward-compatibility to Py2.6 goes against that philosophy."

The benefit (to me, and to many others) of 3.x over 2.x is the promise of more 
future maintenance, not the lack of cruft.  In fact, if I made a list of my 
current top ten problems with Python, "cruft" wouldn't even make it in.  There 
is lots of useful software that will not work in the 3.0 series, and without 
forward compatibility there is no way to get there from here.

As Guido said, if 3.0 is going to break compatibility, that burdens the 2.x 
series with the need to provide transitional functionality.  The upgrade path 
needs to be available in one version or the other, or 2.x needs to be 
maintained forever.  You can't have it both ways.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 09:04 pm, [EMAIL PROTECTED] wrote:

>I'm wondering if we might be going the wrong way about warning about
>compatibility between 2.x and 3.x. Perhaps it might be better if the 3.0
>alpha had a 2.x compatibility mode command-line flag, which is removed late
>in the beta cycle.

Please, no.

I don't think command-line flags are going to help much, because the problem is 
not the complexity of downloading and compiling py3k, it is the complexity of 
porting software that has modules from a variety of different sources.

More importantly however, I'm not even going to be looking into porting 
anything to py3k until a year or so after its release - and that's only if 2.6 
has given me some tools to deal with the transition.  The reason I am posting 
to this thread is that I had *written off py3k entirely* before Anthony started 
making noises about forward compatibility.  The support cycle of Ubuntu Dapper 
makes it likely that Twisted will be supporting Python 2.4 until at least 2011. 
 I assume 2.5 will last a year after that, so assuming 2.6 has perfect forward 
compatibility, we'll be looking at a py3k port in early 2013.  Features 
available in the beta releases aren't going to help me there.

Plus, this defeats the whole purpose of py3k.  Why break compatibility if 
you're going to include a bunch of gross compatibility code?  Compatibility has 
already been broken in the py3k branch, there's no point in putting it back 
there.  Adding it to 2.x might make sense though.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file(file)

[glyph]

On 12:37 am, [EMAIL PROTECTED] wrote:

>For security reasons I might be asking for file's constructor to be
>removed from the type for Python source code at some point (it can be
>relocated to an extension module if desired).  By forcing people to go
>through open() to create a file object you can more easily control
>read/write access to the file system (assuming the proper importation
>of extension modules has been blocked).  Not removing the constructor
>allows any code that has been explicitly given a file object but not
>open() to just get the class and call the constructor to open a new
>file.

This is a general problem with type access.  Secure versions of any type should 
not allow access to the type period.  It is hardly unique to files, and is not 
limited to constructors either.  How do you, e.g., allow a restricted piece of 
code write access to only a specified area of the filesystem?

More importantly, given the random behavior that open() will be growing 
(opening sockets?  dynamic dispatch on URL scheme???) file() will likely remain 
a popular way to be sure you are accessing the filesystem.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file(file)

[glyph]

On 02:42 am, [EMAIL PROTECTED] wrote:

>Wrapper around open() that does proper checking of its arguments.  I
>will be discussing my security stuff at PyCon if you are attending and
>are interested.

I am both, so I guess I'll see you there :).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater

[glyph]

On 08:19 am, [EMAIL PROTECTED] wrote:
>Georg Brandl schrieb:
>>> If Python 3.0 was simply a release which removed deprecated features,
>>> there would clearly be no issue. I would update my code in advance of
>>> the 3.0 release to not use any of those features being removed, and
>>> I'm all set. But that's not what I'm hearing. Python 3 is both adding
>>> new ways to do things, and removing the older way, in the same
>>> version, with no overlap. This makes me very anxious.
>>
>> It has always been planned that in those cases that allow it, the new way to 
>> do
>> it will be introduced in a 2.x release too, and the old way removed only in 
>> 3.x.
>
>What does that mean for the example James gave: if dict.items is going
>to be an iterator in 3.0, what 2.x version can make it return an
>iterator, when it currently returns a list?
>
>There simply can't be a 2.x version that *introduces* the new way, as it
>is not merely a new API, but a changed API.

The API isn't changed.  It's just dependent on its execution context in a 
confusing way.  That difference right now is being reflected as "2.x VM" vs. 
"3.0 VM" but there are other ways to reflect it more explicitly.  It would 
certainly be possible to have:

   from __future__ import items_is_iter

be the same as:

   __py3k_compat_items_is_iter__ = True

and have the 2.x series' items() method check the globals() of the calling 
scope to identify the return value of items() in that particular context.

If the actual data structures of module dictionaries and stack objects are too 
expensive there are other, similar things that could be done at the C level.  
This implementation strategy is just the obvious thing that occurred to me 
after maybe 2 minutes of consideration.  I'm sure someone more familiar with 
the internals of Python could come up with something better.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com