Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Dirkjan Ochtman]

On Fri, Jul 3, 2009 at 01:01, Mark Hammond wrote:
> Although this has come up in the past, I don't recall a resolution.
>
> What is your plan to handle svn:eol-style?  We have some files in the tree
> which need that support and it isn't clear to me how that would work with
> the existing win32text extension provided with current mercurial releases.
>  (I've an outstanding patch to hg which should address some of these issues,
> but without the 'rules' being versioned I fear that would still fall short.)

What files would need what? Are there any files that really need to be
\r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember
one file was discussed separately, but I think the outcome there was
that it could just always be \r\n (since it wasn't used at all on
non-Windows platforms). Anyway, knowing specific requirements (or
where to find them) would help here.

> Even more generally, how will you suggest Windows users work?  Will local
> files, in general, have windows line endings or unix?  If the latter, will
> there be hooks in-place to prevent editors on Windows 'accidently' mixing
> eol styles?  If so, this cycles back to the first question - how would we
> know which files get treated that way?

There will be a server-side hook to check whitespace. People will also
be able to install it for commit-time.

I think just using \n by default everywhere is a good default (though
I almost always use Windows client machine, I do all nearly all of my
development through a terminal on several Linux boxen), except where
it isn't. People who want to use can set up the win32text stuff to get
\r\n on Windows if they feel they need that -- we can provide
information about that in the dev FAQ (although it would be nice if
someone else who was more familiar with it -- like yourself! -- would
write it).

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Łukasz Langa]

On 2009-06-22 at 14:23, Tarek Ziadé wrote:


Hello,

We have polished out PEP 376 and its code prototype at Distutils-SIG.
It seems to fullfill now all the requirements,
so I am mailing it here again, for a new round of feedback, if needed.



Hello.
I have read the current version of the PEP from trunk and would like  
to make some comments about it.


The .egg-info as a directory

At our company, after much fiddling about the current distutils and  
setuptools, we have developed a solution that enables us to serve  
pseudo-eggs. What I mean by that is that by plugging into the PEP 302  
infrastructure, we were able to make non-filesystem-based repositories  
behave like `.egg` files. There are a couple of use cases for that  
implementation, the most important ones being:
* being able to deploy encrypted and sealed blobs with a binary  
loader, to discourage fiddling around the source code by its users
* being able to serve modules from a database back-end instead of the  
filesystem


This solution depends on the `EGG-INFO` to be fetched from the `.egg`  
itself. We wouldn't want to use a static filesystem-based folder  
because it breaks the encryption use-case by revealing the  
`SOURCES.txt` information. It also breaks the database back-end  
usecase by presenting static versioning information whereas the whole  
point of using a database backed "egg" is to have the newest version  
installed at all times.


Moreover, the proposed ``egginfo_dirname()`` routine is a step-back  
from the ``pkg_resources`` approach where we don't enforce resources  
to reside on a traditional filesystem.


The uninstall feature
=
This is a great feature but I don't seem to understand why there seems  
to be a consensus here that it's good for it not to even warn about  
dependencies, let alone manage them. ``easy_install`` and co. does  
manage dependencies while installing, why shouldn't it be symmetrical  
in that regard? Moreover, wouldn't dependency management be useful  
while using ``easy_install`` to *upgrade* an installation of a package?


The second issue is that while having ``python -m  
distutils.uninstall`` is the preferred way to go, I would argue that  
enabling an ``uninstall`` entry-point in ``setup.py`` is desirable as  
well, if only for retaining an intuitive symmetrical system where I  
can do ``python setup.py install`` and ``python setup.py uninstall``  
as well. I'm not forced to download the sources just to uninstall the  
package (``distutils.uninstall`` covers this use case) but if I do  
have the source code available, it would feel very natural to use the  
``setup.py`` provided.


There already is the precedent of ``setup.py develop --uninstall``.  
Having a ``setup.py uninstall`` could handle this use case as well.


More high-level remarks
===
This isn't probably the best place to cover this but may serve as a  
rationale for the above .egg-info problems we have with the proposed  
changes.


It's great that you push the metadata functionality to core distutils.  
At the same time, it would be feasible to think over the whole `egg`  
concept. We would argue that having the `egg` format as a container  
more-less agnostic to the underlying data storage structure would be  
**very** useful. Imagine installable `egg` support packages which  
would enable you to:

* use other compression formats beside ZIP
* use other means of module storage beside the filesystem
* use sealed and encrypted archives for deployment
* use programmatic `egg`s for licensing, etc.
* ``easy_install`` from different protocols, archive formats, etc.

Decoupling the `egg` format from the concrete "zip-based" or  
"directory-based" implementations is a step forward in that direction.  
In that regard, the solution PEP 376 is proposing isn't ultimately  
solving the issues we're having.


Thanks for your time.

--
Best regards,
Łukasz Langa
Senior Developer
STX-Next Sp. z o.o.
tel: +48 791 080 144
skype: lukaszlanga

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Mercurial migration: progress report (PEP 385)

[Stephen J. Turnbull]

Dirkjan Ochtman writes:

 > Mercurial has two basic ways of using branches: cloned branches, where
 > each branch is kept in a separate repository, and named branches,
 > where each revision keeps metadata to note on which branch it belongs.
 > The former makes it easier to distinguish branches, at the expense of
 > requiring more disk space on the client. The latter makes it a little
 > easier to switch between branches, but often has somewhat unintuitive
 > results for people (though this has been getting better in recent
 > versions of Mercurial).

I'll have to try them again now that 1.3 is out, but I found Mercurial
named branches fundamentally unsuited to release management.

 > I'm still a bit on the fence about whether Python should adopt cloned
 > branches and named branches. Since it usually makes more sense to tag
 > releases on the maintenance branch, for example, mainline history
 > would not contain release tags if we used cloned branches.

Ditto named branches.  The problem is that (unless the internal
implementation has changed very recently) a Mercurial revision can be
on exactly one named branch (or on the trunk).

 > A disadvantage might be that the used clones will be a good bit larger
 > (since they essentially contain all other branches as well). This can
 > me mitigated by keeping non-release (feature) branches in separate
 > clones.

Which defeats the purpose of having named branches, really.  (I mean
the version control purpose; obviously it still can save disk space.)

 > Also note that it's still possible to clone a single named
 > branch from a combined clone, by specifying the branch as in hg clone
 > http://hg.python.org/main/#2.6-maint.

Unless you're really short on space, though, that's not a big deal.
What would be more important to me (not that I matter for the purpose
of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
the other way around: pulling an external branch into a named branch.
I have a feeling that working with such a repository with others would
be a little difficult.

 > too many problems. For example, Mercurial itself just uses '1.2.1' as
 > a tag, where CPython would currently use r121.

Stick with the CPython notation.  At XEmacs, continuity of tags has
made our beta testers happy.  (Well, the two of them who bothered to
mention it, anyway. :-)

 > code.python.org was also proposed as the hostname. Personally, I think
 > that using the VCS name in the hostname is good because it prevents
 > confusion: it should be clear that you can't use svn or bzr for
 > hg.python.org.

Agreed, although "can't" is a little too strong.  It might work (there
are a lot of places where http://ftp.example.com works just fine, for
example), but we don't want people to expect it to, and
"http://REPOHOST.python.org/"; should take your browser or your client
to the official repo (which will be the hg repo), not to some index of
repos that happen to live on the same host.

 > Mercurial itself has a stable branch which is a ''strict'' subset of
 > the unstable branch.

As others (MvL, I think) have commented, this isn't really relevant to
Python which generally has four mainlines going at once.  I don't see
why the requirements are going to change with the shift to hg, and I
see no reason why hg won't handle the existing workflow just fine.

Note that PEP 374 was written on the assumption that the existing
workflow will *not* change until the committers have gotten used to
Mercurial, and then it will change in the natural way.  Ie, some one
will say, "you know, this bit of red tape isn't needed any more if we
do X, so let's do X", and after a cascade of 100*"+1 " we do X. :-)

Other than that, looks to be shaping up well.  (Note: I don't have any
comments on subversion-specific aspects, as XEmacs went directly from
CVS to Mercurial).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 3/07/2009 9:28 PM, Dirkjan Ochtman wrote:

On Fri, Jul 3, 2009 at 01:01, Mark Hammond  wrote:
   

Although this has come up in the past, I don't recall a resolution.

What is your plan to handle svn:eol-style?  We have some files in the tree
which need that support and it isn't clear to me how that would work with
the existing win32text extension provided with current mercurial releases.
  (I've an outstanding patch to hg which should address some of these issues,
but without the 'rules' being versioned I fear that would still fall short.)
 


What files would need what? Are there any files that really need to be
\r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember
one file was discussed separately, but I think the outcome there was
that it could just always be \r\n (since it wasn't used at all on
non-Windows platforms). Anyway, knowing specific requirements (or
where to find them) would help here.

   

Even more generally, how will you suggest Windows users work?  Will local
files, in general, have windows line endings or unix?  If the latter, will
there be hooks in-place to prevent editors on Windows 'accidently' mixing
eol styles?  If so, this cycles back to the first question - how would we
know which files get treated that way?
 


There will be a server-side hook to check whitespace. People will also
be able to install it for commit-time.

I think just using \n by default everywhere is a good default (though
I almost always use Windows client machine, I do all nearly all of my
development through a terminal on several Linux boxen), except where
it isn't.
So we must work without effective EOL support?  I fear we will end up 
like the mozilla hg repo with some files in windows line endings and 
some with linux.  While my editing tools are good enough to preserve 
existing EOL styles, I've found myself accidentally checking in new \r\n 
terminated files in a repo which otherwise uses \n line endings.  IMO, 
SVN's EOL support was better than no EOL support.



  People who want to use can set up the win32text stuff to get
\r\n on Windows if they feel they need that -- we can provide
information about that in the dev FAQ (although it would be nice if
someone else who was more familiar with it -- like yourself! -- would
write it).
   
This is exactly why I was asking for your advice - I can't work out how 
to work effectively with win32text as it stands myself, so remain stuck 
accidently checking in files with inappropriate line endings and stuck 
working out how to move pywin32's CVS repo with abandoning the very 
primitive EOL safety it offers...


Cheers,

Mark

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Dirkjan Ochtman]

On Fri, Jul 3, 2009 at 15:31, Mark Hammond wrote:
> So we must work without effective EOL support?  I fear we will end up like
> the mozilla hg repo with some files in windows line endings and some with
> linux.  While my editing tools are good enough to preserve existing EOL
> styles, I've found myself accidentally checking in new \r\n terminated files
> in a repo which otherwise uses \n line endings.  IMO, SVN's EOL support was
> better than no EOL support.

This is why we'll have hooks -- to prevent you from pushing changesets
with inappropriate, to say the least, and, if you're willing to do a
little bit of extra work, to prevent you from committing them.

> This is exactly why I was asking for your advice - I can't work out how to
> work effectively with win32text as it stands myself, so remain stuck
> accidently checking in files with inappropriate line endings and stuck
> working out how to move pywin32's CVS repo with abandoning the very
> primitive EOL safety it offers...

As long as the difference between \r\n- and \n-based files is clear
and can be reasoned about, I don't see why having some of both (I'm
assuming an overwhelming majority will have one, and only a few the
other) is a big problem. But feel free to enlighten me!

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Dirkjan Ochtman]

On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull wrote:
> I'll have to try them again now that 1.3 is out, but I found Mercurial
> named branches fundamentally unsuited to release management.

Can you explain why, please? It's not clear from what you say below.

> Ditto named branches.  The problem is that (unless the internal
> implementation has changed very recently) a Mercurial revision can be
> on exactly one named branch (or on the trunk).

That's still true.

> Which defeats the purpose of having named branches, really.  (I mean
> the version control purpose; obviously it still can save disk space.)

Why does it defeat the purpose? What, in your opinion, is the purpose?

> Unless you're really short on space, though, that's not a big deal.
> What would be more important to me (not that I matter for the purpose
> of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
> the other way around: pulling an external branch into a named branch.
> I have a feeling that working with such a repository with others would
> be a little difficult.

Can you give an example?

> Stick with the CPython notation.  At XEmacs, continuity of tags has
> made our beta testers happy.  (Well, the two of them who bothered to
> mention it, anyway. :-)

Right; Benjamin also mentioned that processing the tags just to be
consistent with the recent tagging scheme would probably be the best
solution.

> As others (MvL, I think) have commented, this isn't really relevant to
> Python which generally has four mainlines going at once.  I don't see
> why the requirements are going to change with the shift to hg, and I
> see no reason why hg won't handle the existing workflow just fine.

It will handle it, for sure, but I think it would all go easier if we
could work with stricter subset branches (and leave the effective
cherrypicking for the occasional problem).

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Paul Moore]

2009/7/3 Łukasz Langa :
>
> On 2009-06-22 at 14:23, Tarek Ziadé wrote:
>
>> Hello,
>>
>> We have polished out PEP 376 and its code prototype at Distutils-SIG.
>> It seems to fullfill now all the requirements,
>> so I am mailing it here again, for a new round of feedback, if needed.
>
>
> Hello.
> I have read the current version of the PEP from trunk and would like to make
> some comments about it.
>
> The .egg-info as a directory
> 
> At our company, after much fiddling about the current distutils and
> setuptools, we have developed a solution that enables us to serve
> pseudo-eggs. What I mean by that is that by plugging into the PEP 302
> infrastructure, we were able to make non-filesystem-based repositories
> behave like `.egg` files. There are a couple of use cases for that
> implementation, the most important ones being:
> * being able to deploy encrypted and sealed blobs with a binary loader, to
> discourage fiddling around the source code by its users
> * being able to serve modules from a database back-end instead of the
> filesystem
>
> This solution depends on the `EGG-INFO` to be fetched from the `.egg`
> itself. We wouldn't want to use a static filesystem-based folder because it
> breaks the encryption use-case by revealing the `SOURCES.txt` information.
> It also breaks the database back-end usecase by presenting static versioning
> information whereas the whole point of using a database backed "egg" is to
> have the newest version installed at all times.

This is a good point. Distutils only installs files in the filesystem
- it has no facilities for installing packages based on any other sort
of PEP 302 based importers. Hence, PEP 376 in principle should only
relate to filesystem-based distributions. But it also mentions
zipfile-based distributions: "Notice that the API is organized in five
classes that work with directories and Zip files (so it works with
files included in Zip files, see PEP 273 for more details [8])."

This is wrong. The PEP should either (1) restrict itself to filesystem
based implementations (leaving the problem of other PEP 302 loaders to
systems that manage these) or (2) be defined in a sufficiently general
way that it can be implemented for *any* PEP 302 based loader - which
probably means extending the PEP 302 protocols - and supplying zipfile
functions as an example of how this is used.

I believe that (1) is unlikely to be sufficient for real world use.
Zip files (eggs, py2exe embedded modules, etc) are far too important a
real world use case to ignore. The problem with (2) is that it
requires significant extra work. But special-casing zip files (as the
present implementation appears to do) will break as soon as any other
PEP 302 compliant format becomes popular.

> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the
> ``pkg_resources`` approach where we don't enforce resources to reside on a
> traditional filesystem.

On the other hand, pkgutil.get_data is the standard library means of
reading resources from a package. This is PEP 302 compliant now. This
new PEP doesn't affect that.

> The uninstall feature
> =
> This is a great feature but I don't seem to understand why there seems to be
> a consensus here that it's good for it not to even warn about dependencies,
> let alone manage them. ``easy_install`` and co. does manage dependencies
> while installing, why shouldn't it be symmetrical in that regard? Moreover,
> wouldn't dependency management be useful while using ``easy_install`` to
> *upgrade* an installation of a package?

easy_install is not a standard library feature - the fact that it has
no uninstall facility is not relevant.

python setup.py install is the standard library feature. The new
uninstall is intended to correspond to that.

> More high-level remarks
> ===
> This isn't probably the best place to cover this but may serve as a
> rationale for the above .egg-info problems we have with the proposed
> changes.
>
> It's great that you push the metadata functionality to core distutils. At
> the same time, it would be feasible to think over the whole `egg` concept.
> We would argue that having the `egg` format as a container more-less
> agnostic to the underlying data storage structure would be **very** useful.
> Imagine installable `egg` support packages which would enable you to:
> * use other compression formats beside ZIP
> * use other means of module storage beside the filesystem
> * use sealed and encrypted archives for deployment
> * use programmatic `egg`s for licensing, etc.

PEP 302 gives you (in theory) all of this now. I'd recommend you look
at it - and at Brett's importlib, which will make experimenting with
such things far easier.

What PEP 302 doesn't provide is package management. But Python itself
doesn't provide package management, except in the form of distutils
setup.py install - which is solely filesystem based.

Maybe there's a case for

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Mark Hammond wrote:
> So we must work without effective EOL support?  I fear we will end up
> like the mozilla hg repo with some files in windows line endings and
> some with linux.  While my editing tools are good enough to preserve
> existing EOL styles, I've found myself accidentally checking in new \r\n
> terminated files in a repo which otherwise uses \n line endings.  IMO,
> SVN's EOL support was better than no EOL support.

If Mercury doesn't have automatic whitespace conversion along the lines
of svn:eol-style, then the existing white-space checking script probably
needs to be updated to enforce the appropriate line endings for all files.

If we default to Unix line endings for most files, then the checking
script can be made aware of which files should always have Windows line
endings (I believe the various Visual Studio files need them, batch
files are probably best left with Windows line endings in the
repository, and I expect there are other files in PC and PCbuild that
need them as well).

However, I expect that would still be painful to work with for Windows
developers, even if it prevented any line ending problems from making
their way into the main repository. I believe that is where the
win32text extensions can help. Looking at the Wiki page for win32text
[1], I believe it would be a matter of configuring the extension to
encode and decode all files with the extensions:

*.py
*.pyw
*.h
*.c
*.in
*.rst
*.asdl

That said, I don't see a way to tell win32text to also translate files
which don't have an extension at all (e.g. NEWS or ACKS), and there
doesn't seem to be a way to tell it to *skip* files matching certain
patterns (if there was, we could just tell it to ignore extensions like
.bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert
everything else)

Cheers,
Nick.

[1] http://mercurial.selenic.com/wiki/Win32TextExtension

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Nick Coghlan]

Paul Moore wrote:
> To be honest, this is a major can of worms. But if PEP 376 is not
> going to support PEP 302, then it must state that fact explicitly, to
> avoid giving people false expectations - particularly with Brett's
> importlib in Python 3.1, which will make it far easier for people to
> experiment with new packaging formats such as the ones Lukasz mentions
> above. And it MUST fail gracefully in the face of unsupported importer
> types.

importlib is in 2.7 as well.

I agree that even if the reference implementation of PEP 376 only
handles normal directories and zipfiles, the PEP itself needs to explain
how the developer of a custom PEP 302 importer or loader can hook into
the mechanism in order to provide metadata that distutils can
understand. Or, as you suggest, state explicitly that the proposal at
this stage is specifically limited to filesystem and zipfile packages
and defer extension to arbitrary PEP 302 importers and loaders to a
later PEP.

To be honest, I'd personally be OK with the latter strategy - while
other PEP 302 loaders and importers do exist (as Lukasz'z post shows),
it would be unfortunate to unduly hold up improved installation metadata
for the vast majority of typical use cases while we try to figure out
ways to support the more esoteric use cases. Supporting both would
obviously be better, but given the choice between the status quo and
partial support, I believe this is a case where the partial support
would still be worthwhile.

I will note (and I believe this is also the main point that Lukasz was
making) that having the distribution metadata outside the distribution
as currently proposed in PEP 376 is going to make any eventual PEP 302
integration much harder - 302 importers only provide a mechanism for
accessing files inside the distribution, not "adjacent" to them, so the
mechanism in the PEP doesn't generalise properly.

I suspect this limitation of the PEP 302 APIs is the origin of the
setuptools format that embeds the metadata inside the distribution - it
lets you get at the metadata without having to assume that it exists
directly on the filesystem anywhere.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Tarek Ziadé]

2009/7/3 Paul Moore :
> This is a good point. Distutils only installs files in the filesystem
> - it has no facilities for installing packages based on any other sort
> of PEP 302 based importers. Hence, PEP 376 in principle should only
> relate to filesystem-based distributions. But it also mentions
> zipfile-based distributions: "Notice that the API is organized in five
> classes that work with directories and Zip files (so it works with
> files included in Zip files, see PEP 273 for more details [8])."
>
> This is wrong. The PEP should either (1) restrict itself to filesystem
> based implementations (leaving the problem of other PEP 302 loaders to
> systems that manage these) or (2) be defined in a sufficiently general
> way that it can be implemented for *any* PEP 302 based loader - which
> probably means extending the PEP 302 protocols - and supplying zipfile
> functions as an example of how this is used.
>
> I believe that (1) is unlikely to be sufficient for real world use.
> Zip files (eggs, py2exe embedded modules, etc) are far too important a
> real world use case to ignore. The problem with (2) is that it
> requires significant extra work. But special-casing zip files (as the
> present implementation appears to do) will break as soon as any other
> PEP 302 compliant format becomes popular.
>> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the
>> ``pkg_resources`` approach where we don't enforce resources to reside on a
>> traditional filesystem.
>
> On the other hand, pkgutil.get_data is the standard library means of
> reading resources from a package. This is PEP 302 compliant now. This
> new PEP doesn't affect that.
>

Right. While it would be feasible to make pgutil works with PEP 302 loaders,
we would still need to define in a generic way the content of the RECORD files.

Right now it works for directory and zipped files since it's expressed
with '/' separated paths.

And if I understand PEP 302 right, any backend would be able to handle
these paths
no matter how they are stored, as long as the implement APIs like get_data()

> What PEP 302 doesn't provide is package management. But Python itself
> doesn't provide package management, except in the form of distutils
> setup.py install - which is solely filesystem based.
>
> Maybe there's a case for extending PEP 302 and distutils to allow
> integrated management of other forms of importer format, but that's a
> huge other project, which no-one to my knowledge is even looking at.

Sounds like a fully-featured packaging managment system, which is
imho, out of scope.

And I don't see PEP 376 making it impossible for someone to build such a
packaging system on the top of distutils.

I've started one myself for the sake of experimentation, with built-in
multiversion
support, for a full replacement of site-packages.

> Eggs are fundamentally a PEP 302 zip file format. There are some extra
> bits of metadata for setuptools/easy_install in there (as I understand
> things) but essentially they are zip files. When you say "decoupling
> the egg format", I assume you mean "decoupling the egg metadata" -
> which is fine, but to properly decouple, you need API level access to
> the metadata. PEP 376 offers read-only access, but as you rightly
> point out, it is only for filesystem data (and some form of zip file,
> which appears to be limited in some way, as it isn't PEP 302 based,
> and the actual format isn't defined anywhere).

And also PEP 376 goal is to define a single standard location of
egg-info files for filesystem data.

The zip form was built so it could work with zipped site-packages directories,
like what the py2app project uses.

>
> The basic point here is that PEP 376 needs to define precisely how
> pkgutil.get_distributions() scans sys.path looking for ".egg-info
> directories". What does it do for sys.path entries that don't
> correspond to filesystem directories? (Note - these may or may not be
> zip files. Even if they are zip files, an earlier entry on
> sys.path_hooks could have taken precedence. At the very least, you
> should only process path entries as zip files if their importer - in
> sys.path_importer_cache or via an explicit path hook scan - is a
> zipimporter object.).

I'll add more details on that part. right now it visits directories
and zip files.

Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Tarek Ziadé]

On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan wrote:
> I will note (and I believe this is also the main point that Lukasz was
> making) that having the distribution metadata outside the distribution
> as currently proposed in PEP 376 is going to make any eventual PEP 302
> integration much harder - 302 importers only provide a mechanism for
> accessing files inside the distribution, not "adjacent" to them, so the
> mechanism in the PEP doesn't generalise properly.
>
> I suspect this limitation of the PEP 302 APIs is the origin of the
> setuptools format that embeds the metadata inside the distribution - it
> lets you get at the metadata without having to assume that it exists
> directly on the filesystem anywhere.

Maybe we could rework the pkgutil classes for PEP 376 so they look
like an implementation
of the PEP 302 protocol for directories and zip files, with an extra
"get_metadata()" API
and state that it could be an extension for PEP 302 later.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Georg Brandl]

Dirkjan Ochtman schrieb:

> - Fifth, here's a list of things, off the top of my head, that still need 
> doing:
> 
> * Get agreement on branch strategy and branch processing (list of
> branches + proposed handling at
> http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
> REVIEW

Do you have a key to the second column in that file? E.g. the difference
between "strip" and "discard" is not clear to me. "strip partial"?

Why are there branch names starting with "../"?

[PEP 385]

> ==
> 
> Motivation
> 
> After having decided to switch to the Mercurial DVCS, the actual
> migration still has to be performed. In the case of an important piece
> of infrastructure like the version control system for a large,
> distributed project like Python, this is a significant effort. This
> PEP is an attempt to describe the steps that must be taken for further
> discussion. It's somewhat similar to PEP 347 [1], which discussed the
> migration to SVN.
> 
> To make the most of hg, I (Dirkjan) would like to make a high-fidelity
> conversion, such that (a) as much of the svn metadata as possible is
> retained, and (b) all metadata is converted to formats that are common
> in Mercurial. This way, tools written for Mercurial can be optimally
> used. In order to do this, I want to use the hgsubversion [2] software
> to do an initial conversion. This hg extension is focused on providing
> high-quality conversion from Subversion to Mercurial for use in
> two-way correspondence, meaning it doesn't throw away as much
> available metadata as other solutions.
> 
> Such a conversion also seems like a good time to reconsider the
> contents of the repository and determine if some things are still
> valuable. In this spirit, the following sections also propose
> discarding some of the older metadata.
> Timeline
> 
> TBD; needs fully working hgsubversion and consensus on this document.
> Transition plan
> Branch strategy
> 
> Mercurial has two basic ways of using branches: cloned branches, where
> each branch is kept in a separate repository, and named branches,
> where each revision keeps metadata to note on which branch it belongs.
> The former makes it easier to distinguish branches, at the expense of
> requiring more disk space on the client. The latter makes it a little
> easier to switch between branches, but often has somewhat unintuitive
> results for people (though this has been getting better in recent
> versions of Mercurial).
> 
> I'm still a bit on the fence about whether Python should adopt cloned
> branches and named branches. Since it usually makes more sense to tag
> releases on the maintenance branch, for example, mainline history
> would not contain release tags if we used cloned branches. Also,
> Mercurial 1.2 and 1.3 have the necessary tools to make named branches
> less painful (because they can be properly closed and closed heads are
> no longer considered in relevant cases).
> 
> A disadvantage might be that the used clones will be a good bit larger
> (since they essentially contain all other branches as well). This can
> me mitigated by keeping non-release (feature) branches in separate
> clones. Also note that it's still possible to clone a single named
> branch from a combined clone, by specifying the branch as in hg clone
> http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a
> separate clone problably also makes sense.

* Does it work with "hg pull" etc. too, afterwards?
* Does it support more than one branch?

> XXX To do: size comparison for selected separation scenarios.
> Converting branches
> 
> There are quite a lot of branches in SVN's branches directory. I
> propose to clean this up a bit, by employing the following the
> strategy:
> 
> * Keep all release (maintenance) branches
> * Discard branches that haven't been touched in 18 months, unless
> somone indicates there's still interest in such a branch
> * Keep branches that have been touched in the last 18 months,
> unless someone indicates the branch can be deprecated

I would just kill all feature branches unless someone indicates it is
still used.  There are very few active feature branches.

(I guess in the case a branch gets killed erroneously it could still be
re-created after the conversion?)

> Converting tags
> 
> The SVN tags directory contains a lot of old stuff. Some of these are
> not, in fact, full tags, but contain only a smaller subset of the
> repository. I think we should keep all release tags, and consider
> other tags for inclusion based on requests from the developer
> community. I'd like to consider unifying the release tag naming scheme
> to make some things more consistent, if people feel that won't create
> too many problems. For example, Mercurial itself just uses '1.2.1' as
> a tag, where CPython would currently use r121.

+1 for readable tag names.
+1 for throwing out old questionable tag names.

> Generating .hgignore
> 
> The

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (06/26/09 - 07/03/09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2231 open (+26) / 16000 closed (+38) / 18231 total (+64)

Open issues with patches:   890

Average duration of open issues: 661 days.
Median duration of open issues: 413 days.

Open Issues Breakdown
   open  2203 (+26)
pending28 ( +0)

Issues Created Or Reopened (65)
___

Remove checks for win NT 06/29/09
CLOSED http://bugs.python.org/issue4856reopened amaury.forgeotdarc  
  
   patch   

Rstrip Incorrectly Strips Some Strings   06/26/09
CLOSED http://bugs.python.org/issue6346created  mkubilus
  
   

hpux11.00-parisc: dtoa.c: "Failed to find an exact-width 32-bit  06/26/09
CLOSED http://bugs.python.org/issue6347created  srid
  
   patch   

solaris/aix: Py_Initialize: can't initialize sys standard stream 06/26/09
   http://bugs.python.org/issue6348created  srid
  
   

email.Maildir cannot roundtrip messages. 06/27/09
CLOSED http://bugs.python.org/issue6349created  bremner 
  
   

Example at end of HTMLParser documentation uses old-style print  06/27/09
CLOSED http://bugs.python.org/issue6350created  MLModel 
  
   

Compiler warning in _cursesmodule.c  06/28/09
   http://bugs.python.org/issue6351created  hagen   
  
   

Compiler warning in unicodeobject.c  06/28/09
   http://bugs.python.org/issue6352created  hagen   
  
   patch   

"L" integer suffix in Python 3.1 tutorial06/28/09
CLOSED http://bugs.python.org/issue6353created  delroth 
  
   patch   

Old floating point representation in 3.1 tutorial06/28/09
CLOSED http://bugs.python.org/issue6354created  delroth 
  
   

bogus NULL check in PyCapsule06/28/09
CLOSED http://bugs.python.org/issue6355created  bkramer 
  
   patch   

Segfault of sys.setrecursionlimit   if limit exceeds the value 180 06/28/09
CLOSED http://bugs.python.org/issue6356created  str1442 
  
   

tempfile.NamedTemporaryFile does not accept the delete= paramete 06/28/09
CLOSED http://bugs.python.org/issue6357created  stutzbach   
  
   

os.popen exit code inconsistent  06/28/09
   http://bugs.python.org/issue6358created  abbeyj  
  
   

pyexpat.c calls trace function incorrectly for exceptions06/28/09
   http://bugs.python.org/issue6359created  nedbat  
  
   

Simplify string decoding in xmlrpc.client.   06/29/09
   http://bugs.python.org/issue6360created  alexandre.vassalotti
  
   patch   

I/O object wrappers shouldn't close their underlying file when d 06/29/09
   http://bugs.python.org/issue6361created  alexandre.vassalotti
  
   

multiprocessing: handling of errno after signals in sem_acquire( 06/29/09
   http://bugs.python.org/issue6362created  ryles   
  
   

__future__ statements break doctest  06/29/09
CLOSED http://bugs.python.org/issue6363created  pooryorick  
  
   

Re: [Python-Dev] PEP 376

[Paul Moore]

2009/7/3 Tarek Ziadé :
> On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan wrote:
>> I will note (and I believe this is also the main point that Lukasz was
>> making) that having the distribution metadata outside the distribution
>> as currently proposed in PEP 376 is going to make any eventual PEP 302
>> integration much harder - 302 importers only provide a mechanism for
>> accessing files inside the distribution, not "adjacent" to them, so the
>> mechanism in the PEP doesn't generalise properly.
>>
>> I suspect this limitation of the PEP 302 APIs is the origin of the
>> setuptools format that embeds the metadata inside the distribution - it
>> lets you get at the metadata without having to assume that it exists
>> directly on the filesystem anywhere.
>
> Maybe we could rework the pkgutil classes for PEP 376 so they look
> like an implementation
> of the PEP 302 protocol for directories and zip files, with an extra
> "get_metadata()" API
> and state that it could be an extension for PEP 302 later.

That sounds like an excellent approach And given that PEP 302 is
nothing more than an API spec, it's nice and easy to extend PEP 302 to
say that importers can/should implement this (after all, the only 2
"real" cases in the wild are done, via PEP 376).

Warning: I've not thought all this through fully, so I may be missing
some subtleties! The ultimate spec needs to be clearly worded, and as
a consequence of this discussion it occurs to me that the current PEP
376 is very far from clear in defining exactly what the naming, layout
and location of egg-info directories is - it makes a lot of
assumptions that may break for more general path importers

I'll do some thinking, and maybe come up with some examples of PEP 302
edge cases, so that we can at least be sure that they've not been
ignored (rejected as to stupid to care about, on the other hand, I'm
happy with :-))

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[P.J. Eby]

At 02:54 PM 7/3/2009 +0100, Paul Moore wrote:
Eggs are fundamentally a PEP 302 zip file format. There are some 
extra bits of metadata for setuptools/easy_install in there (as I 
understand things) but essentially they are zip files. When you say 
"decoupling the egg format", I assume you mean "decoupling the egg 
metadata" - which is fine, but to properly decouple, you need API 
level access to the metadata. PEP 376 offers read-only access, but 
as you rightly point out, it is only for filesystem data (and some 
form of zip file, which appears to be limited in some way, as it 
isn't PEP 302 based, and the actual format isn't defined anywhere). 
The basic point here is that PEP 376 needs to define precisely how 
pkgutil.get_distributions() scans sys.path looking for ".egg-info 
directories". What does it do for sys.path entries that don't 
correspond to filesystem directories? (Note - these may or may not 
be zip files. Even if they are zip files, an earlier entry on 
sys.path_hooks could have taken precedence. At the very least, you 
should only process path entries as zip files if their importer - in 
sys.path_importer_cache or via an explicit path hook scan - is a 
zipimporter object.). To be honest, this is a major can of worms. 
But if PEP 376 is not going to support PEP 302, then it must state 
that fact explicitly, to avoid giving people false expectations - 
particularly with Brett's importlib in Python 3.1, which will make 
it far easier for people to experiment with new packaging formats 
such as the ones Lukasz mentions above. And it MUST fail gracefully 
in the face of unsupported importer types.


Well, we could always resurrect PEP 365, since pkg_resources already 
has documented extensible support for arbitrary importers.  That 
solves backward *and* forward compatibility.  Then PEP 376's 
uninstall facilities could be implemented using pkg_resources' 
existing metadata query features.


The primary downside to that, of course, is that it brings in the 
matter of version specifications and dependencies...  which appear to 
be a contentious topic.  (Note that Tarek is proposing to drop the 
PEP 386 proposal to standardize a much more restrictive scheme than 
seutptools' version parser, precisely because of the controversy.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Paul Moore]

2009/7/3 Nick Coghlan :
> I agree that even if the reference implementation of PEP 376 only
> handles normal directories and zipfiles, the PEP itself needs to explain
> how the developer of a custom PEP 302 importer or loader can hook into
> the mechanism in order to provide metadata that distutils can
> understand. Or, as you suggest, state explicitly that the proposal at
> this stage is specifically limited to filesystem and zipfile packages
> and defer extension to arbitrary PEP 302 importers and loaders to a
> later PEP.
>
> To be honest, I'd personally be OK with the latter strategy - while
> other PEP 302 loaders and importers do exist (as Lukasz'z post shows),
> it would be unfortunate to unduly hold up improved installation metadata
> for the vast majority of typical use cases while we try to figure out
> ways to support the more esoteric use cases. Supporting both would
> obviously be better, but given the choice between the status quo and
> partial support, I believe this is a case where the partial support
> would still be worthwhile.

You put it more clearly than I did. That's basically what I think,
with the one proviso that we should make sure that PEP 376 doesn't
specify something that out and out breaks the more esoteric PEP 302
cases.

When Just and I were developing PEP 302, we found that the best way to
do that was to leave anything that didn't need to be specified, as
unspecified (hence the fact that there's so little defined in PEP
302!). It's easier to add things later than to remove or change them.
That's why I was recommending to Tarek that he take out of the PEP any
details about classes or APIs that couldn't be directly obtained from
the core API. The same applies here (just talking in terms of a
duck-typed notional Distribution class, allows people to implement
their own URLDistribution or SqliteDistribution, or whatever, and not
have to emulate any more of a filesystem than necessary).

> I will note (and I believe this is also the main point that Lukasz was
> making) that having the distribution metadata outside the distribution
> as currently proposed in PEP 376 is going to make any eventual PEP 302
> integration much harder - 302 importers only provide a mechanism for
> accessing files inside the distribution, not "adjacent" to them, so the
> mechanism in the PEP doesn't generalise properly.
>
> I suspect this limitation of the PEP 302 APIs is the origin of the
> setuptools format that embeds the metadata inside the distribution - it
> lets you get at the metadata without having to assume that it exists
> directly on the filesystem anywhere.

Again, agreed. But remember that PEP 302 is driven by looking up a
module or package name. PEP 376 is looking up a *distribution* name.
The docutils example in the PEP shows this:

- docutils/
- roman.py
- docutils-0.5-py2.6.egg-info/

There are 3 things here: a package (docutils), a module (roman) and a
distribution (also named docutils, but could be called George for all
it matters). So none of the PEP 302 lookup mechanisms (which work on
package/module name) apply.

The need for a separate concept (a "distribution") is unfortunate, as
it adds complexity, but there are enough real life cases that make it
clear that it's necessary.

Hmm, I suspect that the implication here is that PEP 302 could do with
an overhaul, to extend it to encompass the concept of a distribution.
I'd be willing to have a look at that.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Paul Moore]

2009/7/3 P.J. Eby :
> Well, we could always resurrect PEP 365, since pkg_resources already has
> documented extensible support for arbitrary importers.  That solves backward
> *and* forward compatibility.  Then PEP 376's uninstall facilities could be
> implemented using pkg_resources' existing metadata query features.

It would be easier to comment on that suggestion if the PEP documented
the reasons why it was rejected. Sadly, that never got added to the
PEP at the time. Do you recall the reasons, and if so would you be
willing to update the PEP with them?

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[P.J. Eby]

At 12:28 AM 7/4/2009 +1000, Nick Coghlan wrote:

I suspect this limitation of the PEP 302 APIs is the origin of the
setuptools format that embeds the metadata inside the distribution - it
lets you get at the metadata without having to assume that it exists
directly on the filesystem anywhere.


I think you have this backwards; it's setuptools that doesn't care 
where (or whether) the metadata exists on the file system; it 
delegates metadata operations to a "metadata provider" that's usually 
an adapter over a PEP 302 "loader".


See 
http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-importers 
for the API details of how to register support for arbitrary PEP 302 
importers and loaders.  (Which presumably, Lukasz is using.  I didn't 
know that anybody was actually using it, but it's nice to know that 
the documentation is apparently sufficient for *some* people.  ;-) )


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Brett Cannon]

On Thu, Jul 2, 2009 at 13:42, Dirkjan Ochtman  wrote:

> In response to some rumblings on python-committers and just to request
> more feedback, a progress report. I know it's long, I've tried to put
> to keep it concise and chunked, though.
>
> - First of all, I've got the basic conversion down, I've done it a few
> times now, with progressively better results. You can view some
> results at http://hg.python.org/, which has a preliminary cpython
> repository. *** The changeset hashes for that repo will change, so you
> won't be able to commit or pull from it in the future.***
>
> - Second of all, some planning. I've thought about it a bit, and I
> think we should aim for going live with hg on August 1. Given that I'm
> on vacation from 8-18 July (and I'm not sure whether I'll be able to
> actually work on it during that time, though I imagine I'll be able to
> spend some time on it at least), that's quite ambitious, so I'm going
> to say it's okay if it slips by a few days. Putting a deadline out
> there is a good thing, anyway.
>

Fine by me as long as people realize that if anything is questionable then
the switch will not happen. Getting this right takes precedence over any
deadline. And obviously we will need to do at least one live conversion on
python.org hardware to make sure everything will work smoothly.


>
> - Third of all, to make this possible, it would be helpful if I got
> more feedback on the PEP. Last time I raised it, there was virtually
> nothing. This time, I'll include it inline so there's hopefully less
> of a barrier to reviewing it.
>
> - Fourth, Mercurial 1.3 was just released! Bet you didn't see that
> coming. It's looking like a pretty good release, with an experimental
> version of the much-coveted subrepository support (like
> svn:externals). This also means that the latest version of
> hgsubversion, the tool I used for the conversion, will be more
> accessible for converting other projects. You know you want to!
>

And will make the idea of splitting out the standard library and tests a
reasonable thing to do.


>
> - Fifth, here's a list of things, off the top of my head, that still need
> doing:
>
> * Get agreement on branch strategy and branch processing (list of
> branches + proposed handling at
> http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE
> REVIEW
> * Get agreement on tag processing (first come up with a list)
> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs
>
> - Sixth (this is the good part), less obvious things that have been
> done or don't need doing:
>
> * .hgignore generation (I've been convinced it's too hard, the current
> version will do)


Yeah, we can do this manually.


>
> * revlog reordering (it's painless and a big win)
>
> I'll get through all of these myself, but obviously any help would be
> welcome. For any hg users, writing docs should be an easy start. For
> others, please review the PEP (below), the branch map in
> http://hg.python.org/pymigr/file/tip/all-branches.txt and the author
> map at http://hg.python.org/pymigr/file/tip/author-map (not that much
> has changed since the start, so if you've looked at it already, feel
> free to skip this part). Right now I'm a little stuck on branch
> processing, because it's a long running script that needs a bunch of
> debugging, but I'll get going on that again.
>
> I think that's all I can think of for now, I'll update the PEP with
> new bits soon. Here it is, ready for your review:
>
> ==
>
> Motivation
>
> After having decided to switch to the Mercurial DVCS, the actual
> migration still has to be performed. In the case of an important piece
> of infrastructure like the version control system for a large,
> distributed project like Python, this is a significant effort. This
> PEP is an attempt to describe the steps that must be taken for further
> discussion. It's somewhat similar to PEP 347 [1], which discussed the
> migration to SVN.
>
> To make the most of hg, I (Dirkjan) would like to make a high-fidelity
> conversion, such that (a) as much of the svn metadata as possible is
> retained, and (b) all metadata is converted to formats that are common
> in Mercurial. This way, tools written for Mercurial can be optimally
> used. In order to do this, I want to use the hgsubversion [2] software
> to do an initial conversion. This hg extension is focused on providing
> high-quality conversion from Subversion to Mercurial for use in
> two-way correspondence, meaning it doesn't throw away as much
> available metadata as other solutions.
>
> Such a conversion also seems like a good time to reconsider the
> contents of the repository and determine if some things are still
> valuable. In this spirit, the following sections also propose
> discarding some of the older metadata.
> T

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Terry Reedy]

Dirkjan Ochtman wrote:


It needs to be decided where the hg repositories will live. I'd like
to propose to keep the hgwebdir instance at hg.python.org. This is an
accepted standard for many organizations, and an easy parallel to
svn.python.org. The 2.7 (trunk) repo might live at
http://hg.python.org/main/, for example, with py3k at
http://hg.python.org/py3k/.


I would very much like the 'k' dropped from the py3 name. It was a funny 
joke when py3 was vaporware, now it is excess baggage which only puzzles 
non-insiders and newcomers.


I think the two repos should be either symmetrically named

hg.python.org/py2
hg.python.org/py3

If one must be designated 'main', it should be py3.

Continuing to call py2 'main' will continue to discourage use of py3.

Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[MRAB]

Terry Reedy wrote:

Dirkjan Ochtman wrote:


It needs to be decided where the hg repositories will live. I'd like
to propose to keep the hgwebdir instance at hg.python.org. This is an
accepted standard for many organizations, and an easy parallel to
svn.python.org. The 2.7 (trunk) repo might live at
http://hg.python.org/main/, for example, with py3k at
http://hg.python.org/py3k/.


I would very much like the 'k' dropped from the py3 name. It was a funny 
joke when py3 was vaporware, now it is excess baggage which only puzzles 
non-insiders and newcomers.


I think the two repos should be either symmetrically named

hg.python.org/py2
hg.python.org/py3

If one must be designated 'main', it should be py3.

Continuing to call py2 'main' will continue to discourage use of py3.


We could regard py3k as the phase from the original concept of Python 3
to its 'prototype', Python 3.0. Python 3.1 would be the first
'real/usable' version.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Aahz]

On Fri, Jul 03, 2009, Brett Cannon wrote:
>
> Should we consider adding a sys.revision attribute and begin the deprecation
> of sys.subversion?

+1
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[Paul Moore]

This post basically follows on from the previous thread about PEP 376
(Changing the .egg-info structure) where it was pointed out that the
PEP doesn't cover PEP 302 import hooks (except in the explicit special
case of zip files). This is likely to be a fairly long posting, but
I'd like to try and cover the subtleties a little, so that people can
comment without having to refer back to too many documents.

Comments are most definitely welcome! While I know PEP 302 reasonably
well, and the zip importer implementation, I'm not an expert on the
egg-info structure, or the practical drivers behind it, so if I miss
key issues because of a too-theoretical approach, I'd be grateful for
corrections.

The basic structure of PEP 302 imports is as follows:

  - scan sys.meta_path checking each finder in turn
  - if nothing found, scan sys.path and for each entry pass it to each
element of sys.path_hooks, to get a finder to check
  - do the default filesystem processing on path items for which no
path hook applies

The first finder (if any) to recognise the module name wins (and
returns a loader responsible for creating the actual module object).

So it's the finders that are responsible for scanning the "filesystem"
(more accurately, the namespace that finder manages).

PEP 302 is entirely couched in terms of modules, packages and module
names. There is no concept of a "distribution" in PEP 376 terms. This
is entirely understandable, as imports don't know about distributions
(see the docutils example in PEP 376 - the docutils *distribution*
contains the *module* roman, but there's nothing about importing roman
that requires you to know that it came from the docutils
distribution).

So we need to introduce a new concept, that of a distribution, into
PEP 302 loaders. And the place it should be located is in the finder
(which handles the "filesystem" aspects of the protocol. So we have
the methods:

Finder.list_distributions()
Returns a list of all distribution names in the "filesystem" managed
by the finder (usually one zip file, path directory, salite database,
or whatever)

Finder.get_metadata(distribution_name)
Returns a metadata object for the given distribution (this is the PEP
376 Distribution object). One note here - get_egginfo_file should be
specified as returning a *file-like object* rather than a file
instance. Another note - get_egginfo_file and get_egginfo_files could
just as easily be named get_metadata_file and get_metadata_files -
just as meaningful, and less tied to the egg format (as well as
avoiding the "egg" name, which I personally dislike :-))

Finder.uninstall(distribution_name, filter=callable, installer=name)
Uninstall the given distribution. It's likely that many finders will
be read-only. In that case, this function should return None.
Otherwise, return a list of the "files" removed. (This may need some
clarification, as many finders won't have a concept of a "file name").

I don't think anything else is needed to support PEP 376.

The prototype implementation of PEP 376 could be extended to work with
finders (doing the relevant sys.meta_path and sys.path_hooks
searches). For the final implementation, the special casing of zip
files would be replaced by an implementation of the extended finder
protocol in the zipimporter object.

There's almost certainly aspects missing from the above proposal. But
it does have some definite advantages, above and beyond simply
allowing PEP 302 importers to participate in the PEP 376 protocol.
Setuptools-style eggs could be handled simply by creating a
specialised finder (IIUC, they currently just use the standard
zipimporter - the specialised version could subclass this) to override
the metadata methods so as to cater for their specialised egg-info
format. Other formats could be handled similarly.

Does this sound sensible? Tarek, would you be OK with me attempting to
modify your prototype to support this protocol? Are there any tests
for PEP 376, so that I can confirm I haven't completely broken
something? If I can, I'll knock up some simple prototype importers for
non-standard examples, and see how they work with all this.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[Tarek Ziadé]

2009/7/3 Paul Moore :
> Does this sound sensible? Tarek, would you be OK with me attempting to
> modify your prototype to support this protocol? Are there any tests
> for PEP 376, so that I can confirm I haven't completely broken
> something? If I can, I'll knock up some simple prototype importers for
> non-standard examples, and see how they work with all this.

Yes that's exactly what I was thinking about after the discussion we
had in the other
thread. This change would allow much more flexibility.

Pleas go ahead, it's located here :

http://bitbucket.org/tarek/pep376/

You can give me a bitbucket account so I can give you write access to the repo,
There are tests as long as you install Nose.

Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Tarek Ziadé]

On Fri, Jul 3, 2009 at 7:02 PM, P.J. Eby wrote:
> At 02:54 PM 7/3/2009 +0100, Paul Moore wrote:
>>
>> Eggs are fundamentally a PEP 302 zip file format. There are some extra
>> bits of metadata for setuptools/easy_install in there (as I understand
>> things) but essentially they are zip files. When you say "decoupling the egg
>> format", I assume you mean "decoupling the egg metadata" - which is fine,
>> but to properly decouple, you need API level access to the metadata. PEP 376
>> offers read-only access, but as you rightly point out, it is only for
>> filesystem data (and some form of zip file, which appears to be limited in
>> some way, as it isn't PEP 302 based, and the actual format isn't defined
>> anywhere). The basic point here is that PEP 376 needs to define precisely
>> how pkgutil.get_distributions() scans sys.path looking for ".egg-info
>> directories". What does it do for sys.path entries that don't correspond to
>> filesystem directories? (Note - these may or may not be zip files. Even if
>> they are zip files, an earlier entry on sys.path_hooks could have taken
>> precedence. At the very least, you should only process path entries as zip
>> files if their importer - in sys.path_importer_cache or via an explicit path
>> hook scan - is a zipimporter object.). To be honest, this is a major can of
>> worms. But if PEP 376 is not going to support PEP 302, then it must state
>> that fact explicitly, to avoid giving people false expectations -
>> particularly with Brett's importlib in Python 3.1, which will make it far
>> easier for people to experiment with new packaging formats such as the ones
>> Lukasz mentions above. And it MUST fail gracefully in the face of
>> unsupported importer types.
>
> Well, we could always resurrect PEP 365, since pkg_resources already has
> documented extensible support for arbitrary importers.  That solves backward
> *and* forward compatibility.  Then PEP 376's uninstall facilities could be
> implemented using pkg_resources' existing metadata query features.
>
> The primary downside to that, of course, is that it brings in the matter of
> version specifications and dependencies...  which appear to be a contentious
> topic.  (Note that Tarek is proposing to drop the PEP 386 proposal to
> standardize a much more restrictive scheme than seutptools' version parser,
> precisely because of the controversy.)

I don't think we can add pkg_resources as-is because it does much more than
providing the query features, which is beyond distutils scope.

I think that part of the PEP 376 code at the end will be very similar
to the subpart of pkg_resources
that has the query features (as it is now with the current prototype)

And what is nice with Paul's proposal on changing PEP 302 is that
setuptools will
be able to propose its EggFormats on the top of the pkgutils/distutils
changes and make its
pkg_resource code lighter.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> So we must work without effective EOL support?

If that's the case (i.e. no effective EOL support, the way svn
supported it), then I think the PEP should make that clear (e.g.
in a discussion section).

For the server-side hooks: it would be good to know exactly
what they check, wrt. line endings.

To find out what files should not be stored with LF line endings,
do "svn pg -R svn:eol-style .|grep CRLF".

For win32text, it would probably be good if the FAQ would provide
the relevant configuration instructions; it would be really helpful
if somebody familiar with Windows and hg could provide detailed
instructions well in advance of August 1.

If we don't have anybody familiar with Windows and hg, we have a
really serious problem.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> It will handle it, for sure, but I think it would all go easier if we
> could work with stricter subset branches (and leave the effective
> cherrypicking for the occasional problem).

So I think the PEP should propose a workflow (or: merge flow) if you
think we would be better off with a different one.

In proposing such a workflow, consider these requirements:
- we current have four active "maintenance" branches (i.e. where
  the entire code basis evolves): trunk, 3k, 2.6, and 3.1 (3.0
  also until this morning).
- in addition, we have two security branches currently: 2.4 and
  2.5, although 2.4 will be closed soon.
- our committers consistently refuse to merge changes across
  branches themselves, and likely continue to do so unless there
  is some feature of hg that I missed (e.g. one were merging
  would happen without any user specifically asking for it)

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> Should we consider adding a sys.revision attribute and begin the
> deprecation of sys.subversion?

I wouldn't mind killing sys.subversion "right away" (i.e. in trunk
and 3k - obviously it has to stay in 2.6 and 3.1, and all the older
branches).

I'm -1 on calling it "sys.revision", as this makes it difficult to
tell what the actual versioning system was, and hence how the
data should be interpreted. It will already be a problem for 2.6,
when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
which will surely crash existing applications.

I'm not sure what the motivation for a sys.revision is; it's
probably similar to the desire of calling the machine code.python.org
(instead of hg.python.org). It gives the illusion of being agnostic
of the actual RCS being used. However, this is a complete illusion:
anybody using it (either code.python.org, or sys.revision), *cannot*
be agnostic of the specific technology.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Terry Reedy]

MRAB wrote:

Terry Reedy wrote:

Dirkjan Ochtman wrote:


It needs to be decided where the hg repositories will live. I'd like
to propose to keep the hgwebdir instance at hg.python.org. This is an
accepted standard for many organizations, and an easy parallel to
svn.python.org. The 2.7 (trunk) repo might live at
http://hg.python.org/main/, for example, with py3k at
http://hg.python.org/py3k/.


I would very much like the 'k' dropped from the py3 name. It was a 
funny joke when py3 was vaporware, now it is excess baggage which only 
puzzles non-insiders and newcomers.


I think the two repos should be either symmetrically named

hg.python.org/py2
hg.python.org/py3

If one must be designated 'main', it should be py3.

Continuing to call py2 'main' will continue to discourage use of py3.


We could regard py3k as the phase from the original concept of Python 3
to its 'prototype', Python 3.0. 


Right. And that phase is over, especially with Barry posting today on 
python-list that there will be no more 3.0.x releases ever.



Python 3.1 would be the first 'real/usable' version.


Right. 'is'.  as Barry also posted.

tjr

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

>> This is exactly why I was asking for your advice - I can't work out how to
>> work effectively with win32text as it stands myself, so remain stuck
>> accidently checking in files with inappropriate line endings and stuck
>> working out how to move pywin32's CVS repo with abandoning the very
>> primitive EOL safety it offers...
> 
> As long as the difference between \r\n- and \n-based files is clear
> and can be reasoned about, I don't see why having some of both (I'm
> assuming an overwhelming majority will have one, and only a few the
> other) is a big problem. But feel free to enlighten me!

If "both" means "both the server side test, and win32text", then Mark
already gave the answer: he cannot use win32test because he does not
know how to operate it (and doesn't bother studying its source code
to understand it in detail).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 3.1 buildbots

[Martin v. Löwis]

I have now turned the 3.0 buildbots off, and created new
builders for 3.1.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Brett Cannon]

On Fri, Jul 3, 2009 at 14:00, "Martin v. Löwis"  wrote:

> > Should we consider adding a sys.revision attribute and begin the
> > deprecation of sys.subversion?
>
> I wouldn't mind killing sys.subversion "right away" (i.e. in trunk
> and 3k - obviously it has to stay in 2.6 and 3.1, and all the older
> branches).
>
> I'm -1 on calling it "sys.revision", as this makes it difficult to
> tell what the actual versioning system was, and hence how the
> data should be interpreted. It will already be a problem for 2.6,
> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
> which will surely crash existing applications.
>
> I'm not sure what the motivation for a sys.revision is; it's
> probably similar to the desire of calling the machine code.python.org
> (instead of hg.python.org). It gives the illusion of being agnostic
> of the actual RCS being used. However, this is a complete illusion:
> anybody using it (either code.python.org, or sys.revision), *cannot*
> be agnostic of the specific technology.


We could add another value in the tuple that specifies the VCS: ('CPython',
'branches/release25-maint', '61464', 'svn'). I agree that VCSs are not
universally the same, but the concept of a revision is universal.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Brett Cannon]

On Fri, Jul 3, 2009 at 11:22, Terry Reedy  wrote:

> Dirkjan Ochtman wrote:
>
>  It needs to be decided where the hg repositories will live. I'd like
>> to propose to keep the hgwebdir instance at hg.python.org. This is an
>> accepted standard for many organizations, and an easy parallel to
>> svn.python.org. The 2.7 (trunk) repo might live at
>> http://hg.python.org/main/, for example, with py3k at
>> http://hg.python.org/py3k/.
>>
>
> I would very much like the 'k' dropped from the py3 name. It was a funny
> joke when py3 was vaporware, now it is excess baggage which only puzzles
> non-insiders and newcomers.
>

Is it really that confusing? I have never heard of anyone asking "what is
py3k?" Plus I like keeping that bit of Python history around. I know I still
use py3k as shorthand for Python 3.x. And we are not that serious of a
bunch. =)


>
> I think the two repos should be either symmetrically named
>
> hg.python.org/py2
> hg.python.org/py3
>

If we make it universal I say it should be '2.x' and '3.x'. The whole 'py'
prefix is redundant.


>
> If one must be designated 'main', it should be py3.
>
> Continuing to call py2 'main' will continue to discourage use of py3.


Yeah, 2.x shouldn't be special anymore.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> We could add another value in the tuple that specifies the VCS:
> ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that
> VCSs are not universally the same, but the concept of a revision is
> universal.

Actually, I think that's not the case. For bzr, the usual way of
identifying a revision is by revision number, which, however, is not
unique within a project, as each branch will use contiguous integers
for numbers. There are also unique identifications - so a bzr revision
has actually two numbers.

More general, in a DVCS, it is not possible to access the revision being
referred to by such a tuple. For sys.subversion, if [0]=='CPython', then
you could go to svn.python.org. For a DVCS, the revision being
identified may not be publically available, or may not live on a host
that you can infer from your proposed sys.revision.

For cloned branches, I wonder how sys.revision[1] would be computed.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Brett Cannon]

On Fri, Jul 3, 2009 at 14:52, "Martin v. Löwis"  wrote:

> > We could add another value in the tuple that specifies the VCS:
> > ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that
> > VCSs are not universally the same, but the concept of a revision is
> > universal.
>
> Actually, I think that's not the case. For bzr, the usual way of
> identifying a revision is by revision number, which, however, is not
> unique within a project, as each branch will use contiguous integers
> for numbers. There are also unique identifications - so a bzr revision
> has actually two numbers.
>
> More general, in a DVCS, it is not possible to access the revision being
> referred to by such a tuple. For sys.subversion, if [0]=='CPython', then
> you could go to svn.python.org. For a DVCS, the revision being
> identified may not be publically available, or may not live on a host
> that you can infer from your proposed sys.revision.
>
> For cloned branches, I wonder how sys.revision[1] would be computed.


So are you saying we should drop the idea of a revision value altogether, or
just embrace the differences and add a sys.mercurial attribute?

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> So are you saying we should drop the idea of a revision value
> altogether, or just embrace the differences and add a sys.mercurial
> attribute?

That's what I would propose. It should be a best-effort(*) approach at
providing all information that is needed to really find the source
used for the specific version.

Regards,
Martin

(*) even for svn it was best-effort only in case there were local
modifications.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Dirkjan Ochtman]

On Fri, Jul 3, 2009 at 23:41, Brett Cannon wrote:
> If we make it universal I say it should be '2.x' and '3.x'. The whole 'py'
> prefix is redundant.

Right, I was aiming for /python/2.x and /python/3.x as well.

Actually, I currently have /cpython to also make CPython less special
among it's peers, but that idea was met with some resistance on
#python-dev.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Brett Cannon]

On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman  wrote:

> On Fri, Jul 3, 2009 at 23:41, Brett Cannon wrote:
> > If we make it universal I say it should be '2.x' and '3.x'. The whole
> 'py'
> > prefix is redundant.
>
> Right, I was aiming for /python/2.x and /python/3.x as well.
>
> Actually, I currently have /cpython to also make CPython less special
> among it's peers, but that idea was met with some resistance on
> #python-dev.


Don't worry about doing that right now. When the stdlib gets separated out
we can revisit this. But for now just leave it as python.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Barry Warsaw]

On Jul 3, 2009, at 5:00 PM, Martin v. Löwis wrote:


I'm -1 on calling it "sys.revision", as this makes it difficult to
tell what the actual versioning system was, and hence how the
data should be interpreted. It will already be a problem for 2.6,
when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
which will surely crash existing applications.


I can release a 2.6.3 right before the cut-over (well, just about any  
time between now and August 1st).  Should we just plan now for a 2.6.3  
on say July 24th, with a release candidate on July 20th?



I'm not sure what the motivation for a sys.revision is; it's
probably similar to the desire of calling the machine code.python.org
(instead of hg.python.org). It gives the illusion of being agnostic
of the actual RCS being used. However, this is a complete illusion:
anybody using it (either code.python.org, or sys.revision), *cannot*
be agnostic of the specific technology.


Agreed.  I originally chose code.python.org because I didn't want to  
be biased (maybe I should have been :).   +1 for hg.python.org.  I'd  
prefer to spell out sys.mercurial_revision.


-Barry



PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> - First of all, I've got the basic conversion down, I've done it a few
> times now, with progressively better results. You can view some
> results at http://hg.python.org/, which has a preliminary cpython
> repository. *** The changeset hashes for that repo will change, so you
> won't be able to commit or pull from it in the future.***

I'd like to see as much infrastructure going "life" well in advance
of the switch over. Ideally, when we do switch, there should be no
surprises (i.e. all problems should be known, and, ideally, be
resolved).

> * Set up hg-ssh infra (should be easy)
> * Set up hooks (should be mostly straightforward)
> * Set up roundup integration (should be made easier by quick revision
> map hgweb extension)
> * Write docs

These are the ones that I was thinking of (and mentioned before, so
they got onto your list :-)

I would drop the roundup integration from the things that need to
be done pre-migration - there currently is no svn integration, so
not having it for hg is not a step backwards.

> * buildbots: both the regular and the community build masters must
> be notified. Fortunately buildbot includes support for hg. I've also
> implemented this for Mercurial itself, so I don't expect problems
> here.

This should also be setup before the switchover, with a few test
buildslaves (say, a Unix one and a Windows one).

> In a DVCS, the committers are not necessarily the same
> people who push, and so we can't check if the committer is a
> contributor. We could use a hook to check if the committer is a
> contributor if we keep a list of registered contributors.

In the first sentence, you say that it can't actually work - so I
think we should drop the test.

> I propose that the revision identifier will be the short version of
> hg's revision hash, for example 'dd3ebf81af43', augmented with '+'
> (instead of 'M') if the working directory from which it was built was
> modified. This mirrors the output of the hg id command, which is
> intended for this kind of usage.

I would like to see this well before the switch also. It could be
a patch (unified diff) stored in the tracker, or it could be an actual
branch that then needs to get merged with all active branches (IIUC).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

Barry Warsaw wrote:
> On Jul 3, 2009, at 5:00 PM, Martin v. Löwis wrote:
> 
>> I'm -1 on calling it "sys.revision", as this makes it difficult to
>> tell what the actual versioning system was, and hence how the
>> data should be interpreted. It will already be a problem for 2.6,
>> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43',
>> which will surely crash existing applications.
> 
> I can release a 2.6.3 right before the cut-over (well, just about any
> time between now and August 1st).  Should we just plan now for a 2.6.3
> on say July 24th, with a release candidate on July 20th?

I'm fine with that plan - but the original problem remains. We will
surely release 2.6.4 at some point, and it will have a different version
identification (based on hg rev ids).

So those existing applications (which are probably few) will then crash
for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
arrange to fake sys.subversion somehow (e.g. freezing it on the last
subversion revision - which might still break applications that insist
on accessing the revision mentioned - not sure whether such applications
actually exist).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Barry Warsaw]

On Jul 3, 2009, at 6:15 PM, Martin v. Löwis wrote:


I'm fine with that plan - but the original problem remains. We will
surely release 2.6.4 at some point, and it will have a different  
version

identification (based on hg rev ids).

So those existing applications (which are probably few) will then  
crash

for 2.6.4, unless we continue maintaining 2.6 in subversion, or just
arrange to fake sys.subversion somehow (e.g. freezing it on the last
subversion revision - which might still break applications that insist
on accessing the revision mentioned - not sure whether such  
applications

actually exist).


Doesn't Mercurial support an Subversion bridge?  Would it be possible  
to provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5,  
2.6, and 3.1?  If so, then the release managers would simply have to  
cut their releases from the svn copy instead of the hg master.  /All/  
other work would be done from the hg master.  This shouldn't be too  
much of a burden since it's done so rarely and would end with the EOL  
of each of those branches.


It would mean maintaining the bridge until all currently released  
versions are EOL.


If that's not possible or feasible, then given the documented  
sys.subversion semantics, I think we should just freeze the tuple at  
e.g. ('CPython', 'branches/release26-maint', None).


-Barry



PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 4/07/2009 12:04 AM, Nick Coghlan wrote:


However, I expect that would still be painful to work with for Windows
developers, even if it prevented any line ending problems from making
their way into the main repository. I believe that is where the
win32text extensions can help. Looking at the Wiki page for win32text
[1], I believe it would be a matter of configuring the extension to
encode and decode all files with the extensions:

*.py
*.pyw
*.h
*.c
*.in
*.rst
*.asdl

That said, I don't see a way to tell win32text to also translate files
which don't have an extension at all (e.g. NEWS or ACKS), and there
doesn't seem to be a way to tell it to *skip* files matching certain
patterns (if there was, we could just tell it to ignore extensions like
.bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert
everything else)


The big problem I have with win32text is that the rules are not 
versioned, meaning we will rely on every (windows only?) developer 
manually enabling the win32text extension, then manually adding these 
rules to an unversioned hgrc file.  Should the rules change over time, 
we will need to communicate that to every developer.  Contrast this 
with, say, SVN, where the rule is a versioned property on the file.


Further, if a repo *already* has mixed line endings and the file doesn't 
appear in the rules, win32text makes *lots* of noise regarding existing 
line endings and to "review your rules before you commit."  In effect, 
it seems to me that win32text will only work effectively if *all* 
developers using a repo are using it, or none are, or other steps have 
been put in place to ensure \r\n line endings in the repo are otherwise 
impossible.  I've completely failed to use win32text on the mozilla hg 
repo for exactly this reason.


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Scott Dial]

Tarek Ziadé wrote:
> On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote:
>> Another general principle: even in the draft PEP, say "is", not "will
>> be".
> 
> Ok I'll fix that. That's a French stuff : in french, "will be" isn't
> speculative at all.
> 

I don't think "will be" is necessarily speculative in English either. I
think the issue is that after the PEP is implemented, the document lives
on. And when one reads, "X will be done." If "X" is not done in the
current implementation, it is unclear whether that is an error or a
promise that at some point in the future the implementation will be
changed to do "X". In other words, the PEP will live on long after you
have completed the implementation and it's at that point that
occurrences of "will be" in the PEP become speculative.

Someone feel free to correct me if I am incorrect about the desired tone
and use of the document..

-Scott

-- 
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:

On Fri, Jul 3, 2009 at 15:31, Mark Hammond  wrote:

So we must work without effective EOL support?  I fear we will end up like
the mozilla hg repo with some files in windows line endings and some with
linux.  While my editing tools are good enough to preserve existing EOL
styles, I've found myself accidentally checking in new \r\n terminated files
in a repo which otherwise uses \n line endings.  IMO, SVN's EOL support was
better than no EOL support.


This is why we'll have hooks -- to prevent you from pushing changesets
with inappropriate, to say the least, and, if you're willing to do a
little bit of extra work, to prevent you from committing them.


This is exactly why I was asking for your advice - I can't work out how to
work effectively with win32text as it stands myself, so remain stuck
accidently checking in files with inappropriate line endings and stuck
working out how to move pywin32's CVS repo with abandoning the very
primitive EOL safety it offers...


As long as the difference between \r\n- and \n-based files is clear
and can be reasoned about, I don't see why having some of both (I'm
assuming an overwhelming majority will have one, and only a few the
other) is a big problem. But feel free to enlighten me!


I'm surprised it isn't obvious.  Think about it this way: CVS had basic 
EOL support and SVN gave slightly better support, and we leveraged this 
support in both those tools.  The end result is a very clean repo with 
consistent line endings and a distinct lack of developer friction 
between the platforms.


Your position seems to be that we *never* needed EOL conversion, 
especially in SVN which offered decent hooks, and that having a source 
repo with a mismatch and unpredictable line endings isn't really a 
problem at all.


Even just at face value, it seems clear we have taken full advantage of 
that feature for around 1.5 decades, so to suggest we never actually 
needed it, or at least don't need it now for some reason, seems slightly 
disingenuous...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

> Doesn't Mercurial support an Subversion bridge?  Would it be possible to
> provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6,
> and 3.1?  If so, then the release managers would simply have to cut
> their releases from the svn copy instead of the hg master.  /All/ other
> work would be done from the hg master.  This shouldn't be too much of a
> burden since it's done so rarely and would end with the EOL of each of
> those branches.
> 
> It would mean maintaining the bridge until all currently released
> versions are EOL.

That sounds very complicated (but Dirkjan should correct me if I'm
wrong).

I have already mentally prepared to port welease to hg, to create
the missing 2.4 and 2.5 releases.

> If that's not possible or feasible, then given the documented
> sys.subversion semantics, I think we should just freeze the tuple at
> e.g. ('CPython', 'branches/release26-maint', None).

That would work for me (I had to re-read the documentation to see
that None is a valid, documented value for version, and allowed
if "the tree was exported").

Interesting to notice that sys.subversion didn't even last the
life time of one feature release :-)

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

[Nick Coghlan]

Tarek Ziadé wrote:
> 2009/7/3 Paul Moore :
>> Does this sound sensible? Tarek, would you be OK with me attempting to
>> modify your prototype to support this protocol? Are there any tests
>> for PEP 376, so that I can confirm I haven't completely broken
>> something? If I can, I'll knock up some simple prototype importers for
>> non-standard examples, and see how they work with all this.
> 
> Yes that's exactly what I was thinking about after the discussion we
> had in the other
> thread. This change would allow much more flexibility.

It's also very similar to the ideas that started to tick over in the
back of my head after the previous discussion (only far more fleshed out
that anything I got to - fortunately I read Paul's message before
spending too much time on it!)

Specifically regarding this comment from Paul:

> Finder.uninstall(distribution_name, filter=callable, installer=name)
> Uninstall the given distribution. It's likely that many finders will
> be read-only. In that case, this function should return None.
> Otherwise, return a list of the "files" removed. (This may need some
> clarification, as many finders won't have a concept of a "file name").

PEP 302 finders and loaders actually *are* expected to have a concept of
a "file name". For Python modules they they are expected to have it so
they can populate __file__ correctly (runpy is also a lot happier with
them when they expose get_filename(), a relatively recent addition to
the PEP 302 API). For other resources they're expected to have it so the
get_data() API can work correctly.

One other thing to consider: it may be desirable to have hooks for
meta_path and path_hooks in distutils that are checked *before* the full
import hooks in the sys module. The reason I say this is that it would
allow someone to override just the metadata retrieval (e.g. to pick up
the extra dependency information saved by setuptools and friends)
without having to worry about breaking the actual imports from those
locations.

Cheers,
Nick.

P.S. +lots on using 'metadata' in the PEP 376 method names rather than
the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
gratuitous in this case.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Mark Hammond wrote:
> On 4/07/2009 12:04 AM, Nick Coghlan wrote:
> The big problem I have with win32text is that the rules are not
> versioned, meaning we will rely on every (windows only?) developer
> manually enabling the win32text extension, then manually adding these
> rules to an unversioned hgrc file.  Should the rules change over time,
> we will need to communicate that to every developer.  Contrast this
> with, say, SVN, where the rule is a versioned property on the file.
> 
> Further, if a repo *already* has mixed line endings and the file doesn't
> appear in the rules, win32text makes *lots* of noise regarding existing
> line endings and to "review your rules before you commit."  In effect,
> it seems to me that win32text will only work effectively if *all*
> developers using a repo are using it, or none are, or other steps have
> been put in place to ensure \r\n line endings in the repo are otherwise
> impossible.  I've completely failed to use win32text on the mozilla hg
> repo for exactly this reason.

I didn't realise Mercurial didn't have a way for a repository to provide
versioned extension settings, but in this case I think using our own
server side hook can deal with the problem (either adding it to the
existing whitespace hook that checks for tab/space sanity in Python
files, or else as a separate hook that just checks line endings).

Such a hook should be able to base the determination of which files it
checks for \r\n line endings on a normal versioned file in the
repository. For everything that *isn't* mentioned in that file, the hook
would check that it had \n line endings.

I believe that would address both of your concerns:
  1. A server side hook means that the rules are enforced on everyone
regardless of whether they use win32text or not (e.g no Linux developers
corrupting a .dsp file by opening it and saving it with \n line endings)
  2. The server side hooks rules will be versioned, allowing people to
copy them to their hgrc file so that win32text will correctly handle the
line endings for them.

Also, on another read of the win32text instructions, I believe the
configuration actually *does* support negative filters, it just doesn't
explain them very well (it only mentions them in the context of
overriding default settings in the config file for a specific
repository). I think a .hgrc file along the following lines might do
what we want (I'm not developing on Windows though, so I haven't
actually tested it):

=
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave Windows specific and known binary files alone
**.bat = !
**.sln = !
**.vcproj = !
**.vsprops = !
**.dps = !
**.dsw = !
**.wse = !
**.ico = !
**.bmp = !
=

Assuming we can find settings that do what we want then a sample HGRC
file could easily be provided in the Misc directory alongside things
like python-mode.el.

An alternative to the above would be to store *everything* (even the
Windows specific files) in the repository with \n line endings (still
enforced by a server side hook), and then *require* that Windows users
run win32text with settings like the following:

=
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:
# But leave known binary files alone
**.ico = !
**.bmp = !

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
# But leave known binary files alone
**.ico = !
**.bmp = !
=

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote:


As long as the difference between \r\n- and \n-based files is clear
and can be reasoned about, I don't see why having some of both (I'm
assuming an overwhelming majority will have one, and only a few the
other) is a big problem. But feel free to enlighten me!


In that case, and given that I expect more Windows users to be clueless 
about EOL issues than users of other operating systems, I propose we 
check in all files initially with Windows line endings 


Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Brett Cannon wrote:
> On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman  > wrote:
> Actually, I currently have /cpython to also make CPython less special
> among it's peers, but that idea was met with some resistance on
> #python-dev.
> 
> Don't worry about doing that right now. When the stdlib gets separated
> out we can revisit this. But for now just leave it as python.

Yes, if we ever do that split we can leave the common code (stdlib, test
suite) under /python and move the CPython specific stuff to /cpython.

+1 on 2.x and 3.x for the main development branches (although that does
lead me to wonder if we should just make the maintenance branches 2.6,
3.1, etc).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 4/07/2009 11:20 AM, Nick Coghlan wrote:


I didn't realise Mercurial didn't have a way for a repository to provide
versioned extension settings, but in this case I think using our own
server side hook can deal with the problem (either adding it to the
existing whitespace hook that checks for tab/space sanity in Python
files, or else as a separate hook that just checks line endings).

Such a hook should be able to base the determination of which files it
checks for \r\n line endings on a normal versioned file in the
repository. For everything that *isn't* mentioned in that file, the hook
would check that it had \n line endings.

I believe that would address both of your concerns:
   1. A server side hook means that the rules are enforced on everyone
regardless of whether they use win32text or not (e.g no Linux developers
corrupting a .dsp file by opening it and saving it with \n line endings)
   2. The server side hooks rules will be versioned, allowing people to
copy them to their hgrc file so that win32text will correctly handle the
line endings for them.


That sounds like it should work - presumably mention of this should go 
into the PEP.


(While that is great for Python, it is a shame I probably can't use the 
same strategy for other hg repos I care about as I don't have access to 
server side hooks - but that if off-topic for this list.)



Also, on another read of the win32text instructions, I believe the
configuration actually *does* support negative filters, it just doesn't
explain them very well (it only mentions them in the context of
overriding default settings in the config file for a specific
repository). I think a .hgrc file along the following lines might do
what we want (I'm not developing on Windows though, so I haven't
actually tested it):


This *appears* to be correct at first glance, but in practice it doesn't 
interact well with wildcards specifications - particularly 
'**=cleverencode'.  I started a thread on the ML and submitted a patch a 
few months ago to fix this and it was accepted, but sadly it seemed to 
have dropped off the queue somewhere.  The patch now conflicts, and I've 
promised to resubmit it when I find time.  But even with that in place 
it doesn't address the more general problem that when only *some* 
developers use the extension - mixed mode files are still very possible, 
at which point win32text starts reporting enough spurious conflicts to 
become unusable for me.  Eg, doing a clean checkout of mozilla with 
win32text enabled results in a working tree with hundreds of files 
reporting every line in the file has changed.


It is worth noting that win32text also works on Linux, and that in 
theory, win32text should be able to allow Linux users to work seamlessly 
with a \r\n repo, in the same way it promises to allow Windows users to 
work with \n repos.  But in practice, it is simply never used by Linux 
developers, even in the hg core dev team.  Patches or comments regarding 
win32text are always met with "but no one here ever uses it".  If they 
did, and some of the core hg team tried to experiment with win32text and 
mixed line ending repos the problems would, I believe, become 
self-evident...


Cheers,

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Martin v. Löwis wrote:
> I would drop the roundup integration from the things that need to
> be done pre-migration - there currently is no svn integration, so
> not having it for hg is not a step backwards.

That's not quite true - the tracker turns text like "r64537" into a link
to the appropriate changeset on svn.python.org. It would be unfortunate
to lose that easy ability to indicate when a bug was fixed (or introduced).

This is more something on the roundup side though: recognising the
presence of a hg id in a comment and turning it into a link to
hg.python.org.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376

[Nick Coghlan]

Scott Dial wrote:
> Someone feel free to correct me if I am incorrect about the desired tone
> and use of the document..

I don't think we're particularly consistent. For example, I actually
ending up adding a note to PEP 343 indicating that people shouldn't read
too much into the verb tense in that document because we were working on
it for over a year and things that were in the future when we started
were well in the past by the time we finished.

In general, I agree that using present tense ("is") in specifications
and documentation such as PEPs ends up standing the test of time better
than using future tense ("will be").

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Mark Hammond wrote:
> This *appears* to be correct at first glance, but in practice it doesn't
> interact well with wildcards specifications - particularly
> '**=cleverencode'.  I started a thread on the ML and submitted a patch a
> few months ago to fix this and it was accepted, but sadly it seemed to
> have dropped off the queue somewhere.  The patch now conflicts, and I've
> promised to resubmit it when I find time.  But even with that in place
> it doesn't address the more general problem that when only *some*
> developers use the extension - mixed mode files are still very possible,
> at which point win32text starts reporting enough spurious conflicts to
> become unusable for me.  Eg, doing a clean checkout of mozilla with
> win32text enabled results in a working tree with hundreds of files
> reporting every line in the file has changed.

Ah, the scope of the issues begin to become clear...

While a server side hook should be able to deal with the mixed mode file
problem, I'm not sure what can be done about the problems with properly
configuring win32text.

If wildcard specifications don't interact properly with filtered
negative specifications then that would appear to rule out that
approach. The presence of text files without extensions like NEWS and
ACKS in the Python repository appears to rule out the use of positive
filters to select only the files that are stored in the repository with
\n line endings.

I spent some more time exploring the approach recommended on the
win32text documentation page as a possible way of handling the situation:
1. Store all text files (even Windows specific ones) in \n format in the
repository
2. Apply the win32text precommit hook to forbid the introduction of \r\n
line endings
3. Use the recommended settings from the win32text documentation:
===
[extensions]
hgext.win32text=

[encode]
# Encode files that don't contain NUL characters.
** = cleverencode:

[decode]
# Decode files that don't contain NUL characters.
** = cleverdecode:
===

This would be the equivalent of setting "svn:eol-style native" on every
file in the repository.

However, in running Martin's query (svn pg -R svn:eol-style .|grep CRLF)
over 2.x and 3.x checkouts, I found that \r\n line endings are currently
enforced for:
 - .bat files under Tools/buildbot
 - .dsp and .dsw files under PC/VC6
 - Lib/email/test/data/msg_26.txt

I believe the rationale for the first two is to allow a source tarball
to be prepared on Linux but still be usable on Windows (e.g. see [1]).

I'm not clear on the rationale for the explicit CRLF line ending on the
email test message, but I would guess it is to ensure that the email
module can handle CRLF line endings correctly regardless of platform.

Only VC6 files appear on the list because later versions of Visual
Studio actually tolerate \n line endings in their project files.

Mercurial's heuristic handling of text vs binary and expected line
endings fails completely for these use cases.

I think there needs to be a solid answer in place for these use cases
before the actual migration to Mercurial takes place. A hand-waving "use
win32text" isn't enough - it needs to be "use win32text with these exact
settings" (with server side hook support to enforce the rules).

And since Mercurial doesn't even allow us to say "this is a binary file"
 the way CVS used to I'm currently not seeing any way for that to happen
except for win32text to be updated to correctly handle wild cards in
combination with negative filters.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Stephen J. Turnbull]

Dirkjan Ochtman writes:
 > On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull wrote:
 > > I'll have to try them again now that 1.3 is out, but I found Mercurial
 > > named branches fundamentally unsuited to release management.
 > 
 > Can you explain why, please? It's not clear from what you say
 > below.

Well, the main problem I had was one that you say has improved: the
various issues of confusion due to the presence of multiple active
heads in a single repository.

IME, Mercurial strongly encourages a non-branching style.  Although I
can't fully explain in concrete terms what makes me feel that way,
it's certainly consistent with your own inclination to advise "subset
branches".  Part of it comes from the fact that you can't have a
single revision on two branches.  I would really like the node of a
release branch to be on both the branch and the mainline so that the
tag appears in the history of both, but that's not possible.

Another issue (which you say has improved) is the handling of multiple
heads.  With unnamed heads, it's just too painful to leave them
hanging around, so you merge them immediately (in fact, all XEmacs
committers currently active use either "pull -u" or the fetch
extension).

With named branches, the additional heads do tend to hang around.  I
found that named branches tended to get inadvertantly pushed, and
worse, they'd end up being the tip, which Mercurial treats specially.
In one case a completely private branch inadvertantly got pushed into
my "pristine" clone of the public repo, and from there into the public
repo, where (since it had been the tip in my private workspace) it
ended up as the tip in public.  Embarrassing, to say the least.
Fortuntely, the branch was more or less ready to be pushed anyway, but
one of my colleagues ended up working on that branch without realizing
that he wasn't on (his) mainline any more, and wondered why some
previously done work suddenly disappeared.  A good time was had by
nobody involved.

I don't know if that has been fixed in hg; the experience was painful
enough that my workflow adjusted immediately.

 > > Ditto named branches.  The problem is that (unless the internal
 > > implementation has changed very recently) a Mercurial revision can be
 > > on exactly one named branch (or on the trunk).
 > 
 > That's still true.
 > 
 > > Which defeats the purpose of having named branches, really.  (I mean
 > > the version control purpose; obviously it still can save disk space.)
 > 
 > Why does it defeat the purpose? What, in your opinion, is the
 > purpose?

I use named branches to collect a sequence of revisions as a named
object, for viewing and manipulation, as differentiated from some
other sequence, for *various* values of "some other sequence".  The
problem is that suppose you have a branch A off the trunk, and you
then (several revisions down the line of development) branch B off A.
Now A meanders off and runs into "not ready for prime time" problems,
while B just swims along.  Problem is, you can't easily find the
history of B relative to the trunk because much of its history (since
forking from the *trunk*) is labeled A.

 > > Unless you're really short on space, though, that's not a big deal.
 > > What would be more important to me (not that I matter for the purpose
 > > of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be
 > > the other way around: pulling an external branch into a named branch.
 > > I have a feeling that working with such a repository with others would
 > > be a little difficult.
 > 
 > Can you give an example?

No, I haven't tried it.  What concerns me is that I suspect that the
branch name will end up as part of the revision's internal identifier,
and that means that if you and I separately create the same named
branch we have to choose exactly the same name or the branches won't
be recognized as the same, resulting in the mother of all spurious
merge conflicts.

 > > As others (MvL, I think) have commented, this isn't really relevant to
 > > Python which generally has four mainlines going at once.  I don't see
 > > why the requirements are going to change with the shift to hg, and I
 > > see no reason why hg won't handle the existing workflow just fine.
 > 
 > It will handle it, for sure, but I think it would all go easier if we
 > could work with stricter subset branches (and leave the effective
 > cherrypicking for the occasional problem).

Sure, but what do you propose?  That we nuke Python 3.1, 3.2, and 2.6?
They're all pretty divergent from 2.7 by now, as well as from each
other.  Not to mention the "security branches" that are still around.
And individual developers are for sure going to do interesting things
in their private workspaces.

I see that George Brandl and Martin van Loewis seem to be accomodating
your viewpoint, but I don't get the impression that either you (as the
hg migration proponent) nor they (as core committers) realize how far
apart your assumptions are.  You are talking a

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Martin v. Löwis]

Nick Coghlan wrote:
> Martin v. Löwis wrote:
>> I would drop the roundup integration from the things that need to
>> be done pre-migration - there currently is no svn integration, so
>> not having it for hg is not a step backwards.
> 
> That's not quite true - the tracker turns text like "r64537" into a link
> to the appropriate changeset on svn.python.org. It would be unfortunate
> to lose that easy ability to indicate when a bug was fixed (or introduced).
> 
> This is more something on the roundup side though: recognising the
> presence of a hg id in a comment and turning it into a link to
> hg.python.org.

Ah, right. That must be done, of course (although I suppose there is
little hope to have the existing references continue to work).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Steven D'Aprano]

On Sat, 4 Jul 2009 04:22:57 am Terry Reedy wrote:

> I would very much like the 'k' dropped from the py3 name. It was a
> funny joke when py3 was vaporware, now it is excess baggage which
> only puzzles non-insiders and newcomers.

+1

Some day we'll be using Python3.7 and wondering what the "k" means. 
Alpha, beta,  kappa version?



-- 
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Mark Hammond]

On 4/07/2009 12:30 PM, Nick Coghlan wrote:
...

I think there needs to be a solid answer in place for these use cases
before the actual migration to Mercurial takes place. A hand-waving "use
win32text" isn't enough - it needs to be "use win32text with these exact
settings" (with server side hook support to enforce the rules).

And since Mercurial doesn't even allow us to say "this is a binary file"
  the way CVS used to I'm currently not seeing any way for that to happen
except for win32text to be updated to correctly handle wild cards in
combination with negative filters.


I agree with your conclusion.  My ruminating on this over the last few 
months leaves me thinking this would involve:


* my older 'accepted but then lost' hg patch to allow an explicit 'none' 
rule for a single file to override wildcards.


* win32text be enhanced to use a normal versioned file in the root of 
the repo, much like hgingore, where a project can maintain project wide 
rules.


* win32text be enhanced such that all python developers, regardless of 
platform, are willing to use this extension, even if the majority of 
files happen to use their native line ending (sauce for the goose is 
sauce for the gander, and all that...)


* commit hooks be implemented to enforce this - but this should not be 
necessary if the above was implemented and socially enforced.


Cheers,

Mark


Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Stephen J. Turnbull]

Nick Coghlan writes:

 > I'm not clear on the rationale for the explicit CRLF line ending on the
 > email test message, but I would guess it is to ensure that the email
 > module can handle CRLF line endings correctly regardless of platform.

IIRC, that's just the standard for text email.  Lines must be less
than 1000 characters long, should be no more than 78 characters long,
and are terminated by CRLF.  As they say, "honored more in the breach
than the observance", but that's what the RFCs say.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[P.J. Eby]

At 12:20 PM 7/4/2009 +0900, Stephen J. Turnbull wrote:

IME, Mercurial strongly encourages a non-branching style.  Although I
can't fully explain in concrete terms what makes me feel that way,
it's certainly consistent with your own inclination to advise "subset
branches".  Part of it comes from the fact that you can't have a
single revision on two branches.  I would really like the node of a
release branch to be on both the branch and the mainline so that the
tag appears in the history of both, but that's not possible.


I'm a total newbie to Mercurial - I've basically done little more 
than the tutorial, in fact.  So somebody correct me if what I'm about 
to say is completely stupid.


Wouldn't the simple thing to do in Mercurial, just be to use 
different repositories for long-lived branches?  I mean, if you're 
not merging them that much anyway, what's the point?  Isn't it just 
as easy to pull from another repository, as from another branch 
within that repository?


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Nick Coghlan]

Martin v. Löwis wrote:
> Ah, right. That must be done, of course (although I suppose there is
> little hope to have the existing references continue to work).

I'd guess that the only way to keep those functional is to keep
svn.python.org around in read-only mode.

Although I'm not entirely sure about that - would it be possible to
build up a mapping from svn revision numbers to hg repository IDs as the
history information is being migrated? If that wasn't too difficult, it
might be worth doing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial migration: progress report (PEP 385)

[Stephen J. Turnbull]

P.J. Eby writes:

 > Wouldn't the simple thing to do in Mercurial, just be to use 
 > different repositories for long-lived branches?  I mean, if you're 
 > not merging them that much anyway, what's the point?

I basically agree with that, and so does Dirkjan, I think.  I'm not
sure why he brought up the idea of using named branches.

 > Isn't it just as easy to pull from another repository, as from
 > another branch within that repository?

Not if the other repository is remote and offline, and pulling from a
remote does take more time, as do things like diffs against that
branch.  The temptation is to leave the remote repos on the public
host, rather than put in the effort to make sure your local clones are
up to date.  If you do, there are minor annoyances like that.  So
people who actually work on more than one line of development learn
pretty soon to keep a local clone for each repo they're interested
in.  Problem solved, IMO, but some people worry about the extra space
taken up.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com