Re: [Python-Dev] Backport troubles with mercurial
Am 29.12.2010 01:13, schrieb Amaury Forgeot d'Arc: > Hello, > > The PyPy project recently switched from svn to mercurial. Since this day I > have some > difficulties to perform simple tasks, and my questions did not receive > satisfying answers. > > I was sure the Python project would have the same issues; > so I started from the repositories in http://code.python.org/hg/ and tried > simple > merges between Python versions. > I would like several people to try the same example, and tell how they handle > it. > I'm new to mercurial, and I may have missed something important! > > > Let's say you have to do the simple change shown in the diff below, > and want to "fix" the the 3 usual versions: py3k, release31-maint, > release27-maint. > The diff was made against py3k. > > How would you do it with Mercurial? Please try to do it for real! > > "hg merge" is not the correct command: it merges whole branches, a change > comes with all its ancestors. What we want is to "cherry-pick" a single > change. > > "hg transplant" fails to apply the change to release31-maint because > of a tiny conflict (in the diff context!) that leaves you with an ugly > "hunks FAILED" and a .rej file you have to parse and apply by hand. Yes, this has been an irritation to me as well. It's probably not so hard for a Mercurial coredev to change transplant into generating inline conflict markers though. BTW, we would apply the diff in release31-maint and then "hg merge" it to py3k. transplant still applies to 2.7 though. > "hg transplant" seems to succeed in release27-maint, > but the test fails to run because "support" should be changed to > "test_support". > The change is already committed - to fix it another changeset is needed. > This does not look clean to me: both changesets will be pushed to the > repository, > and diff review (on the python-checkins mailing list) is almost impossible. Right. Seems to me transplant should get a command-line switch that always goes into break-and-continue mode, where the commit is only made after calling "hg transplant --continue". > Furthermore, "hg transplant" does not keep track of integrations. > There is a "transplants" file, but it stays in my local clone. You're wrong, it does keep track of integrations in the commit metadata. The only thing that's clone-local is the transplants cache file. Sadly, transplant only checks against this file, but it is very easy to write a hook that keeps it up to date; I've already done that work. > Faced with a similar case in pypy, we finally manually copied the files > between directories... and the fastest with our example is probably > to do the change manually in all three directories. > > There is surely a better way to work with maintenance branches! > PEP374 suggests to first modify the oldest release, but this does not > seems right to me (I have three reasons in mind) Would you care to explain those? > At the very least before the migration we need precise use cases like this > and recipes for common cases. Which is what we'll certainly get once we have the test repo and Brett is rewriting the dev guide. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 David Cournapeau > The easiest way I found to emulate git cherry-pick (which does exactly > what you want) with hg is to use import/export commands: > http://mercurial.selenic.com/wiki/CommunicatingChanges > > It is indeed quite a pain to use in my experience, because you cannot > easily refer to the original commit the cherry pick is coming from > (i.e. no equivalent to git cherry-pick -x), and the conflict > resolution is quite dumb. > This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes? And finally the biased question: can you find one improvement over the current situation with svn? One alternative is to be careful about where > you apply your patch > ( > http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to-handle-hg-import-rejects > ), > but that's not very convenient either. > I've read all this, and this method does not work, for several reasons: - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. we usually consider py3k as a child of both 2.7 and 3.1, and there is no common parent. - even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge. - in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository. (btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?) -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: > I've read all this, and this method does not work, for several reasons: > > - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. > we usually consider py3k as a child of both 2.7 and 3.1, and there is no > common > parent. Yes. Think of 2.7 as the oddball one, where revisions must be cherry-picked to, whereas the 3.x line can use the "commit in parent branch" system (and much more easily, since there are no changes in syntax and much less changes in code overall). > - even if there was one, there is the problem of changes specifically made > for 2.7 > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k > which has the disadvantage of cluttering the py3k change log. And think of the > horror > if someone forgets this dummy merge. No horror at all: the next committer notices the extra changes in his merge and removes them. *Never* commit merges without reviewing the diff. (The situtation is similar, by the way, to someone typing the wrong revision number when using svnmerge, and not noticing it because he does not review the diff.) > - in any case, the actual repositories in http://code.python.org/hg/ are not > clones > of each other, so "hg merge" won't work and "hg pull" will clone the whole > remote repository. Note that the repos on code.python.org are not the results of our conversion process. Those will be on hg.python.org. The former are repos contributed by Antoine (I think) that he uses with hgsubversion to work on Python using Mercurial right now; they will disappear after hg.python.org works. > (btw, now that I have "hg pull" another repo, how can I remove it? is my clone > broken forever?) No, you can "hg strip" away the root of the wrongly pulled revisions (which also strips all descendants), or use "hg clone -r XXX" to create a new clone with only a specified revision and all its ancestors. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl > > Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: > > > - even if there was one, there is the problem of changes specifically made > > for 2.7 > > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k > > which has the disadvantage of cluttering the py3k change log. And think of > > the > > horror > > if someone forgets this dummy merge. > > No horror at all: the next committer notices the extra changes in his merge > and > removes them. *Never* commit merges without reviewing the diff. (The > situtation is similar, by the way, to someone typing the wrong revision number > when using svnmerge, and not noticing it because he does not review the diff.) Except that it's easy to overlook a diff and not notice another small change mixed in your merge. Checking "hg merge -P" is probably better. But is it possible to "remove" the extra change? What worries me more is the requirement to find the correct branch before I can even edit the file. How would you, Georg, deal with doc patches (typos, bad markup)? With subversion it's easy to work on trunk, and then have svnmerge tell you which chunk is in conflict and does not apply to the maintenance branch. > > > - in any case, the actual repositories in http://code.python.org/hg/ are > > not clones > > of each other, so "hg merge" won't work and "hg pull" will clone the whole > > remote repository. > > Note that the repos on code.python.org are not the results of our conversion > process. Those will be on hg.python.org. The former are repos contributed > by Antoine (I think) that he uses with hgsubversion to work on Python using > Mercurial right now; they will disappear after hg.python.org works. ok. I used them because other the repos I found were really old. And http://hg.python.org/cpython/ probably needs an initial "dummy merge" on every branch. > > > (btw, now that I have "hg pull" another repo, how can I remove it? is my > > clone > > broken forever?) > > No, you can "hg strip" away the root of the wrongly pulled revisions (which > also strips all descendants), or use "hg clone -r XXX" to create a new clone > with only a specified revision and all its ancestors. "hg strip" worked for me. The root revision to strip was simply the first one (=oldest) in "hg outgoing". -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Wed, Dec 29, 2010 at 10:53, Amaury Forgeot d'Arc wrote: > And http://hg.python.org/cpython/ probably needs an initial "dummy merge" > on every branch. Yes, the present delays are all about getting that right. Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 29.12.2010 10:53, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl >> >> Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: >> >> > - even if there was one, there is the problem of changes specifically made >> > for 2.7 >> > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k >> > which has the disadvantage of cluttering the py3k change log. And think of >> > the >> > horror >> > if someone forgets this dummy merge. >> >> No horror at all: the next committer notices the extra changes in his merge >> and >> removes them. *Never* commit merges without reviewing the diff. (The >> situtation is similar, by the way, to someone typing the wrong revision >> number >> when using svnmerge, and not noticing it because he does not review the >> diff.) > > Except that it's easy to overlook a diff and not notice another small change > mixed in your merge. Checking "hg merge -P" is probably better. > But is it possible to "remove" the extra change? No; hg merging is DAG-based, so you will always merge all ancestors. The only way to "remove" it is a null-merge. > What worries me more is the requirement to find the correct branch before I > can > even edit the file. How would you, Georg, deal with doc patches > (typos, bad markup)? Finding the correct branch is not really hard. Bugfixes go to maintenance, features to trunk. You need to think about which category your change is right now too, when deciding what to backport/svnmerge. Accordingly, I would apply doc patches in release31-maint and merge them to trunk, probably all at once with one merge commit. >> > - in any case, the actual repositories in http://code.python.org/hg/ are >> > not clones >> > of each other, so "hg merge" won't work and "hg pull" will clone the whole >> > remote repository. >> >> Note that the repos on code.python.org are not the results of our conversion >> process. Those will be on hg.python.org. The former are repos contributed >> by Antoine (I think) that he uses with hgsubversion to work on Python using >> Mercurial right now; they will disappear after hg.python.org works. > > ok. I used them because other the repos I found were really old. > And http://hg.python.org/cpython/ probably needs an initial "dummy merge" > on every branch. Yes, that is a very old conversion result. We should get a new one in a few days. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >> What worries me more is the requirement to find the correct branch before I >> can >> even edit the file. How would you, Georg, deal with doc patches >> (typos, bad markup)? > > Finding the correct branch is not really hard. Bugfixes go to maintenance, > features to trunk. This works well indeed, provided there is only one maintenance branch. > You need to think about which category your change is > right now too, when deciding what to backport/svnmerge. No, today this decision can take place after the development, some tickets got reopened because a backport was needed. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 29.12.2010 11:09, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl : >>> What worries me more is the requirement to find the correct branch before I >>> can >>> even edit the file. How would you, Georg, deal with doc patches >>> (typos, bad markup)? >> >> Finding the correct branch is not really hard. Bugfixes go to maintenance, >> features to trunk. > > This works well indeed, provided there is only one maintenance branch. Which there is, except for security fixes (but they are rare and require release manager intervention anyway). >> You need to think about which category your change is >> right now too, when deciding what to backport/svnmerge. > > No, today this decision can take place after the development, > some tickets got reopened because a backport was needed. A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one. Of course you may have to think a bit more about bugfix *before* plunging into the code, but is that necessarily a bad thing? We had quite a few bugfixes not backported precisely because it is very easy not to care about them right now. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
Am 28.12.2010 18:08, schrieb Lukas Lueg: > Also, the > load_fast in lne 22 to reference x could be taken out of the loop as x > will always point to the same object That's not true; a debugger may change the value of x. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc wrote: > 2010/12/29 David Cournapeau >> >> The easiest way I found to emulate git cherry-pick (which does exactly >> what you want) with hg is to use import/export commands: >> http://mercurial.selenic.com/wiki/CommunicatingChanges >> >> It is indeed quite a pain to use in my experience, because you cannot >> easily refer to the original commit the cherry pick is coming from >> (i.e. no equivalent to git cherry-pick -x), and the conflict >> resolution is quite dumb. > > This is precisely why I proposed a specific example. > Which precise steps do you take? > How much typing or manual copy/paste is required for this very simple case? > Can you do the merge in less than 10 minutes? I don't know in this specific case. As I said, when I have to use hg, that's the technique I use, and you get the issue you mention. That's a hg limitation AFAICS. > And finally the biased question: > can you find one improvement over the current situation with svn? I am not involved in the hg conversion process nor its decision (I am not even a python committer). Cherry picking is actually easier to do with svn by "accident", because its merge method, up to 1.5 at least, was really dumb and never remembered the ancestors of a previous merge. As for a few data points which may or may not be relevant: in numpy we convereted from svn -> git recently, and it has worked pretty well, with numerous new contributions happening, and better, new contributors appearing. I have been the release manager for numpy for several years, and as such had to do the kind of tasks you mention numerous times with svn, and the only words that comes to mind when remember this period would not be appropriate on a public mailing list: I always found svn to be horrible. I started using git to make my life as release manager simpler, actually. I would be surprised if python's situation did not end up being similar to numpy's one. Other projects related to numpy made the changes to a DVCS (ipython, nipy, lean scikit) before and none of them ever regretted it AFAIK, and sometimes the people who become the most vocal proponents of the new tool were the most sceptic ones before. Certainly, very few people if any asked to revert the process. cheers, David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Amaury Forgeot d'Arc writes: > Which precise steps do you take [to cherrypick with export/import]? hg export -r $CHERRY .tmp.patch xemacs .tmp.patch ;; Move to end of log message, type "Cherry-picked from: ", then ;; C-u M-! hg id -i -r $CHERRY RET C-x C-s C-x C-c> hg import .tmp.patch worked for me the one time I tried it, IIRC. Now I use patch queues, and avoid cherry-picking as much as possible. (Avoiding cherry-pick is not going to be possible for 3.x <-> 2.x porting, of course, but in many cases there a simple patch application is not going to work at all, anyway. Should be relatively rare, no?) CHERRY can be a tag, revno, or other alias for the revision. You may want to add other identifying information with additional arguments to "hg id", but "hg id -i" give the canonical ID. You may want to fix up committer and date using the -u and -d arguments to hg import. > How much typing or manual copy/paste is required for this very simple case? For the simple case, the above can be scripted, and the script takes one argument, CHERRY. It would not be hard to automate adding the cherry-pick notice. > Can you do the merge in less than 10 minutes? If no conflicts, sure, no problem. Probably about 30 seconds, including adding the cherrypicked revid to the log message. With a script, maybe 1 second. If there are conflicts, it depends on the complexity of the conflicts. I can imagine this could get hairy with the export | import method and a megapatch, but I've never had to deal with that in practice. This is one reason why it's recommended that feature branches be constructed of multiple, clearly separated patches. (This may be difficult to do for complex changes, but it's usually a reasonable goal.) > And finally the biased question: > can you find one improvement over the current situation with svn? No, I can't find just one. How many are you willing to read? svn has the advantage (and yes, it's a big one) that developers are already used to it. Otherwise, everything goes in favor of any of the dVCSes. > - even if there was one, there is the problem of changes > specifically made for 2.7 that make no sense in py3k. This is a problem with the committer, not with the VCS. Such changes belong in a separate commit, with an appropriate log entry, in *any* version control system. If that is done, there's no problem, you just don't cherrypick that change. > You'd have to perform a "dummy merge" to py3k which has the > disadvantage of cluttering the py3k change log. I don't see why. Such dummy merges can help only if you are going to rely on hg merge. But you are not going to be merging 2.x and 3.x. True, some patches from the maintenance branch won't be applied to the trunk. This is expected, and it is a problem that needs to be solved, probably with a tool like svnmerge, which will do a dummy merge or a revert commit to exclude that revision from any future merge to trunk. Therefore, if we change our minds, it's perfectly safe to cherrypick. Any conflicts that occur are real conflicts, not merge artifacts. > And think of the horror if someone forgets this dummy merge. What horror? By cherry-picking you have assumed responsibility for managing future cherry-picks from that source to that target in any case; you can no longer expect reliable help from Mercurial. It's not at all clear to me that a dummy merge could accomplish anything useful, especially in the case of 2.x <-> 3.x cherry-picks. If on the other hand you're thinking of the merge-from-maint-branch case, as I say this is a problem that needs solving anyway, just as it did with svn. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >>> You need to think about which category your change is >>> right now too, when deciding what to backport/svnmerge. >> >> No, today this decision can take place after the development, >> some tickets got reopened because a backport was needed. > > A change can of course also be transplanted after the fact. It requires > another > merge, but usually a trivial one. No, it's not trivial with hg: this is the very subject of the thread! -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 David Cournapeau : > On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc > wrote: >> 2010/12/29 David Cournapeau >>> >>> The easiest way I found to emulate git cherry-pick (which does exactly >>> what you want) with hg is to use import/export commands: >>> http://mercurial.selenic.com/wiki/CommunicatingChanges >>> >>> It is indeed quite a pain to use in my experience, because you cannot >>> easily refer to the original commit the cherry pick is coming from >>> (i.e. no equivalent to git cherry-pick -x), and the conflict >>> resolution is quite dumb. >> >> This is precisely why I proposed a specific example. >> Which precise steps do you take? >> How much typing or manual copy/paste is required for this very simple case? >> Can you do the merge in less than 10 minutes? > > I don't know in this specific case. As I said, when I have to use hg, > that's the technique I use, and you get the issue you mention. That's > a hg limitation AFAICS. Yes, Georg identified three things that "hg transplant" should do better: - an option to not commit - a way to add conflict markers in the source instead of the .rej file (In this case, it may be just as easy to use the standard merge tools) - somehow share the "transplants" cache file between clones. > sometimes the people who become the most vocal proponents of the new > tool were the most sceptic ones before. I really was not sceptic before, and I certainly don't want to become one! But yesterday I was blocked the whole afternoon by something I still call an routine task with most other SCMs; and the only answer I get is "that's right, it's a pain" hg will certainly impose a change in the way we develop Python. I'm not sure everybody understands the consequences. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 29.12.2010 15:17, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl : You need to think about which category your change is right now too, when deciding what to backport/svnmerge. >>> >>> No, today this decision can take place after the development, >>> some tickets got reopened because a backport was needed. >> >> A change can of course also be transplanted after the fact. It requires >> another >> merge, but usually a trivial one. > > No, it's not trivial with hg: this is the very subject of the thread! I don't understand: if you make the same change in two branches, but in different changesets, and then merge these branches, Mercurial will usually notice that and not trouble you with conflicts. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 14:17, schrieb Victor Stinner: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > * http://bugs.python.org/issue10348 > * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot >^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/29 Georg Brandl : >>> A change can of course also be transplanted after the fact. It requires >>> another >>> merge, but usually a trivial one. >> >> No, it's not trivial with hg: this is the very subject of the thread! > > I don't understand: if you make the same change in two branches, but > in different changesets, and then merge these branches, Mercurial will > usually notice that and not trouble you with conflicts. Actually I never passed the first step: make the same change to two branches. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 8:17 AM, Victor Stinner wrote: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > * http://bugs.python.org/issue10348 > * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot > ^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to > write a patch, but I never used semaphores (POSIX or SysV). > > There is a third party module which looks complete and stable: > http://semanchuk.com/philip/sysv_ipc/ > > It is released under the BSD license. It supports semaphores, but also > shared memory and message queues. We don't need all of those, semaphores > would be enough. I added its author (Philip Semanchuk) to this thread. Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part. > I don't know how we should decide to use POSIX or SysV semaphores. It > looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD > based OSes), and POSIX is preferred on Linux. Hmmm, "preferred" is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. Speaking of POSIX IPC, I also have a posix_ipc module that's quite similar to sysv_ipc and the platform notes in the documentation tell one what support to expect for POSIX IPC under various platforms: http://semanchuk.com/philip/posix_ipc/#platforms Cheers Philip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Wed, 29 Dec 2010 23:12:28 +0900, "Stephen J. Turnbull" wrote: > worked for me the one time I tried it, IIRC. Now I use patch queues, > and avoid cherry-picking as much as possible. (Avoiding cherry-pick > is not going to be possible for 3.x <-> 2.x porting, of course, but in > many cases there a simple patch application is not going to work at > all, anyway. Should be relatively rare, no?) No. We merge bug fixes to 2.7 on a routine basis. It is the rule rather than the exception, by a long shot. (Features of course only go into trunk...but those are irrelevant to this conversation, since there's no requirement to merge them anywhere...as are bug fixes we choose not to backport, since those don't go into the Python3 maint branch either.) Some such merges apply cleanly, many more have only a few conflicts requiring only trivial tweaks, which are made easy by the in-line merge conflict markers. A few require re-engineering of the patch, and of course it is a bit of a pain with svnmerge to deal with the ones where the target file names have changed (but again, there aren't that many of those). [*] So, since we are going to be maintaining 2.7 for a while, this is a workflow problem that we *must* solve to make the hg transition worthwhile. I have no doubt that we will, but I also have no doubt we need to *solve* it, not just wave our hands. Many thanks to Georg and Djirkan for working on the problem(s). -- R. David Murray www.bitdance.com [*] I'm speaking of stdlib changes, which is what I do, but I suspect the situation is similar for the C code, and certainly the fact that bug fixes backported to Python3 maint are routinely backported to 2.7 is true. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
>> I would like to know if it should be considered as a release blocker. >> Georg Brandl said yes on IRC. > > Under the condition that it is within reason to fix it before the > release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> Hi all, What Victor says above is correct, although I wasn't aware > that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to > 7.2 it was broken but 7.2 worked OK in my limited testing. In any > case, the sysv_ipc module is mine and it's mature and you're welcome > to pillage it in whole or in part. The question really is whether you would be willing to port Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that this would be practically new code, but it would be much appreciated. One question is whether it's actually feasible to implement that API with SysV semaphores. >> I don't know how we should decide to use POSIX or SysV semaphores. >> It looks like SysV is preferred on FreeBSD and Darwin (and maybe >> all BSD based OSes), and POSIX is preferred on Linux. > > Hmmm, "preferred" is a tricky word here. I would phrase it slightly > differently: POSIX IPC is preferred everywhere (by me, at least) > because it's a more modern API. However, SysV IPC is fully supported > everywhere while the same can't be said about POSIX IPC. If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?) Alternatively, we could have a whitelist of systems on which POSIX IPC is used (==['linux']), and use sysv ipc everywhere else. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. Löwis" wrote: >>> I would like to know if it should be considered as a release blocker. >>> Georg Brandl said yes on IRC. >> >> Under the condition that it is within reason to fix it before the >> release. > > What *should* be possible is to disable building > SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, > multiprocessing locks would stop working on FreeBSD, and concurrent > futures; the tests would recognize this lack of features and get > skipped. > > Regards, > Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. jesse ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 8:17 AM, Victor Stinner wrote: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > * http://bugs.python.org/issue10348 > * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot > ^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to > write a patch, but I never used semaphores (POSIX or SysV). > > There is a third party module which looks complete and stable: > http://semanchuk.com/philip/sysv_ipc/ > > It is released under the BSD license. It supports semaphores, but also > shared memory and message queues. We don't need all of those, semaphores > would be enough. I added its author (Philip Semanchuk) to this thread. > > I don't know how we should decide to use POSIX or SysV semaphores. It > looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD > based OSes), and POSIX is preferred on Linux. > > Victor The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. Moving from POSIX IPC to SYSV should *not* be on the plate for a release blocker - that's a much larger task. jesse ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 18:54, schrieb Jesse Noller: > On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. Löwis" > wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. >>> >>> Under the condition that it is within reason to fix it before the >>> release. >> >> What *should* be possible is to disable building >> SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, >> multiprocessing locks would stop working on FreeBSD, and concurrent >> futures; the tests would recognize this lack of features and get >> skipped. >> >> Regards, >> Martin > > The multiprocessing test suite already skips the tests which use the > (broken) functionality on BSD correctly. This logic needs to be added > to the concurrent.futures library. I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place. Also, what specific test are you referring to? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: > The concurrent.futures tests should (like the multiprocessing test > suite) detect the lack of support and skip the tests on the broken > platforms. I'm sort of surprised FreeBSD support is still broken in > this way though (echoed by Philip) although it could be an issue on > that particular buildbot. If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. In other words, a user of concurrent.futures really needs to tweak their FreeBSD install to make in fully functional. There should be a way (through sysctl, presumably) to query the maximum number of semaphores and skip the relevant tests on that basis. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Compile() and Windows/Mac newlines
In python-list thread "Does Python 3.1 accept \r\n in compile()?"
jmfauth notes that
compile('print(999)\r\n', '', 'exec')
works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as
SyntaxError.
I started to respond that this is part of Py3 cleanup with newlines
converted on input and back-compatibility with ancient Python not
needed. Then I saw in 3.2 manual
"Changed in version 3.2: Allowed use of Windows and Mac newlines. Also
input in 'exec' mode does not have to end in a newline anymore. Added
the optimize parameter."
I verified second statement ("print(999)" works) (and remember commit
for third), but original above gives same error. Should "Allowed use of
Windows and Mac newlines." be deleted? What else could it mean other
than use of '\r' or '\r\n'?
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 2:14 PM, R. David Murray wrote: > On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: >> The concurrent.futures tests should (like the multiprocessing test >> suite) detect the lack of support and skip the tests on the broken >> platforms. I'm sort of surprised FreeBSD support is still broken in >> this way though (echoed by Philip) although it could be an issue on >> that particular buildbot. > > If I'm reading the issue correctly, it isn't that it doesn't work, > it's that the concurrent.futures tests fail because they create more > semaphores than the default FreeBSD install supports. In other words, > a user of concurrent.futures really needs to tweak their FreeBSD install > to make in fully functional. I think that's correct. Furthermore, the default 7.2 install doesn't have the necessary kernel modules loaded in order to use POSIX semaphores. See the notes here for FreeBSD: http://semanchuk.com/philip/posix_ipc/#platforms kldstat on my barely-customized 7.2 installation gives a list of three modules: kernel, acpi.ko and linux.ko (I assume because I asked for Linux executable format compatibility when I installed). bye Philip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 10:43 AM, Martin v. Löwis wrote: >> Hi all, What Victor says above is correct, although I wasn't aware >> that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to >> 7.2 it was broken but 7.2 worked OK in my limited testing. In any >> case, the sysv_ipc module is mine and it's mature and you're welcome >> to pillage it in whole or in part. > > The question really is whether you would be willing to port > Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that > this would be practically new code, but it would be much appreciated. > > One question is whether it's actually feasible to implement that API > with SysV semaphores. > >>> I don't know how we should decide to use POSIX or SysV semaphores. >>> It looks like SysV is preferred on FreeBSD and Darwin (and maybe >>> all BSD based OSes), and POSIX is preferred on Linux. >> >> Hmmm, "preferred" is a tricky word here. I would phrase it slightly >> differently: POSIX IPC is preferred everywhere (by me, at least) >> because it's a more modern API. However, SysV IPC is fully supported >> everywhere while the same can't be said about POSIX IPC. > > If you can make the above change, the question then is what API > multiprocessing semaphores should be built upon. It seems that you > are saying that they should use SysV IPC, and only fall back to > POSIX IPC if SysV IPC doesn't work/exist (are there any platforms > where this would be the case?) Actually, I'd probably recommend the opposite. IMO POSIX IPC is better designed and easier to work with so it would be my first choice. SysV would be a fallback for when POSIX IPC is unavailable, broken, incomplete, etc. bye Philip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile() and Windows/Mac newlines
2010/12/29 Terry Reedy :
> In python-list thread "Does Python 3.1 accept \r\n in compile()?"
> jmfauth notes that
> compile('print(999)\r\n', '', 'exec')
> works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as
> SyntaxError.
>
> I started to respond that this is part of Py3 cleanup with newlines
> converted on input and back-compatibility with ancient Python not needed.
> Then I saw in 3.2 manual
>
> "Changed in version 3.2: Allowed use of Windows and Mac newlines. Also input
> in 'exec' mode does not have to end in a newline anymore. Added the optimize
> parameter."
>
> I verified second statement ("print(999)" works) (and remember commit for
> third), but original above gives same error. Should "Allowed use of Windows
> and Mac newlines." be deleted? What else could it mean other than use of
> '\r' or '\r\n'?
$ ./python
Python 3.2b2 (py3k:87559, Dec 28 2010, 17:39:51)
[GCC 4.4.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> compile("print(999)\r\n", "blah", "exec")
at 0xb353e8, file "blah", line 1>
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile() and Windows/Mac newlines
On 12/29/2010 2:31 PM, Terry Reedy wrote:
In python-list thread "Does Python 3.1 accept \r\n in compile()?"
jmfauth notes that
compile('print(999)\r\n', '', 'exec')
works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as
SyntaxError.
I started to respond that this is part of Py3 cleanup with newlines
converted on input and back-compatibility with ancient Python not
needed. Then I saw in 3.2 manual
"Changed in version 3.2: Allowed use of Windows and Mac newlines. Also
input in 'exec' mode does not have to end in a newline anymore. Added
the optimize parameter."
I verified second statement ("print(999)" works) (and remember commit
for third), but original above gives same error. Should "Allowed use of
Windows and Mac newlines." be deleted? What else could it mean other
than use of '\r' or '\r\n'?
After tracing the questioned comment to B.Peterson's r76232 merged from
2.7 r76230 "fix several compile() issues by translating newlines in the
tokenizer", I decided to open http://bugs.python.org/issue10792
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 14:14:03 -0500 "R. David Murray" wrote: > On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: > > The concurrent.futures tests should (like the multiprocessing test > > suite) detect the lack of support and skip the tests on the broken > > platforms. I'm sort of surprised FreeBSD support is still broken in > > this way though (echoed by Philip) although it could be an issue on > > that particular buildbot. > > If I'm reading the issue correctly, it isn't that it doesn't work, > it's that the concurrent.futures tests fail because they create more > semaphores than the default FreeBSD install supports. Doesn't it suggest a possible resource leak somewhere? Or do the concurrent tests really rely on creating many semaphores? (as opposed to say 5 or 10 of them) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile() and Windows/Mac newlines
On 12/29/2010 2:53 PM, Benjamin Peterson wrote:
Type "help", "copyright", "credits" or "license" for more information.
compile("print(999)\r\n", "blah", "exec")
at 0xb353e8, file "blah", line 1>
I made a mistake in testing. Issue closed. Sorry for the noise.
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 1:34 PM, "Martin v. Löwis" wrote: > Am 29.12.2010 18:54, schrieb Jesse Noller: >> On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. Löwis" >> wrote: > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. >>> >>> What *should* be possible is to disable building >>> SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, >>> multiprocessing locks would stop working on FreeBSD, and concurrent >>> futures; the tests would recognize this lack of features and get >>> skipped. >>> >>> Regards, >>> Martin >> >> The multiprocessing test suite already skips the tests which use the >> (broken) functionality on BSD correctly. This logic needs to be added >> to the concurrent.futures library. > > I'm not so sure that skipping the test is the right approach. Doesn't > that mean that the code will still fail at runtime with > difficult-to-explain messages? I'd rather prefer if the functionality > wasn't available in the first place. > > Also, what specific test are you referring to? > > Regards, > Martin > If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Le mercredi 29 décembre 2010 à 21:04 +0100, Antoine Pitrou a écrit : > Doesn't it suggest a possible resource leak somewhere? I already checked that: all locks are destroyed correctly on each test. - test_all_completed_some_already_completed() uses 51 SemLock objects - test_first_completed() uses 41 SemLock objects - test_processes_terminate() uses 41 SemLock objects - etc. FreeBSD has an arbitrary limit of 30 semaphores. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
>> If you can make the above change, the question then is what API >> multiprocessing semaphores should be built upon. It seems that you >> are saying that they should use SysV IPC, and only fall back to >> POSIX IPC if SysV IPC doesn't work/exist (are there any platforms >> where this would be the case?) > > Actually, I'd probably recommend the opposite. IMO POSIX IPC is > better designed and easier to work with so it would be my first > choice. "better designed and easier to work with" don't count if you have to do both, anyway. The code will be ugly - so what, once it is written? It's not being exposed to the user. > SysV would be a fallback for when POSIX IPC is unavailable, > broken, incomplete, etc. The problem seems to be that it is not possible to reliably determine these cases. In particular, the ridiculous low limits seem to be difficult to detect, and it's not clear what a required minimum level should be. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
>>> The multiprocessing test suite already skips the tests which use the >>> (broken) functionality on BSD correctly. This logic needs to be added >>> to the concurrent.futures library. >> Also, what specific test are you referring to? Can you kindly point me to the test that skips if broken functionality on BSD is detected? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> If the functionality is not supported then users get an import error > (within multiprocessing). However, RDM's understanding is correct, and > the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 3:49 PM, "Martin v. Löwis" wrote: >> If the functionality is not supported then users get an import error >> (within multiprocessing). However, RDM's understanding is correct, and >> the test is creating more than supported. > > Hmm. The tests do the absolute minimum stuff that exercises the code; > doing anything less, and they would be useless. Of course, one may > wonder why test_first_completed manages to create 41 SemLock objects, > when all it tries to do is two future calls. > > So if the minimal test case fails, I'd claim that the module doesn't > work on FreeBSD, period. ISTM that Posix IPC is just not a feasible > approach to do IPC synchronization on FreeBSD, so it's better to say > that multiprocessing is not supported on FreeBSD (until SysV IPC is > getting used, hoping that this will fare better). > Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 29.12.2010 22:34, schrieb Jesse Noller: > > > On Dec 29, 2010, at 3:49 PM, "Martin v. Löwis" wrote: > >>> If the functionality is not supported then users get an import error >>> (within multiprocessing). However, RDM's understanding is correct, and >>> the test is creating more than supported. >> >> Hmm. The tests do the absolute minimum stuff that exercises the code; >> doing anything less, and they would be useless. Of course, one may >> wonder why test_first_completed manages to create 41 SemLock objects, >> when all it tries to do is two future calls. >> >> So if the minimal test case fails, I'd claim that the module doesn't >> work on FreeBSD, period. ISTM that Posix IPC is just not a feasible >> approach to do IPC synchronization on FreeBSD, so it's better to say >> that multiprocessing is not supported on FreeBSD (until SysV IPC is >> getting used, hoping that this will fare better). >> > > Whatever you choose to say Martin. It does work, and is supported to the > limitations that FreeBSD imposes. So what do you propose to do? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 4:54 PM, "Martin v. Löwis" wrote: > Am 29.12.2010 22:34, schrieb Jesse Noller: >> >> >> On Dec 29, 2010, at 3:49 PM, "Martin v. Löwis" wrote: >> If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. >>> >>> Hmm. The tests do the absolute minimum stuff that exercises the code; >>> doing anything less, and they would be useless. Of course, one may >>> wonder why test_first_completed manages to create 41 SemLock objects, >>> when all it tries to do is two future calls. >>> >>> So if the minimal test case fails, I'd claim that the module doesn't >>> work on FreeBSD, period. ISTM that Posix IPC is just not a feasible >>> approach to do IPC synchronization on FreeBSD, so it's better to say >>> that multiprocessing is not supported on FreeBSD (until SysV IPC is >>> getting used, hoping that this will fare better). >>> >> >> Whatever you choose to say Martin. It does work, and is supported to the >> limitations that FreeBSD imposes. > > So what do you propose to do? > I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD. FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> I don't have a good suggestion (or a computer with a keyboard > anywhere near me) right now, but making a migration/fallback to SYSV > style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. > I would document the limitation in the futures/multiprocessing > documentation, and skip that particular test for now on FreeBSD. Just skipping the test case seems inappropriate, when it really tests the core functionality of the module in question. The failing test really means that the module does not work on the system. > FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This > isn't a bug in either futures or multiprocessing though. I don't claim that it's a bug. I claim that FreeBSD is just not supported (saying that FreeBSD doesn't support the multiprocessing module might be more appropriate). Somebody will have to correctly fix it - either the FreeBSD people, or the multiprocessing authors, or the concurrent.future authors. Meanwhile, we must admit that it just doesn't work (or else the test case would not fail). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 5:24 PM, Martin v. Löwis wrote: >> I don't have a good suggestion (or a computer with a keyboard >> anywhere near me) right now, but making a migration/fallback to SYSV >> style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. Note that we only know that the test fails on FreeBSD 7.2. The current release is FreeBSD 8.1. The 8.x series might work just fine. I don't know which test is failing under 7.2. Has anyone tested it under 7.3 or 8.x? bye Philip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, 29 Dec 2010 23:24:32 +0100 "Martin v. Löwis" wrote: > > I don't have a good suggestion (or a computer with a keyboard > > anywhere near me) right now, but making a migration/fallback to SYSV > > style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. So, IMO, skipping tests is enough. At worse, multiprocessing can issue a warning when imported under these OSes. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
"Martin v. Löwis" writes: >> I don't have a good suggestion (or a computer with a keyboard >> anywhere near me) right now, but making a migration/fallback to SYSV >> style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing. That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release. That assumes that normal use of the module will need as many semaphores as the tests, but if not, presumably the documentation comments could become more a caution than a requirement. I've also been considering retiring the 6.x buildbot in favor of an 8.x (I only have resources for 2, and even that is slow). Updating the 7.x buildbot would also include dynamic adjustment of the limit, which I think based on the prior discussion her was mentioned as being in 7.3. Though 7.4 releases late in January so could jump right to that. -- David PS: In regards to another comment in this thread, while 7.2 doesn't load the POSIX support by default, the buildbot does have the appropriate loader configuration to include it. PPS: I am about to be traveling on business through 1/10, so unlikely to make any buildbot changes before then. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou wrote: > On Wed, 29 Dec 2010 23:24:32 +0100 > "Martin v. Löwis" wrote: >> > I don't have a good suggestion (or a computer with a keyboard >> > anywhere near me) right now, but making a migration/fallback to SYSV >> > style semaphores a release blocker seems like a mistake to me. >> >> And indeed, I don't propose to make that a release blocker. Instead, >> I propose to disable support for the module (either multiprocessing >> or concurrent.futures only) on FreeBSD, and make such disabling a >> release blocker. > > I don't really agree with this. There's no need to explicitly forbid > use of multiprocessing from FreeBSD. First, it is not our task to > validate that each and every OS conforms to the APIs it claims to > implement. Second, such disabling would make life uselessly more > complicated for users the day FreeBSD actually fixes their stuff. Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with the arbitrary limits. So +1 for skipping the affected tests on FreeBSD so the buildbot state can tell us something useful (conditionally wrapping them with "expected failure" may be even better, as then we'll find out when the FreeBSD out-of-the-box configuration actually supports running them). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Le mercredi 29 décembre 2010 à 21:49 +0100, "Martin v. Löwis" a écrit : > Of course, one may wonder why test_first_completed manages > to create 41 SemLock objects, when all it tries to do is two future > calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On 12/29/2010 5:44 PM, David Bolen wrote: Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing. That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release. Testing on updated buildbots and documenting older BSD limits and fixes seems like the best solution. Martin is right that we really do want the tests to run to catch any real bugs (on our side). -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 5:46 PM, Nick Coghlan wrote: > On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou wrote: >> On Wed, 29 Dec 2010 23:24:32 +0100 >> "Martin v. Löwis" wrote: I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. >>> >>> And indeed, I don't propose to make that a release blocker. Instead, >>> I propose to disable support for the module (either multiprocessing >>> or concurrent.futures only) on FreeBSD, and make such disabling a >>> release blocker. >> >> I don't really agree with this. There's no need to explicitly forbid >> use of multiprocessing from FreeBSD. First, it is not our task to >> validate that each and every OS conforms to the APIs it claims to >> implement. Second, such disabling would make life uselessly more >> complicated for users the day FreeBSD actually fixes their stuff. > > Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with > the arbitrary limits. So +1 for skipping the affected tests on FreeBSD > so the buildbot state can tell us something useful (conditionally > wrapping them with "expected failure" may be even better, as then > we'll find out when the FreeBSD out-of-the-box configuration actually > supports running them). A bit of info about FreeBSD 7.2 and 8.0 (the only versions I have installed). My installs are set up explicitly for testing and so they have had very little customization. - `sysctl -a|grep p1003` displays the config values specific to POSIX IPC. I think `sysctl -a|grep ipc` shows values for SysV IPC. It looks like the POSIX 1003_1b names in sysctl are similar to the names here (e.g. SEM_NSEMS_MAX, SEM_VALUE_MAX, etc.) -- http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html - Under my 7.2, sysctl.p1003_1b.sem_nsems_max == 0 and I can't create semaphores with my posix_ipc extension. After `kldload sem`, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works even without `kldload sem`. `kldstat` doesn't show sem.ko as loaded, so I assume that POSIX semaphores are now a default part of the kernel. - Under my 7.2 *and* 8.0, sysctl.p1003_1b.mq_open_max == 0 and I can't create a message queue with my posix_ipc extension. After `kldload mqueuefs`, sysctl.p1003_1b.mq_open_max remains 0 (huh?) but my posix_ipc message queue demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max is read only even if I change it in /etc/sysctl.conf. According to this post, sem_nsems_max doesn't become dynamic until 8.1: http://www.gossamer-threads.com/lists/python/dev/875319?do=post_view_threaded Hope this helps a bit Philip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: Le mercredi 29 décembre 2010 à 21:49 +0100, "Martin v. Löwis" a écrit : Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Great detective work! This would suggest that ProcessPoolExecutors are useable on FreeBSD 7.2 so long as the user doesn't create more than two at once (which probably isn't a big deal for most apps). So skipping the test is probably the way to go. Cheers, Brian Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 12:49 PM, Martin v. Löwis wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. I actually think that my tests may be overdone - in order to probe for specific race conditions they use a lot of locks to force calls to complete in a specific order. I'm thinking about pairing the tests down to only demonstrate basic correctness. This should fix the tests on FreeBSD and Windows. Then, when Python 3.2 is released, I can gradually introduce more comprehensive tests while ensuring that I keep the buildbots green on all supported platforms. Thoughts? Cheers, Brian So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
R. David Murray writes: > We merge bug fixes to 2.7 on a routine basis. It is the rule rather > than the exception, by a long shot. For bugfixes, of course it's routine. I understand that. My point was that the case Amaury fears, where the (new) VCS makes things harder than patching or porting by hand, is likely to be relatively uncommon, sandwiched between the "typo fixed in comment" conflicts (aka "trivial tweaks") and those that require reengineering. Also, while workflow helpers will make a big difference to the non-VCS-geeks (ie, almost all Python developers), properly speaking this isn't really an issue with Mercurial, because all of the methods for this purpose are basically "diff | patch", although the executables are called things like "svn" and "python". They all demand workflow helper scripts to regulate the flow of desired and undesired patches. The difference is that the tool for hg is a SMOP, while that for svn is a SMOEP[1]. > So, since we are going to be maintaining 2.7 for a while, this is > a workflow problem that we *must* solve to make the hg transition > worthwhile. I have no doubt that we will, but I also have no doubt we > need to *solve* it, not just wave our hands. Certainly. I think I already said that, no? My point is simply that while Amaury's expression of his requirements is welcome, and his experimenting with hg is extremely valuable, indeed a necessary part of the transition, everything he describes so far is a known problem that we basically know how to solve. He talks about changes to the workflow, but frankly, I don't see a *need* for that.[2] IMO, changes to the workflow will be driven by kaizen, not by some brave new world revolution (Guido inter alia insisted on that) nor by thumb-in-dike disaster recovery (PEP 374 did a pretty good job on that, if I do say so myself). I wish I had more time to do real work on this (not to mention email, thank *you*, David!), but it seems like every time I start programming, I fall asleep ... and six hours later, it's back to day job or family services. :-/ Footnotes: [1] Simpler Matter Of Existing Program. [2] Aside for a need for establishing which hg commands correspond to which parts of the existing workflow, and perhaps creating helper scripts. Ie, I think the chances are pretty good that most people who have already tried hg are at least a little VCS-geeky, and probably they adjust personal workflow to the new VCS. That will not be so transparent to the "the tool should work for me, not vice-versa" majority. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Thu, 30 Dec 2010 14:42:48 +0900, "Stephen J. Turnbull" wrote: > R. David Murray writes: > > > We merge bug fixes to 2.7 on a routine basis. It is the rule rather > > than the exception, by a long shot. > > For bugfixes, of course it's routine. I understand that. My point > was that the case Amaury fears, where the (new) VCS makes things > harder than patching or porting by hand, is likely to be relatively > uncommon, sandwiched between the "typo fixed in comment" conflicts > (aka "trivial tweaks") and those that require reengineering. I thought Amaury was saying it was harder than svnmerge, not harder than patching by hand (ie: that it *was* patching by hand, including .rej files and the lack of tracking). I also heard Georg say that this should be fixable, and that he's partly fixed the tracking issue, but not the patch/merge issue (and that doing so will require a change in the hg core). > Also, while workflow helpers will make a big difference to the > non-VCS-geeks (ie, almost all Python developers), properly speaking > this isn't really an issue with Mercurial, because all of the methods > for this purpose are basically "diff | patch", although the > executables are called things like "svn" and "python". They all > demand workflow helper scripts to regulate the flow of desired and > undesired patches. The difference is that the tool for hg is a SMOP, > while that for svn is a SMOEP[1]. Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :) > > So, since we are going to be maintaining 2.7 for a while, this is > > a workflow problem that we *must* solve to make the hg transition > > worthwhile. I have no doubt that we will, but I also have no doubt we > > need to *solve* it, not just wave our hands. > > Certainly. I think I already said that, no? My point is simply that Ah, I guess I did miss that, sorry. > while Amaury's expression of his requirements is welcome, and his > experimenting with hg is extremely valuable, indeed a necessary part > of the transition, everything he describes so far is a known problem > that we basically know how to solve. He talks about changes to the > workflow, but frankly, I don't see a *need* for that.[2] Well, there will be *some* workflow change since we're talking about committing to 3.2 maint before the new trunk instead of vice versa. But you are right that that is, mostly, a detail. I'm not as convinced that changes in workflow will be that minimal, though. I don't have much experience with the dvcses yet, but what experience I have had leads me to think that understanding the DAG is a non-trivial and necessary mental leap and does affect the workflow. I don't feel as though I've made that leap yet. Hopefully Brett will be able to document this in the Python context so that the *required* leap will be much smaller. > IMO, changes to the workflow will be driven by kaizen, not by some > brave new world revolution (Guido inter alia insisted on that) nor by > thumb-in-dike disaster recovery (PEP 374 did a pretty good job on > that, if I do say so myself). Well, I hope you are right. I'm looking forward to the new version of the test repository and doing some real world experiments. > I wish I had more time to do real work on this (not to mention email, > thank *you*, David!), but it seems like every time I start You are welcome; thanks for the feedback. (I sometimes feel like I'm working in a bit of a vacuum, though Barry does chime in occasionally...but I do realize that people are busy; that's why I inherited this job in the first place, after all :) -- R. David Murray www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
