Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 2:35 AM, Nick Coghlan ncogh...@gmail.com wrote: On 23 November 2014 at 17:14, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 2:01 AM, Nick Coghlan ncogh...@gmail.com wrote: Travis isn't the only CI system on the internet, and for pure Sphinx documentation cases, ReadTheDocs runs just as well off BitBucket as it does off GitHub. Sure it’s not the only CI system, but as far as I know bitbucket doesn’t have near the integration possible with CI systems. I make a PR on github I get it tested and the merge button turns green to let me know that the tests run. Travis is popular enough that I’ve seen bitbucket projects hosting a mirror on github *just* for travis. In the absence of a proposal to change version control systems (again), the lack of Mercurial hosting on GitHub makes it rather a moot point. Given that we can barely muster up any enthusiasm for rehosting *without* changing version control systems (and the number of CI systems that integrate with hg.python.org repos other than the main CPython one is exactly zero), any proposal that involves doing even *more* work seems doubly doomed. I’d volunteer to do the work to get the PEPs, and possibly other repositories onto Github if we so decided to do so. Don’t let the lack of volunteer stop that because I will find the time to do it if need be. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 5:23 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam ron3...@gmail.com wrote: Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? The 2.x correspondence between list comp and for loops was *already broken* in 3.0 by moving execution to a new function to prevent name binding in comprehension from leaking the way they did in 2.x. We did not change for loops to match. The following, which is an example of equivalence in 2.7, raises in 3.4 because 'itertools' is not defined. def binder(ns, name, ob): ns[name] = ob for mod in ['sys']: binder(locals(), mod, __import__(mod)) print(sys) [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] print(itertools) Ceasing to leak *any* local bindings in 3.0 broke equivalence and code depending on such bindings. Ceasing to leak StopIteration would break equivalence a bit more, but only a bit, and code dependent on such leakage, which I expect is extremely rare. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 23 Nov 2014 18:11, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 2:35 AM, Nick Coghlan ncogh...@gmail.com wrote: In the absence of a proposal to change version control systems (again), the lack of Mercurial hosting on GitHub makes it rather a moot point. Given that we can barely muster up any enthusiasm for rehosting *without* changing version control systems (and the number of CI systems that integrate with hg.python.org repos other than the main CPython one is exactly zero), any proposal that involves doing even *more* work seems doubly doomed. I’d volunteer to do the work to get the PEPs, and possibly other repositories onto Github if we so decided to do so. Don’t let the lack of volunteer stop that because I will find the time to do it if need be. It's the other way around: someone would have to volunteer to make the case that switching version control systems will actually help in any way whatsoever with the core reviewer bandwidth problem. We do *not* have a shortage of people wanting to contribute. While ongoing outreach efforts are essential to promote increased diversity in the contributor base and to counter natural attrition, there is currently no major problem that needs solving on that front. CPython is high profile enough that folks are willing to do battle with the current complicated contribution process, so we're already one of the most active open source projects in the world, in *spite* of the problems with the existing workflow. This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work). Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward). Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online. Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day job, and you almost always want to go for the lowest impact, lowest risk approach that will alleviate the bottleneck you're worried about while changing the smallest possible number of elements in the overall workflow management system. That underlying calculation doesn't really change much even when the units shift from budget dollars to volunteers' time and energy. Regards, Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
Nick Coghlan writes: By contrast, proposals to switch from Mercurial to Git impose a *massive* burden on contributors that don't already know git. Let's not get carried away here. The *massive* burden is the moaning from git-haters (is there a 12-step program for them?) Agreed, learning any new VCS is a burden, and git seems harder than most -- but even RMS is swallowing git for Emacs, despite the fact that the g in git doesn't stand for GNU. with your GitHub ID, and the functional isomorphism [git - hg] I agree that there is an isomorphism, but the philosophical restrictions on hg functionality are quite annoying. I do things like git reset and git commit --amend a lot. I tend to commit before getting coffee, but I don't want that in the permanent record -- git rebase --interactive is a good buddy of mine. And so on. hg *deliberately* gets in the way of such workflows (although perhaps it's not as hard to opt in to the necessary features as it used to be). Nevertheless, I tend to agree with you that moving to Github now is a big move. I just think you should avoid the git dox suck argument. I'd also like to mention that in my opinion the network externalities argument is being misused. True, everybody has a github account, and even if they don't, their little sisters do. So what? There are big network externalities involved, but that doesn't necessarily mean that Bitbucket can't catch up, and most projects I know have branches hosted on both Bitbucket and Github (and often SourceForge or Launchpad as well). People who really prefer one or the other for practical reasons can usually use them without too much difficulty, wherever the official repo may be hosted. More likely to have a clear outcome, the main network externality *we* should be concerned with is *within* the Python ecosystem. *If* the big projects whose core members tend not to hang out here so much (NumPy, SciPy, Twisted, Django, ...) are vastly more likely to be found on Bitbucket than Github (or vice versa), I think that's potentially much more important than the little sisters with github accounts. I also agree with you that the facts that Mercurial is a Python application, and I guess so is most of Github, are important. But again, let's not get carried away. Although practicality beats purity applies here. The Github features are very attractive; we need to look at how useful they will be to contributors before deciding that the warm fuzzy Python community is more important. Finally, Guido is right: Github, and to a somewhat lesser extent Bitbucket and Google Code have gotten code hosting right, compared to the *forges. The people who maintain infrastructure for Python are the kind of contributor who would probably spend more time on reviewing and mentoring and release engineering if they weren't maintaining infrastructure as far as I can see. If the infrastructure maintenance can be delegated (it's not clear to me that it can), that would be a big factor. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, 23 Nov 2014 22:58:02 +0900 Stephen J. Turnbull step...@xemacs.org wrote: Nick Coghlan writes: By contrast, proposals to switch from Mercurial to Git impose a *massive* burden on contributors that don't already know git. Let's not get carried away here. The *massive* burden is the moaning from git-haters (is there a 12-step program for them?) Is there a 12-step program for people who can't help commenting on the workflow of core developers and contributors while they don't seem to ever contribute anything concrete? Seriously, stick to the topics where you have some semblant of legitimacy. Whether we choose to use git or hg you have absolutely no authority to comment about. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014, at 01:25, Nick Coghlan wrote: On 23 November 2014 at 16:03, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 12:59 AM, Nick Coghlan ncogh...@gmail.com wrote: Note that if folks prefer Git, BitBucket supports both. I would object strongly to unilaterally forcing existing contributors to switch from Mercurial to git. Going to all the trouble to move to an external repository and choosing the least popular option out of the two main ones seems like a bad idea in general. Moving repos from hg.python.org to bitbucket.org is just a matter of switching some URLs around, and changing some backend systems to cope with the new location. The end result should be to make life better for existing contributors *and* new contributors using the web UI, and be largely transparent to folks using command line tools. By contrast, proposals to switch from Mercurial to Git impose a *massive* burden on contributors that don't already know git. But how many people are there who will have this massive burden imposed on them? I imagine there's few among us who haven't had to learn git for our job or other projects. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 08:53 PM, Guido van Rossum wrote: In order to save everyone's breath, I am *accepting* the proposal of PEP 479. Excellent. Chris, thank you for your time, effort, and thoroughness in championing this PEP. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 2:11 AM, Ethan Furman et...@stoneleaf.us wrote: On 11/22/2014 08:53 PM, Guido van Rossum wrote: In order to save everyone's breath, I am *accepting* the proposal of PEP 479. Excellent. Chris, thank you for your time, effort, and thoroughness in championing this PEP. Thank you, it's nice to have a successful one to counterbalance the failure of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I'm actually quite impressed with how on-topic the discussion threads went. They didn't ramble nearly as much as I thought they would. Thank you to all who participated, contributed suggestions, complained about the existing text, and generally worked hard to make sure I had more than enough material to draw on! Particular thanks to Guido, pushing changes through and being patient with my mistakes in markup, and adding content directly to the document. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 23 November 2014 at 15:25, Chris Angelico ros...@gmail.com wrote: Thank you, it's nice to have a successful one to counterbalance the failure of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I think it's worth pointing out that both this and PEP 463 were complex an contentious topics, and the discussion in both cases was well-focused and civil. I don't think the fact that you were the author of both PEPs is unrelated to this. Thanks for taking on *both* of these PEPs and handling them so well. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan ncogh...@gmail.com wrote: More generally, I'm very, very disappointed to see folks so willing to abandon fellow community members for the sake of following the crowd. Perhaps we should all just abandon Python and learn Ruby or JavaScript because they're better at getting press in Silicon Valley? That's a really low blow, Nick. I think these are the facts: - Hg/Git are equivalent in functionality (at least to the extent that the difference can't be used to force a decision), and ditto for BitBucket/GitHub, with one crucial exception (see below) - We're currently using Hg for most projects under the PSF umbrella (however, there's https://github.com/python/pythondotorg) - Moving from Hg to Git is a fair amount of one-time work (converting repos) and is inconvenient to core devs who aren't already used to Git (learning a new workflow) - Most newer third-party projects are already on GitHub - GitHub is way more popular than BitBucket and slated for long-term success But here's the kicker for me: **A DVCS repo is a social network, so it matters in a functional way what everyone else is using.** So I give you that if you want a quick move into the modern world, while keeping the older generation of core devs happy (not counting myself :-), BitBucket has the lowest cost of entry. But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. Note: I am not (yet) proposing we switch CPython itself. Switching it would be a lot of work, and it is specifically out of scope for this discussion. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan ncogh...@gmail.com wrote: On 23 Nov 2014 18:11, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 2:35 AM, Nick Coghlan ncogh...@gmail.com wrote: In the absence of a proposal to change version control systems (again), the lack of Mercurial hosting on GitHub makes it rather a moot point. Given that we can barely muster up any enthusiasm for rehosting *without* changing version control systems (and the number of CI systems that integrate with hg.python.org repos other than the main CPython one is exactly zero), any proposal that involves doing even *more* work seems doubly doomed. I’d volunteer to do the work to get the PEPs, and possibly other repositories onto Github if we so decided to do so. Don’t let the lack of volunteer stop that because I will find the time to do it if need be. It's the other way around: someone would have to volunteer to make the case that switching version control systems will actually help in any way whatsoever with the core reviewer bandwidth problem. We do *not* have a shortage of people wanting to contribute. While ongoing outreach efforts are essential to promote increased diversity in the contributor base and to counter natural attrition, there is currently no major problem that needs solving on that front. CPython is high profile enough that folks are willing to do battle with the current complicated contribution process, so we're already one of the most active open source projects in the world, in *spite* of the problems with the existing workflow. The *immediate* problem is making it easier to accept contributions from people. The long-term, never-ending problem is making the whole process of submitting a patch and getting it accepted as easy as possible for everyone involved, contributor and committer alike. If the goal is to make it so we can accept PRs for easier patch acceptances then that can be accomplished on either Bitbucket or GitHub. But if we want to make it easier for people to make contributions then GitHub is arguably better than Bitbucket, whether it's through familiarity of GitHub for most people thanks to other FLOSS projects or from the superior tooling around GitHub (both the platform itself and the ecosystem that has sprung up around it). This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work). Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward). Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online. Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 11:55 AM, Brett Cannon br...@python.org wrote: This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work). Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward). Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online. Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. Commit hooks don’t work on Github or bitbucket anyways. So either way they’d need rewritten to support whichever platform’s web hooks instead. From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. Right, I would argue that Github is at least a little bit better because of the commit status API. It’s *really* nice to see right on a PR if it breaks anything or not before you merge it without ever having to run a thing manually. That isn’t something that only Travis can do so if we want some other system to do it that’s totally possible. It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be. Off the top of my head for the docs I know of https://github.com/python/docsbuild-scripts/blob/master/build_docs.py https://github.com/python/docsbuild-scripts/blob/master/build_docs.py which will need to be updated to git clone
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/22/2014 11:13 PM, Donald Stufft wrote: On Nov 23, 2014, at 1:49 AM, Nick Coghlan wrote: I took the git knowledge I acquired by necessity at Red Hat and figured out how to apply it to hg. All the same features are there in hg, they're just switched off by default (mainly because the core Mercurial devs are adamant that any potentially history destroying operation in a version control system must be opt-in). If you could find the time to write up something about that I'm sure it would be helpful. :) We already have lots of potential contributors (if we didn't, review bandwidth wouldn't be the bottleneck the way it is today), and the functional differences between GitHub and BitBucket from a barrier to entry perspective are so low as to be trivial. That’s not really true. It’s more than just “can I log in”, potential contributors are more likely to already know how to use Github too and are more likely to not want to deal with another site. I know personally if I see a project is on bitbucket my chances of contributing to that project drop drastically, even if they are using git on bitbucket, just because I know that I’m going to get frustrated to some degree. I feel the same way, only in reverse. I've learned hg, and to a lesser extent bitbucket, but have not learned git nor github, and would rather not (available bandwidth and all that). Moving from self-hosted Mercurial repos to externally hosted Mercurial repos is a low risk change. It reduces maintenance overhead and lowers barriers to external contribution, both without alienating existing contributors by forcing them to change their workflows. Proposing to *also* switch from Mercurial to git significantly increases the cost of the change, while providing minimal incremental benefit. Whatever our personal feelings of hg vs git, and bitbucket vs github, that makes sense. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 12:30 PM, Raymond Hettinger wrote: Pre-PEP 479: --- def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value Post-PEP 479: def middleware_generator(source_generator): it = source_generator() try: input_value = next(it) except StopIteration: return # This causes StopIteration to be reraised output_value = do_something_interesting(input_value) yield output_value While I am in favor of PEP 479, and I have to agree with Raymond that this isn't pretty. Currently, next() accepts an argument of what to return if the iterator is empty. Can we enhance that in some way so that the overall previous behavior could be retained? Something like: def middleware_generator(source_generator): it = source_generator() input_value = next(it, gen_exit=True) # or exc_type=GeneratorExit ? output_value = do_something_interesting(input_value) yield output_value Then, if the iterator is empty, instead of raising StopIteration, or returning some value that would then have to be checked, it could raise some other exception that is understood to be normal generator termination. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 08:55 AM, Brett Cannon wrote: Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy. Minimizing downtime and impact is important for us, too. The Python job board has now been down for nine months -- that's hardly good PR. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
I can agree with most of these points. Some more things to consider: - Git is 20x faster than Hg (that's 99% of the reason I switched and hate using Darcs) - People attached to Hg can use Hg-Git; I've used it several times with nice results. It can also be used to easily convert Hg repos to Git (I was using it to maintain a Pygments mirror, but got bored) - GitHub is much nicer than BitBucket. I usually look for a git mirror of a BitBucket repo since I hate browsing on BB - Even Plan9Port moved from Hg to GitHub a few days ago - Some Google devs even host their projects on GitHub (not Google Code) Thank goodness no one has mentioned CodePlex. BTW, I frown on projects that choose something largely inferior over a better choice just because one is written in X language. Now, if python.org was written in Ruby, that would just be sad, but certain things just don't matter. Like DVCS's. Guido van Rossum gu...@python.org wrote: On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan ncogh...@gmail.com wrote: More generally, I'm very, very disappointed to see folks so willing to abandon fellow community members for the sake of following the crowd. Perhaps we should all just abandon Python and learn Ruby or JavaScript because they're better at getting press in Silicon Valley? That's a really low blow, Nick. I think these are the facts: - Hg/Git are equivalent in functionality (at least to the extent that the difference can't be used to force a decision), and ditto for BitBucket/GitHub, with one crucial exception (see below) - We're currently using Hg for most projects under the PSF umbrella (however, there's https://github.com/python/pythondotorg) - Moving from Hg to Git is a fair amount of one-time work (converting repos) and is inconvenient to core devs who aren't already used to Git (learning a new workflow) - Most newer third-party projects are already on GitHub - GitHub is way more popular than BitBucket and slated for long-term success But here's the kicker for me: **A DVCS repo is a social network, so it matters in a functional way what everyone else is using.** So I give you that if you want a quick move into the modern world, while keeping the older generation of core devs happy (not counting myself :-), BitBucket has the lowest cost of entry. But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. Note: I am not (yet) proposing we switch CPython itself. Switching it would be a lot of work, and it is specifically out of scope for this discussion. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 08:55 AM, Brett Cannon wrote: Fourth, do any core developers feel strongly about not using GitHub? Dous GitHub support hg? If not, I am strongly opposed. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 1:06:18 PM Ethan Furman et...@stoneleaf.us wrote: On 11/23/2014 08:55 AM, Brett Cannon wrote: Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy. Minimizing downtime and impact is important for us, too. The Python job board has now been down for nine months -- that's hardly good PR. That has nothing to do with downtime and everything to do with volunteer time. My point about downtime is that if I can't commit to the cpython repo for a day it isn't going to cause me to freak out or cost anyone thousands of dollars or more in revenue. -Brett -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman et...@stoneleaf.us wrote: On 11/23/2014 08:55 AM, Brett Cannon wrote: Fourth, do any core developers feel strongly about not using GitHub? Dous GitHub support hg? If not, I am strongly opposed. Depends on what you mean by support. If you mean natively, then no. If you mean I want more of a hg CLI then you can get that with http://hg-git.github.io/ . And can I just say this is all bringing back wonderful flashbacks of the SourceForge to our own infrastructure move as well as the svn to hg move. =/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/23/2014 04:08 AM, Terry Reedy wrote: On 11/22/2014 5:23 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam ron3...@gmail.com wrote: Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? The 2.x correspondence between list comp and for loops was *already broken* in 3.0 by moving execution to a new function to prevent name binding in comprehension from leaking the way they did in 2.x. We did not change for loops to match. The following, which is an example of equivalence in 2.7, raises in 3.4 because 'itertools' is not defined. def binder(ns, name, ob): ns[name] = ob for mod in ['sys']: binder(locals(), mod, __import__(mod)) print(sys) [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] print(itertools) Ceasing to leak *any* local bindings in 3.0 broke equivalence and code depending on such bindings. Ceasing to leak StopIteration would break equivalence a bit more, but only a bit, and code dependent on such leakage, which I expect is extremely rare. I think they would be rare too. With the passage of the PEP, it will change what is different about them once it's in full effect. The stop hack won't work in both, and you may get a RuntimeError in generator expressions where you would get StopIteration in list-comps. (Is this correct?) Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 11:56:49 AM Guido van Rossum gu...@python.org wrote: On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan ncogh...@gmail.com wrote: More generally, I'm very, very disappointed to see folks so willing to abandon fellow community members for the sake of following the crowd. Perhaps we should all just abandon Python and learn Ruby or JavaScript because they're better at getting press in Silicon Valley? That's a really low blow, Nick. I think these are the facts: - Hg/Git are equivalent in functionality (at least to the extent that the difference can't be used to force a decision), and ditto for BitBucket/GitHub, with one crucial exception (see below) - We're currently using Hg for most projects under the PSF umbrella (however, there's https://github.com/python/pythondotorg) - Moving from Hg to Git is a fair amount of one-time work (converting repos) and is inconvenient to core devs who aren't already used to Git (learning a new workflow) - Most newer third-party projects are already on GitHub - GitHub is way more popular than BitBucket and slated for long-term success But here's the kicker for me: **A DVCS repo is a social network, so it matters in a functional way what everyone else is using.** So I give you that if you want a quick move into the modern world, while keeping the older generation of core devs happy (not counting myself :-), BitBucket has the lowest cost of entry. But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. Note: I am not (yet) proposing we switch CPython itself. Switching it would be a lot of work, and it is specifically out of scope for this discussion. If we want to test the complexity of moving something to GitHub then probably the best repo to use is the peps one: - Very few people directly use that repo (you and me alone could probably manage it if we enforced all changes through a PR as I could then do approvals from work instead of having to wait until I was at home with an hg checkout available) - It's used on the website so it would require updating infrastructure - It isn't a lot of overhead to tell people who email the peps mailing list to please send a pull request through GitHub since it isn't tracked in the issue tracker anyway - There is a benefit of setting up some CI integration to know when a PR is actually incorrectly formatted And if people want to test the impact of Bitbucket we could do it for something like the HOWTOs as that too involves infrastructure but is not used by a lot of people. In fact we can make it known we are piloting this approach on Bitbucket and see what kind of contributions it triggers (ditto for the peps since I'm sure some people will want to send in typo PRs and such). IOW I don't see why we can't pilot this between now and April for the language summit and see what difference it all makes so we can have an informed discussion in Montreal with more than 4 full months of experience under our belts. Then we can discuss Bitbucket vs. GitHub, docs vs. everything moving vs. nothing, etc. That was this stops all being conjecture and more about seeing if there is an actual impact. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 1:31:36 PM Brett Cannon br...@python.org wrote: On Sun Nov 23 2014 at 11:56:49 AM Guido van Rossum gu...@python.org wrote: On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan ncogh...@gmail.com wrote: More generally, I'm very, very disappointed to see folks so willing to abandon fellow community members for the sake of following the crowd. Perhaps we should all just abandon Python and learn Ruby or JavaScript because they're better at getting press in Silicon Valley? That's a really low blow, Nick. I think these are the facts: - Hg/Git are equivalent in functionality (at least to the extent that the difference can't be used to force a decision), and ditto for BitBucket/GitHub, with one crucial exception (see below) - We're currently using Hg for most projects under the PSF umbrella (however, there's https://github.com/python/pythondotorg) - Moving from Hg to Git is a fair amount of one-time work (converting repos) and is inconvenient to core devs who aren't already used to Git (learning a new workflow) - Most newer third-party projects are already on GitHub - GitHub is way more popular than BitBucket and slated for long-term success But here's the kicker for me: **A DVCS repo is a social network, so it matters in a functional way what everyone else is using.** So I give you that if you want a quick move into the modern world, while keeping the older generation of core devs happy (not counting myself :-), BitBucket has the lowest cost of entry. But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. Note: I am not (yet) proposing we switch CPython itself. Switching it would be a lot of work, and it is specifically out of scope for this discussion. If we want to test the complexity of moving something to GitHub then probably the best repo to use is the peps one: - Very few people directly use that repo (you and me alone could probably manage it if we enforced all changes through a PR as I could then do approvals from work instead of having to wait until I was at home with an hg checkout available) - It's used on the website so it would require updating infrastructure - It isn't a lot of overhead to tell people who email the peps mailing list to please send a pull request through GitHub since it isn't tracked in the issue tracker anyway - There is a benefit of setting up some CI integration to know when a PR is actually incorrectly formatted And if people want to test the impact of Bitbucket we could do it for something like the HOWTOs as that too involves infrastructure but is not used by a lot of people. In fact we can make it known we are piloting this approach on Bitbucket and see what kind of contributions it triggers (ditto for the peps since I'm sure some people will want to send in typo PRs and such). Actually the tutorial might be best to measure ease of contribution for people on Bitbucket since we can also ask people who use the tutorial to test out pointing people to the Bitbucket repo if people want to send a PR. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 10:14 AM, Brett Cannon wrote: On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman et...@stoneleaf.us wrote: Dous GitHub support hg? If not, I am strongly opposed. Depends on what you mean by support. If you mean natively, then no. If you mean I want more of a hg CLI then you can get that with http://hg-git.github.io/ . Well, if somebody documents it, I suppose I can follow along. ;) And can I just say this is all bringing back wonderful flashbacks of the SourceForge to our own infrastructure move as well as the svn to hg move. =/ My apologies. Change can be hard. My concern is that we will end up with multiple different workflows depending on which part of Python we're working on, which will lead to more time spent learning more about how to do it instead of doing it, and more time spent recovering from errors because of the differences. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 10:31 AM, Brett Cannon wrote: If we want to test the complexity of moving something to GitHub then probably the best repo to use is the peps one: And if people want to test the impact of Bitbucket we could do it for something like the HOWTOs as that too involves infrastructure but is not used by a lot of people. +1 -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 1:41 PM, Ethan Furman et...@stoneleaf.us wrote: On 11/23/2014 10:14 AM, Brett Cannon wrote: On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman et...@stoneleaf.us wrote: Dous GitHub support hg? If not, I am strongly opposed. Depends on what you mean by support. If you mean natively, then no. If you mean I want more of a hg CLI then you can get that with http://hg-git.github.io/ . Well, if somebody documents it, I suppose I can follow along. ;). I haven't used it in a while, but it was something like this (after it's installed): - Put the git repo url in .hgrc: [paths] default=git+https://github.com/my_user/my_repo.git [auth] github.prefix = git+https://github.com/my_user/my_repo.git github.username = my_user github.password = my_password Unless you put the login info there, it won't work (for some reason). Then, whenever you're going to push, first do: hg bookmark -r default master I'd love to test this right now, but I'm on Chrome OS... And can I just say this is all bringing back wonderful flashbacks of the SourceForge to our own infrastructure move as well as the svn to hg move. =/ My apologies. Change can be hard. My concern is that we will end up with multiple different workflows depending on which part of Python we're working on, which will lead to more time spent learning more about how to do it instead of doing it, and more time spent recovering from errors because of the differences. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated. Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 05:55 PM, Brett Cannon wrote: I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. Well, since most people already know git this part is probably not a big deal, although we'd have to consider alienating core developers who are not git-savvy. Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. There are two categories of hooks we use: hooks that run before a push is accepted, and hooks that run afterwards. The latter are not a concern, since anything that GH/BB doesn't support can be run as a web service on python.org infrastructure, which then gets POST requests from the hosting platforms. These are * tracker update * IRC notification * sending email to python-checkins * triggering buildbot The more problematic category are pre-push hooks. We use them for checking and rejecting commits with * disallowed branches * non-conformant whitespace * wrong EOL style * multiple heads per named branch As far as I know, neither GH nor BB support such hooks, since they are highly project-specific. However, they are only used in cpython and related repositories, so that doesn't concern migration of doc-only repos. Sure, you can let the CI run the checks, but that doesn't prohibit merging and is circumvented by direct pushes to the repository that don't go through the PR system. (And please don't make me as a coredev open a PR for every change.) From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. In my opinion, scattering repos over github, bitbucket and hg.python.org is even less friendly to contributors than a centralized place. (We are already approaching this, with pydotorg and infrastructure repos on github.) So I'm going to add some thoughts here about migrating the main CPython to git+hub. We have to consider how well our branch workflow works with the PR workflow. There's no gain in the ability to easily merge a PR to one branch via github when the subsequent merge of 3.x to default/master requires a local pull/push cycle, as well as the 2.x backport. As far as I know, you'd have to open a pull/merge request yourself and instantly merge it, except if there are conflicts between branches, in which case you are again forced to go local. I don't need to mention that this doesn't work well when someone makes a concurrent commit to the source branch in the meantime. And I don't think you'd want to force people to submit 3 pull requests for the individual branches. The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be. Compiling that list is likely the most difficult step of
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 07:03 PM, Ryan wrote: I can agree with most of these points. Some more things to consider: - Git is 20x faster than Hg (that's 99% of the reason I switched and hate using Darcs) You won't get much traction with this argument around here. As long as there aren't specific complaints about slowness (which probably could be fixed by a discussion on mercurial-devel), Hg is fast enough, just like Python is fast enough compared to C. (Otherwise, why bother.) - GitHub is much nicer than BitBucket. I usually look for a git mirror of a BitBucket repo since I hate browsing on BB I agree, Github's UI is nicer than BB. It is also true that most features that are introduced on BB are inspired by GH. - Even Plan9Port moved from Hg to GitHub a few days ago Projects move all the time. Why is this one special? - Some Google devs even host their projects on GitHub (not Google Code) Thank goodness no one has mentioned CodePlex. BTW, I frown on projects that choose something largely inferior over a better choice just because one is written in X language. Now, if python.org http://python.org was written in Ruby, that would just be sad, but certain things just don't matter. Like DVCS's. You can read up PEP 374 for the reasons for Hg. The main reasons against git were lacking Windows support (which probably is good enough now) and developer preference. Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Please reconsider PEP 479.
Hi, I have serious concerns about this PEP, and would ask you to reconsider it. [ Very short summary: Generators are not the problem. It is the naive use of next() in an iterator that is the problem. (Note that all the examples involve calls to next()). Change next() rather than fiddling with generators. ] I have five main concerns with PEP 479. 1. Is the problem, as stated by the PEP, really the problem at all? 2. The proposed solution does not address the underlying problem. 3. It breaks a fundamental aspect of generators, that they are iterators. 4. This will be a hindrance to porting code from Python 2 to Python 3. 5. The behaviour of next() is not considered, even though it is the real cause of the problem (if there is a problem). 1. The PEP states that The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. I don't believe that to be the case; if someone knows what StopIteration is and how it is used, then the interaction is entirely as expected. I believe the naive use of next() in an iterator to be the underlying problem. The interaction of generators and next() is just a special case of this. StopIteration is not a normal exception, indicating a problem, rather it exists to signal exhaustion of an iterator. However, next() raises StopIteration for an exhausted iterator, which really is an error. Any iterator code (generator or __next__ method) that calls next() treats the StopIteration as a normal exception and propogates it. The controlling loop then interprets StopIteration as a signal to stop and thus stops. *The problem is the implicit shift from signal to error and back to signal.* 2. The proposed solution does not address this issue at all, but rather legislates against generators raising StopIteration. 3. Generators and the iterator protocol were introduced in Python 2.2, 13 years ago. For all of that time the iterator protocol has been defined by the __iter__(), next()/__next__() methods and the use of StopIteration to terminate iteration. Generators are a way to write iterators without the clunkiness of explicit __iter__() and next()/__next__() methods, but have always obeyed the same protocol as all other iterators. This has allowed code to rewritten from one form to the other whenever desired. Do not forget that despite the addition of the send() and throw() methods and their secondary role as coroutines, generators have primarily always been a clean and elegant way of writing iterators. 4. Porting from Python 2 to Python 3 seems to be hard enough already. 5. I think I've already covered this in the other points, but to reiterate (excuse the pun): Calling next() on an exhausted iterator is, I would suggest, a logical error. However, next() raises StopIteration which is really a signal to the controlling loop. The fault is with next() raising StopIteration. Generators raising StopIteration is not the problem. It also worth noting that calling next() is the only place a StopIteration exception is likely to occur outside of the iterator protocol. An example -- Consider a function to return the value from a set with a single member. def value_from_singleton(s): if len(s) 2: #Intentional error here (should be len(s) == 1) return next(iter(s)) raise ValueError(Not a singleton) Now suppose we pass an empty set to value_from_singleton(s), then we get a StopIteration exception, which is a bit weird, but not too bad. However it is when we use it in a generator (or in the __next__ method of an iterator) that we get a serious problem. Currently the iterator appears to be exhausted early, which is wrong. However, with the proposed change we get RuntimeError(generator raised StopIteration) raised, which is also wrong, just in a different way. Solutions - My preferred solution is to do nothing except improving the documentation of next(). Explain that it can raise StopIteration which, if allowed to propogate can cause premature exhaustion of an iterator. If something must be done then I would suggest changing the behaviour of next() for an exhausted iterator. Rather than raise StopIteration it should raise ValueError (or IndexError?). Also, it might be worth considering making StopIteration inherit from BaseException, rather than Exception. Cheers, Mark. P.S. 5 days seems a rather short time to respond to a PEP. Could we make it at least a couple of weeks in the future, or better still specify a closing date for comments. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 3:03 PM, Georg Brandl g.bra...@gmx.net wrote: The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. This in particular is not really true on Github at least. By default PRs are made against whichever branch you have configured as the default branch in the Github UI. This does default to master but it doesn’t have to be, for instance pip has this configured for the develop branch. Although I think this specific point is moot because if things were on Git it’d probably make the most sense to have the default integration branch be ``master`` just like for the Hg repos they use default. Even if someone makes a PR against the wrong branch, it “degrades” into essentially the same UX as you have now, you can turn a PR into a patch by adding a .patch or .diff to the end of the PR URL and then you have a patch file. In additional github makes it easy to check out PRs directly with only a minor bit of one time configuration in your local clone. I can checkout any PR by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). Honestly the worst part about someone sending a PR to the wrong branch is it degrades into the same terrible UX that *every* patch has to go through on a hg.python.org repository right now. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 3:04:05 PM Georg Brandl g.bra...@gmx.net wrote: On 11/23/2014 05:55 PM, Brett Cannon wrote: I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. Well, since most people already know git this part is probably not a big deal, although we'd have to consider alienating core developers who are not git-savvy. Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. There are two categories of hooks we use: hooks that run before a push is accepted, and hooks that run afterwards. The latter are not a concern, since anything that GH/BB doesn't support can be run as a web service on python.org infrastructure, which then gets POST requests from the hosting platforms. These are * tracker update * IRC notification * sending email to python-checkins * triggering buildbot The more problematic category are pre-push hooks. We use them for checking and rejecting commits with * disallowed branches * non-conformant whitespace * wrong EOL style * multiple heads per named branch As far as I know, neither GH nor BB support such hooks, since they are highly project-specific. However, they are only used in cpython and related repositories, so that doesn't concern migration of doc-only repos. Sure, you can let the CI run the checks, but that doesn't prohibit merging and is circumvented by direct pushes to the repository that don't go through the PR system. (And please don't make me as a coredev open a PR for every change.) I'm not even going to touch the idea of requiring code review for all patches in the middle of this discussion. =) From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. In my opinion, scattering repos over github, bitbucket and hg.python.org is even less friendly to contributors than a centralized place. (We are already approaching this, with pydotorg and infrastructure repos on github.) So I'm going to add some thoughts here about migrating the main CPython to git+hub. I don't think we would ever split ourselves across three separate hosting services. At most I see two -- one for docs, one for CPython -- but I would honestly expect it to be only one long-term. We have to consider how well our branch workflow works with the PR workflow. There's no gain in the ability to easily merge a PR to one branch via github when the subsequent merge of 3.x to default/master requires a local pull/push cycle, as well as the 2.x backport. As far as I know, you'd have to open a pull/merge request yourself and instantly merge it, except if there are conflicts between branches, in which case you are again forced to go local. I don't need to mention that this doesn't work well when someone makes a concurrent commit to the source branch in the meantime. And I don't think you'd want to force people to submit 3 pull requests for the individual branches. The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 09:38 PM, Donald Stufft wrote: On Nov 23, 2014, at 3:03 PM, Georg Brandl g.bra...@gmx.net wrote: The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. This in particular is not really true on Github at least. By default PRs are made against whichever branch you have configured as the default branch in the Github UI. This does default to master but it doesn’t have to be, for instance pip has this configured for the develop branch. Although I think this specific point is moot because if things were on Git it’d probably make the most sense to have the default integration branch be ``master`` just like for the Hg repos they use default. Sure, although as is the majority of commits to CPython are bugfixes, which normally go to 2.7, 3.4 and 3.5 (master). Even if someone makes a PR against the wrong branch, it “degrades” into essentially the same UX as you have now, you can turn a PR into a patch by adding a .patch or .diff to the end of the PR URL and then you have a patch file. In additional github makes it easy to check out PRs directly with only a minor bit of one time configuration in your local clone. I can checkout any PR by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). Honestly the worst part about someone sending a PR to the wrong branch is it degrades into the same terrible UX that *every* patch has to go through on a hg.python.org repository right now. I'm not saying it's worse, but most of the time it's no better for the committer, especially since the branches have to be juggled in most cases. Although, when it's the same amount of work for the committer, but nicer for the contributor, that's a net win, I can see that. What is absolutely essential though is a way to automatically open an issue on bugs.python.org for each PR, otherwise we have to look for issues in two different places. (Sure, GH treats PRs like GH issues, but we wouldn't use the GH issue tracker.) Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 09:42 PM, Brett Cannon wrote: The more problematic category are pre-push hooks. We use them for checking and rejecting commits with * disallowed branches * non-conformant whitespace * wrong EOL style * multiple heads per named branch As far as I know, neither GH nor BB support such hooks, since they are highly project-specific. However, they are only used in cpython and related repositories, so that doesn't concern migration of doc-only repos. Sure, you can let the CI run the checks, but that doesn't prohibit merging and is circumvented by direct pushes to the repository that don't go through the PR system. (And please don't make me as a coredev open a PR for every change.) I'm not even going to touch the idea of requiring code review for all patches in the middle of this discussion. =) As far as I can see, our Rietveld is very well used already. *Requiring* code review for *all* patches is detrimental in a volunteer project IMO. (I'm saying this especially because many of my changes are small to trivial doc fixes.) The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. How do other projects tend to manage their bugfix vs. in-dev branches? Is it a lot of cherrypicking? Probably our long development cycles doesn't help with this as it makes cross-branch merging difficult thanks to divergence, else this could be automated somehow or simply not be an issue. Otherwise we would have to do a merge from branch to branch locally as we do now and make sure people committed in the branch we wanted them to. For smaller projects, it's not as much of an issue. In Sphinx, there are two branches (stable and default), and when there are pull requests to the default branch that should go to stable, I cherry-pick them manually. And as you said, merging or transplanting between branches is much more likely to succeed automatically. The but it is much easier to contribute simple changes argument is a bit simplified: for any nontrivial patch, the time spent on working out the code should outweight time spent with hg diff or click on pull request. And while Travis CI is nice, running relevant tests locally is *much* quicker than waiting for a full test suite run on a virtualized testing machine. Sure, but it's definitely easier to just wait for the Travis green on the PR than have to apply a patch and run the tests myself manually. You'll likely have to run them manually anyway while merging to the other branches. As for typo fixes, the world does not end when some typos aren't fixed. Anyway, for the docs we have an explicit offer to send anything, patch or just suggestion, to d...@python.org mailto:d...@python.org, and people do make use of it. No github account even required. Which is great, but I'm willing to bet clicking Edit in GitHub is easier still than creating the patch and emailing it. I have contributed doc fixes to a bunch of projects because of that Edit button (which is why I think Nick is so anxious to get it). If I had to do any checkout then I wouldn't have done them. And even having to email a diff might be too much for some changes. I did say patch or just suggestion. Most emails to d...@python.org come without a patch, just a plain description of what's wrong, and that is just fine. I would also judge that 50% of the emails come from people who wouldn't have a github account in any case (so that the big shiny Edit button isn't an easy option). And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases). I see no problem with that, provided there's a cronjob that syncs the version in Doc/ to the external version reasonably often. Would that really be necessary? At least for the HOWTOs how often are they edited simultaneously as some code in CPython? The Clinic HOWTO is probably the only one that might get updated simultaneously. I'd also be curious to know how often the tutorial is updated simultaneously as well. I'd like the HOWTOs to stay part of Doc/, so changes in the external repo have to be merged in there somehow, and not only at release time. Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 4:08 PM, Georg Brandl g.bra...@gmx.net wrote: On 11/23/2014 09:38 PM, Donald Stufft wrote: On Nov 23, 2014, at 3:03 PM, Georg Brandl g.bra...@gmx.net wrote: The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. This in particular is not really true on Github at least. By default PRs are made against whichever branch you have configured as the default branch in the Github UI. This does default to master but it doesn’t have to be, for instance pip has this configured for the develop branch. Although I think this specific point is moot because if things were on Git it’d probably make the most sense to have the default integration branch be ``master`` just like for the Hg repos they use default. Sure, although as is the majority of commits to CPython are bugfixes, which normally go to 2.7, 3.4 and 3.5 (master). Even if someone makes a PR against the wrong branch, it “degrades” into essentially the same UX as you have now, you can turn a PR into a patch by adding a .patch or .diff to the end of the PR URL and then you have a patch file. In additional github makes it easy to check out PRs directly with only a minor bit of one time configuration in your local clone. I can checkout any PR by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). Honestly the worst part about someone sending a PR to the wrong branch is it degrades into the same terrible UX that *every* patch has to go through on a hg.python.org repository right now. I'm not saying it's worse, but most of the time it's no better for the committer, especially since the branches have to be juggled in most cases. Right, well if you’re just merging one branch into another you can create a PR from one branch to another directly in the Github Web UI, which will run tests and then you can press the merge button on that PR too. It’s something like 3-4 clicks or so. If you want to selectively send a change from one branch to another that will require manually making the change locally yea. Although, when it's the same amount of work for the committer, but nicer for the contributor, that's a net win, I can see that. What is absolutely essential though is a way to automatically open an issue on bugs.python.org for each PR, otherwise we have to look for issues in two different places. (Sure, GH treats PRs like GH issues, but we wouldn't use the GH issue tracker.) There’s a web hook that fires when a pull request is created, it could even be smart enough to look for Issue numbers in the PR and commits so that it associates it with an existing issue instead of creating a new issue. You can see all of the events we can subscribe to here: https://developer.github.com/webhooks/#events In a related note, while it’s not possible to *reject* a push if things don’t match up it is entirely possible to detect it after the fact and send an email out to folks or what have you. Similar to how tests themselves are done now. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
Georg Brandl g.bra...@gmx.net wrote: On 11/23/2014 07:03 PM, Ryan wrote: I can agree with most of these points. Some more things to consider: - Git is 20x faster than Hg (that's 99% of the reason I switched and hate using Darcs) You won't get much traction with this argument around here. As long as there aren't specific complaints about slowness (which probably could be fixed by a discussion on mercurial-devel), Hg is fast enough, just like Python is fast enough compared to C. (Otherwise, why bother.) I know, but, when checking out large repos, it stings a little. - GitHub is much nicer than BitBucket. I usually look for a git mirror of a BitBucket repo since I hate browsing on BB I agree, Github's UI is nicer than BB. It is also true that most features that are introduced on BB are inspired by GH. - Even Plan9Port moved from Hg to GitHub a few days ago Projects move all the time. Why is this one special? Because, for ages, everything related to Rob Pike used Google Code and Mercurial, largely because he works at Google. - Some Google devs even host their projects on GitHub (not Google Code) Thank goodness no one has mentioned CodePlex. BTW, I frown on projects that choose something largely inferior over a better choice just because one is written in X language. Now, if python.org http://python.org was written in Ruby, that would just be sad, but certain things just don't matter. Like DVCS's. You can read up PEP 374 for the reasons for Hg. The main reasons against git were lacking Windows support (which probably is good enough now) and developer preference. Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam ron3...@gmail.com wrote:
With the passage of the PEP, it will change what is different about them
once it's in full effect. The stop hack won't work in both, and you may get
a RuntimeError in generator expressions where you would get StopIteration in
list-comps. (Is this correct?)
them being list comps and generator expressions?
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon m...@hotpy.org wrote: Hi, I have serious concerns about this PEP, and would ask you to reconsider it. Hoping I'm not out of line in responding here, as PEP author. Some of your concerns (eg 5 days is too short) are clearly for Guido, not me, but perhaps I can respond to the rest of it. [ Very short summary: Generators are not the problem. It is the naive use of next() in an iterator that is the problem. (Note that all the examples involve calls to next()). Change next() rather than fiddling with generators. ] StopIteration is not a normal exception, indicating a problem, rather it exists to signal exhaustion of an iterator. However, next() raises StopIteration for an exhausted iterator, which really is an error. Any iterator code (generator or __next__ method) that calls next() treats the StopIteration as a normal exception and propogates it. The controlling loop then interprets StopIteration as a signal to stop and thus stops. *The problem is the implicit shift from signal to error and back to signal.* The situation is this: Both __next__ and next() need the capability to return literally any object at all. (I raised a hypothetical possibility of some sort of sentinel object, but for such a sentinel to be useful, it will need to have a name, which means that *by definition* that object would have to come up when iterating over the .values() of some namespace.) They both also need to be able to indicate a lack of return value. This means that either they return a (success, value) tuple, or they have some other means of signalling exhaustion. I'm not sure what you mean by your However above. In both __next__ and next(), this is a signal; it becomes an error as soon as you call next() and don't cope adequately with the signal, just as KeyError is an error. 2. The proposed solution does not address this issue at all, but rather legislates against generators raising StopIteration. Because that's the place where a StopIteration will cause a silent behavioral change, instead of cheerily bubbling up to top-level and printing a traceback. 3. Generators and the iterator protocol were introduced in Python 2.2, 13 years ago. For all of that time the iterator protocol has been defined by the __iter__(), next()/__next__() methods and the use of StopIteration to terminate iteration. Generators are a way to write iterators without the clunkiness of explicit __iter__() and next()/__next__() methods, but have always obeyed the same protocol as all other iterators. This has allowed code to rewritten from one form to the other whenever desired. Do not forget that despite the addition of the send() and throw() methods and their secondary role as coroutines, generators have primarily always been a clean and elegant way of writing iterators. This question has been raised several times; there is a distinct difference between __iter__() and __next__(), and it is only the latter which is aware of StopIteration. Compare these three classes: class X: def __init__(self): self.state=0 def __iter__(self): return self def __next__(self): if self.state == 3: raise StopIteration self.state += 1 return self.state class Y: def __iter__(self): return iter([1,2,3]) class Z: def __iter__(self): yield 1 yield 2 yield 3 Note how just one of these classes uses StopIteration, and yet all three are iterable, yielding the same results. Neither Y nor Z is breaking iterator protocol - but neither of them is writing an iterator, either. 4. Porting from Python 2 to Python 3 seems to be hard enough already. Most of the code broken by this change can be fixed by a mechanical replacement of raise StopIteration with return; the rest need to be checked to see if they're buggy or unclear. There is an edge case with return some_value vs raise StopIteration(some_value) (the former's not compatible with 2.7), but apart from that, the recommended form of code for 3.7 will work in all versions of Python since 2.2. 5. I think I've already covered this in the other points, but to reiterate (excuse the pun): Calling next() on an exhausted iterator is, I would suggest, a logical error. How do you know that it's exhausted, other than by calling next() on it? It also worth noting that calling next() is the only place a StopIteration exception is likely to occur outside of the iterator protocol. This I agree with. An example -- Consider a function to return the value from a set with a single member. def value_from_singleton(s): if len(s) 2: #Intentional error here (should be len(s) == 1) return next(iter(s)) raise ValueError(Not a singleton) Now suppose we pass an empty set to value_from_singleton(s), then we get a StopIteration exception, which is a bit weird, but not too bad. Only a little weird - and no different from the way you'd get a
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/23/2014 3:18 AM, Nick Coghlan wrote: Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. Exactly. None of the patches I care about have gone anywhere for years. Two of them were done by others, apparently using the right processes, and still went nowhere, so it doesn't encourage me to attempt to even learn the process to make a more appropriate contribution of mine (which, sadly, I fixed too many different bugs in one file or two files, and have not split them into individual patches). Another has recently been making some progress... I guess that means I don't care enough about them, to fight to force them through the system, it is easier just to apply the patches to my installation... but that is a sad state of affairs. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support for Linux perf
2014-11-22 7:44 GMT-05:00 Julian Taylor jtaylor.deb...@googlemail.com: On 17.11.2014 23:09, Francis Giraldeau wrote: Hi, ... The PEP-418 is about performance counters, but there is no mention o Anyway, I think we must change CPython to support tools such as perf. Any thoughts? there are some patches available adding systemtap and dtrace probes, which should at least help getting function level profiles: http://bugs.python.org/issue21590 Thanks for these links, the patches looks interesting. As Jonas mentioned, Perf should be able to unwind a Python stack. It does at the interpreter level, and the frame info is scattered in virtual memory. It needs to be access offline. I think it could be possible to use the function entry and exit hooks in the interpreter to save important frame info, such as function name, file and line number, in a memory map known to perf. Then, we can tell Perf to record this compact zone of data in the sample as extra field for offline use. Then, at the analysis time, each ELF interpreter frame could be matched with the corresponding Python frame info. I think the perf handler can't sleep, and accesses on each function entry/exit will also ensure the page is present in main memory when the sample is recorded. Thanks again for your inputs, I'll post any further developments. Cheers, Francis ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/23/2014 04:15 PM, Chris Angelico wrote:
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adamron3...@gmail.com wrote:
With the passage of the PEP, it will change what is different about them
once it's in full effect. The stop hack won't work in both, and you may get
a RuntimeError in generator expressions where you would get StopIteration in
list-comps. (Is this correct?)
them being list comps and generator expressions?
Yes
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
Comprehensions insert/append items, so you wouldn't get those unless you
are reading from another object and would bubble out of generators too.
Which is good because it's most likely an error that needs fixing.
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
I think the difference is very specific to StopIteration.
And I think we need to wait and see what happens in real code.
Cheers,
Ron
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 10:18 AM, Ron Adam ron3...@gmail.com wrote:
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
Comprehensions insert/append items, so you wouldn't get those unless you are
reading from another object and would bubble out of generators too. Which is
good because it's most likely an error that needs fixing.
My point is that doing the same errant operation on a list or a dict
will give different exceptions. In the same way, calling next() on an
empty iterator will raise StopIteration normally, but might raise
RuntimeError instead. It's still an exception, it still indicates a
place where code needs to be changed (unlike, say, a ValueError when
doing type conversions, which often means *data* needs to be changed),
so it doesn't hugely matter _which_ exception is raised. Also, I'm
hoping that someone can improve on my patch by having the full
traceback from the original StopException become visible - then, the
RuntimeError's __context__ will give all the info you would want.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 08:17:00AM -0800, Ethan Furman wrote: While I am in favor of PEP 479, and I have to agree with Raymond that this isn't pretty. Currently, next() accepts an argument of what to return if the iterator is empty. Can we enhance that in some way so that the overall previous behavior could be retained? [...] Then, if the iterator is empty, instead of raising StopIteration, or returning some value that would then have to be checked, it could raise some other exception that is understood to be normal generator termination. We *already* have an exception that is understood to be normal generator termination. It is called StopIteration. Removing the long-standing ability to halt generators with StopIteration, but then recreating that ability under a different name is the worst of both worlds: - working code is still broken; - people will complain that the new exception X is silently swallowed by generators, just as they complained about StopIteration; - it is yet another subtle difference between Python 2 and 3; - it involves a code smell (no constant arguments to functions); - and does nothing to help generators that don't call next(). The current behaviour is nice and clean and has worked well for over a decade. The new behaviour exchanges consistency in one area (generators behave like all other iterators) for consistency in another (generator expressions will behave like comprehensions in the face of StopIteration). But trying to have both at the same time via a new exception adds even more complexity and would leave everyone unhappy. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun Nov 23 2014 at 4:18:37 PM Georg Brandl g.bra...@gmx.net wrote: On 11/23/2014 09:42 PM, Brett Cannon wrote: [SNIP] And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases). I see no problem with that, provided there's a cronjob that syncs the version in Doc/ to the external version reasonably often. Would that really be necessary? At least for the HOWTOs how often are they edited simultaneously as some code in CPython? The Clinic HOWTO is probably the only one that might get updated simultaneously. I'd also be curious to know how often the tutorial is updated simultaneously as well. I'd like the HOWTOs to stay part of Doc/, so changes in the external repo have to be merged in there somehow, and not only at release time. Right, I'm trying to understand *why* you want the HOWTOs to stay in Doc/. I dread having to update the porting HOWTO because it requires updating 2.7, 3.4, and default. And if the process is automated to pull from an external repo then what is the benefit of syncing the files and duplicating them across 4 repos? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 11/24/2014 12:21 AM, Brett Cannon wrote: On Sun Nov 23 2014 at 4:18:37 PM Georg Brandl g.bra...@gmx.net mailto:g.bra...@gmx.net wrote: On 11/23/2014 09:42 PM, Brett Cannon wrote: [SNIP] And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases). I see no problem with that, provided there's a cronjob that syncs the version in Doc/ to the external version reasonably often. Would that really be necessary? At least for the HOWTOs how often are they edited simultaneously as some code in CPython? The Clinic HOWTO is probably the only one that might get updated simultaneously. I'd also be curious to know how often the tutorial is updated simultaneously as well. I'd like the HOWTOs to stay part of Doc/, so changes in the external repo have to be merged in there somehow, and not only at release time. Right, I'm trying to understand *why* you want the HOWTOs to stay in Doc/. I dread having to update the porting HOWTO because it requires updating 2.7, 3.4, and default. And if the process is automated to pull from an external repo then what is the benefit of syncing the files and duplicating them across 4 repos? That they are still part of the docs on docs.python.org and what people download from there. I don't like resources like that scattered about. It makes sense for the devguide, since that is really a different topic. Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. Encouraging a software, or social, monopoly is never the right thing for the long term. http://nedbatchelder.com/blog/201405/github_monoculture.html I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. I'm sure that we'll get *more* contributions, but will they be *better* contributions? I know that there are people who think that mailing lists are old and passe, and that we should shift discussion to a social media site like Reddit. If we did, we'd probably get twenty times as many comments, and the average quality would probably plummet. More is not necessarily a good thing. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano st...@pearwood.info wrote: On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. Encouraging a software, or social, monopoly is never the right thing for the long term. http://nedbatchelder.com/blog/201405/github_monoculture.html I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. I'm sure that we'll get *more* contributions, but will they be *better* contributions? I know that there are people who think that mailing lists are old and passe, and that we should shift discussion to a social media site like Reddit. If we did, we'd probably get twenty times as many comments, and the average quality would probably plummet. More is not necessarily a good thing. If we need to ensure that we're getting better contributions than we are now, then we should be interviewing committers, rejecting newcomers (or the opposite, multiplying core-mentors by 100), and running this like a business. I've written some crappy code that got committed, so I should probably be fired. Enabling our community to be active contributors is an important thing. Give them a means to level up and we'll all be better off from it. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 23/11/14 22:54, Chris Angelico wrote: On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon m...@hotpy.org wrote: Hi, I have serious concerns about this PEP, and would ask you to reconsider it. Hoping I'm not out of line in responding here, as PEP author. Some of your concerns (eg 5 days is too short) are clearly for Guido, not me, but perhaps I can respond to the rest of it. [ Very short summary: Generators are not the problem. It is the naive use of next() in an iterator that is the problem. (Note that all the examples involve calls to next()). Change next() rather than fiddling with generators. ] StopIteration is not a normal exception, indicating a problem, rather it exists to signal exhaustion of an iterator. However, next() raises StopIteration for an exhausted iterator, which really is an error. Any iterator code (generator or __next__ method) that calls next() treats the StopIteration as a normal exception and propogates it. The controlling loop then interprets StopIteration as a signal to stop and thus stops. *The problem is the implicit shift from signal to error and back to signal.* The situation is this: Both __next__ and next() need the capability to return literally any object at all. (I raised a hypothetical possibility of some sort of sentinel object, but for such a sentinel to be useful, it will need to have a name, which means that *by definition* that object would have to come up when iterating over the .values() of some namespace.) They both also need to be able to indicate a lack of return value. This means that either they return a (success, value) tuple, or they have some other means of signalling exhaustion. You are grouping next() and it.__next__() together, but they are different. I think we agree that the __next__() method is part of the iterator protocol and should raise StopIteration. There is no fundamental reason why next(), the builtin function, should raise StopIteration, just because __next__(), the method, does. Many xxx() functions that wrap __xxx__() methods add additional functionality. Consider max() or min(). Both of these methods take an iterable and if that iterable is empty they raise a ValueError. If next() did likewise then the original example that motivates this PEP would not be a problem. I'm not sure what you mean by your However above. In both __next__ and next(), this is a signal; it becomes an error as soon as you call next() and don't cope adequately with the signal, just as KeyError is an error. 2. The proposed solution does not address this issue at all, but rather legislates against generators raising StopIteration. Because that's the place where a StopIteration will cause a silent behavioral change, instead of cheerily bubbling up to top-level and printing a traceback. I must disagree. It is the FOR_ITER bytecode (implementing a loop or comprehension) that silently converts a StopIteration exception into a branch. I think the generator's __next__() method handling of exceptions is correct; it propogates them, like most other code. 3. Generators and the iterator protocol were introduced in Python 2.2, 13 years ago. For all of that time the iterator protocol has been defined by the __iter__(), next()/__next__() methods and the use of StopIteration to terminate iteration. Generators are a way to write iterators without the clunkiness of explicit __iter__() and next()/__next__() methods, but have always obeyed the same protocol as all other iterators. This has allowed code to rewritten from one form to the other whenever desired. Do not forget that despite the addition of the send() and throw() methods and their secondary role as coroutines, generators have primarily always been a clean and elegant way of writing iterators. This question has been raised several times; there is a distinct difference between __iter__() and __next__(), and it is only the I just mentioned __iter__ as it is part of the protocol, I agree that __next__ is relevant method. latter which is aware of StopIteration. Compare these three classes: class X: def __init__(self): self.state=0 def __iter__(self): return self def __next__(self): if self.state == 3: raise StopIteration self.state += 1 return self.state class Y: def __iter__(self): return iter([1,2,3]) class Z: def __iter__(self): yield 1 yield 2 yield 3 Note how just one of these classes uses StopIteration, and yet all three are iterable, yielding the same results. Neither Y nor Z is breaking iterator protocol - but neither of them is writing an iterator, either. All three raise StopIteration, even if it is implicit. This is trivial to demonstrate: def will_it_raise_stop_iteration(it): try: while True: it.__next__() except StopIteration: print(Raises StopIteration) except: print(Raises something else) 4. Porting from Python 2 to
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 6:57 PM, Steven D'Aprano st...@pearwood.info wrote: On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. Encouraging a software, or social, monopoly is never the right thing for the long term. http://nedbatchelder.com/blog/201405/github_monoculture.html I don’t think this is really all that big of a deal. If we want to move off of Github doing so is easy. There are lots of (not nearly as good as but probably still better than what we have now) OSS software that gives you a github like flow. The only *data* that is really in there is what’s stored in the repository itself (since I don’t think for anything major we’d ever put issues there or use the wiki) which is trivial to move around. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano st...@pearwood.info
wrote:
On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote:
But I strongly believe that if we want to do the right thing for the
long term, we should switch to GitHub.
Encouraging a software, or social, monopoly is never the right thing for
the long term.
http://nedbatchelder.com/blog/201405/github_monoculture.html
I promise you that once the pain of the switch is over you will feel
much better about it. I am also convinced that we'll get more
contributions this way.
I'm sure that we'll get *more* contributions, but will they be *better*
contributions?
I know that there are people who think that mailing lists are old and
passe, and that we should shift discussion to a social media site like
Reddit. If we did, we'd probably get twenty times as many comments, and
the average quality would probably plummet. More is not necessarily a
good thing.
Reddit: Markdown. Edit. Save.
Is this a challenge for software configuration management
and software requirements traceability?
Could this problem be solved by utilizing namespaced URIs to reference
commits
and issues?
In indexing of these repositories, it would be useful to produce JSON-LD
with an appropriate schema for expressing primary
and other repo URIs,
that could be resolved into the appropriate ssh:// URLs
for each particular ascii domain regex.
- [ ] schema.org/SofwareApplication
- [ ] SEON Ontologies
- [ ] Warehouse/PyPi RDFa, JSON-LD
With secondarily productivity-enhancing URI/URL edges between development
artifacts and feedback review cycles being an overall objective for
infrastructure
and devops comprehension:
* Mailing lists, reviews, issues past current and future, documentation
pages, releases
* Indices (is there a sphinx index that could build the PEPs?)
* Reviews (rietveld, issue tracker)
* Issues Past
* Issues Current
* Issues Future
* Docs
* PEPs
* HOWTOs
* {this could be a useful set of types with URIs and current URLs}
When/if a r'#\d+' is referenced, how do the URLs resolve at render time?
When/if a 'r#issue/tracker/uri/{num}' URL is pasted, is there a pingback?
* sphinx extlinks
* https://github.com/lunaryorn/sphinxcontrib-issuetracker
* https://en.wikipedia.org/wiki/Requirements_traceability
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 2:38 PM, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 3:03 PM, Georg Brandl g.bra...@gmx.net wrote: The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. This in particular is not really true on Github at least. By default PRs are made against whichever branch you have configured as the default branch in the Github UI. This does default to master but it doesn’t have to be, for instance pip has this configured for the develop branch. Although I think this specific point is moot because if things were on Git it’d probably make the most sense to have the default integration branch be ``master`` just like for the Hg repos they use default. I'm liking the hubflow/gitflow git workflow extensions, where trunk is also develop. There are feature branches, hotfix branches, release branches, etc. # .git/config : prefix: v git hf release start 0.1.0 echo 0.1.0 VERSION.txt git hf release finish 0.1.0 # (merges, release commit message, return to develop) These are some of the most helpful (git) branching diagrams I've ever seen: * https://datasift.github.io/gitflow/IntroducingGitFlow.html * http://www.infoq.com/articles/agile-version-control The support for tagging releases is helpful. (It's possible to host package releases for various platforms as GitHub releases). Even if someone makes a PR against the wrong branch, it “degrades” into essentially the same UX as you have now, you can turn a PR into a patch by adding a .patch or .diff to the end of the PR URL and then you have a patch file. In additional github makes it easy to check out PRs directly with only a minor bit of one time configuration in your local clone. I can checkout any PR by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). Honestly the worst part about someone sending a PR to the wrong branch is it degrades into the same terrible UX that *every* patch has to go through on a hg.python.org repository right now. In this respect, hg mq is tightly integrated, and written in Python. Patch Queue / build system of VCS integration could be helpful. * http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html * http://hgbook.red-bean.com/read/advanced-uses-of-mercurial-queues.html I agree that - [ ] Build status displayed on the PR [is an extremely useful feature for a code forge to offer] - [ ] Markdown/ReStructuredText Checkboxes are helpful * GitHub: Issues * BitBucket: PR Task Lists integrated with review. My mouse cursor tends to be near the scroll bar alot, as well. -- https://en.wikipedia.org/wiki/Forge_(software)#Technology https://en.wikipedia.org/wiki/Comparison_of_open-source_software_hosting_facilities --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com -- Wes Turner https://westurner.github.io/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 06:08:07PM -0600, Brian Curtin wrote: On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano st...@pearwood.info wrote: I'm sure that we'll get *more* contributions, but will they be *better* contributions? I know that there are people who think that mailing lists are old and passe, and that we should shift discussion to a social media site like Reddit. If we did, we'd probably get twenty times as many comments, and the average quality would probably plummet. More is not necessarily a good thing. If we need to ensure that we're getting better contributions than we are now, then we should be interviewing committers, rejecting newcomers (or the opposite, multiplying core-mentors by 100), and running this like a business. I've written some crappy code that got committed, so I should probably be fired. None of those things are guarenteed to lead to better contributions. The quality of code from the average successful business is significantly lower than that from successful FOSS projects like Python. Interviews just weed out people who are poor interviewees, not poor performers. And any organisation that fires contributors for relatively trivial mistakes like crappy code would soon run out of developers. My point is that increasing the number of contributions is not, in and of itself, a useful aim to have. More contributions is just a means to an end, the end we want is better Python. Enabling our community to be active contributors is an important thing. Give them a means to level up and we'll all be better off from it. Right. But this isn't a feel-good exercise where anyone who wants a Gold Star for contributing gets commit privileges. (That would enable our community to be active contributors too.) Barriers to contribute work two ways: (1) we miss out on good contributions we would want; (2) we also miss out on poor contributions that would just have to be rejected. Enabling more people to contribute increases both. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 07:39:30PM -0500, Donald Stufft wrote: I don’t think this is really all that big of a deal. If we want to move off of Github doing so is easy. There are lots of (not nearly as good as but probably still better than what we have now) OSS software that gives you a github like flow. The only *data* that is really in there is what’s stored in the repository itself (since I don’t think for anything major we’d ever put issues there or use the wiki) which is trivial to move around. Assuming PRs are enabled on the new repo, how will that interact with patch review/the issue tracker? Is the goal here to replace the existing process wholesale in one step? It doesn't seem fair to say that moving to GitHub will be easy unless interrupting every contributor's flow is considered free, or some Rietveld bridge script magically writes itself. The former impacts review bandwidth, the latter infrastructure bandwidth (which already seems quite contended, e.g. given the job board is still MIA). David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
Chris already responded; I'll top-post to clarify my position. I acted quickly because too many people were expending too much energy debating the issue without coming up with a different resolution, and I am unhappy with the status quo. Many people don't seem to see the difference between the iterator *protocol*, which determines how the caller discovered that the iterator is exhausted, and various iterator *implementations*, which determine how you feed a series of values to the caller and indicate the end of that series. Generators are one possible implementation. The crucial insight, leading to the solution, is that a generator is one step removed from the object representing the protocol. When you call a generator, two objects are created: a generator protocol object (this is what we typically refer to as a generator object, as opposed to a generator function), and a stack frame. The stack frame is initially in a suspended state, and is poised to execute the code of the generator function. The protocol object has a __next__() method. When this is called, the associated stack frame is resumed until it either yields or returns -- if it yields, __next__() returns a value, but if it returns, __next__() raises StopIteration. There is no need for the stack frame to raise StopIteration -- the StopIteration is provided for free by the generator protocol object. The confusion stems from the fact that if the stack frame neither yields nor returns but raises an exception, this exception is simply passed through the protocol object, and in most cases (e.g. an AttributeError) this simply bubbles out until it is either caught or terminates the program or thread with a printed stack trace. But in the special case that the frame raises StopIteration, once this is raised by the protocol object's __next__(), this is indistinguishable from the StopIteration that this object throws when the stack frame simply returns. It wouldn't be so bad if we had the occasional generator author writing raise StopIteration instead of return to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. One more thing. You correctly say that the built-in next() function is not the same thing as the __next__() method of the protocol. While this is true, tinkering with next() is not feasible -- if we were to change the interface of next() we would have to change pretty much every call to it. (I did a little research on a large proprietary codebase to which I have access -- I found innumerable next() calls, almost all of them guarded by a try: ... except StopIteration: ... block. Even the ones inside generators. On the other hand, I found almost no code depending on the behavior that the PEP changes, apart from a few places where a redundant raise StopIteration was found at the end of a generator. So. To reiterate my point. A generator frame uses yield value to produce a value and return to terminate the iteration; the generator protocol object wrapping the frame translates these into return value and raise StopIteration, respectively. Raising StopIteration in the frame is redundant and can cause errors to pass silently, hence this is the right thing to change. On Sun, Nov 23, 2014 at 12:18 PM, Mark Shannon m...@hotpy.org wrote: Hi, I have serious concerns about this PEP, and would ask you to reconsider it. [ Very short summary: Generators are not the problem. It is the naive use of next() in an iterator that is the problem. (Note that all the examples involve calls to next()). Change next() rather than fiddling with generators. ] I have five main concerns with PEP 479. 1. Is the problem, as stated by the PEP, really the problem at all? 2. The proposed solution does not address the underlying problem. 3. It breaks a fundamental aspect of generators, that they are iterators. 4. This will be a hindrance to porting code from Python 2 to Python 3. 5. The behaviour of next() is not considered, even though it is the real cause of the problem (if there is a problem). 1. The PEP states that The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. I don't believe that to be the case; if someone knows what StopIteration is and how it is used, then the interaction is entirely as expected. I believe the naive use of next() in an iterator to be the underlying problem. The interaction of generators and next() is just a special case of this. StopIteration is not a normal exception, indicating a problem, rather it exists to signal exhaustion of an iterator. However, next() raises StopIteration for an exhausted
Re: [Python-Dev] Please reconsider PEP 479.
On 11/23/14, Mark Shannon m...@hotpy.org wrote: [...] You are grouping next() and it.__next__() together, but they are different. I think we agree that the __next__() method is part of the iterator protocol and should raise StopIteration. There is no fundamental reason why next(), the builtin function, should raise StopIteration, just because __next__(), the method, does. Many xxx() functions that wrap __xxx__() methods add additional functionality. Consider max() or min(). Both of these methods take an iterable and if that iterable is empty they raise a ValueError. If next() did likewise then the original example that motivates this PEP would not be a problem. FWIW , I fully agree with this . My (personal) impression about PEP 479 is that 1. All the drawbacks mentioned by Raymond Hettinger , Nick Coghlan et al. in the other thread are **really** serious 2. The PEP actually over-complicates the so-far-*natural* definition of the iterator protocol for generators ... and proposed solution adds more issues than it really solves . 3. The fault is with next() raising StopIteration. Generators raising StopIteration is not the problem. since the later is just an instance of sub-typing whereas the former is more about an exceptional branch I'm not sure what you mean by your However above. In both __next__ and next(), this is a signal; it becomes an error as soon as you call next() and don't cope adequately with the signal, just as KeyError is an error. 2. The proposed solution does not address this issue at all, but rather legislates against generators raising StopIteration. Because that's the place where a StopIteration will cause a silent behavioral change, instead of cheerily bubbling up to top-level and printing a traceback. I must disagree. It is the FOR_ITER bytecode (implementing a loop or comprehension) that silently converts a StopIteration exception into a branch. I think the generator's __next__() method handling of exceptions is correct; it propogates them, like most other code. This is really true and is the basis for composing generator expressions (the discussion's been too long I do not want to add more examples to emphasize this point) . IMHO StopIteration should be propagated up to the caller in the context of iterator composition (i.e. definition) as opposite to the case of client code actually *using* (i.e. consuming) the generator (and the difference between both scenarios is somehow similar to has-a vs is-a in classical OO subtyping scenarios) . In the later case (use) raising ValueError or RuntimeError (I'd prefer the former) would be really helpful , so I really favor doing so in next() method rather than over-complicating generators (and breaking the iterator subtype condition) for no good (IMHO) reason . [...] p.s. I know that the PEP has been accepted by the BDFL , but I really think this is an important concern , that's why I insist for the sake of helping ... in case this represents a serious violation of established rules please send me a private message and I will not do it again (... but I'm hoping , after all, that post-acceptance debate will not be considered as harmful when there's a good reason according to someone ...) -- Regards, Olemis - @olemislc Apache(tm) Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 04:49 PM, Nick Coghlan wrote: Moving from self-hosted Mercurial repos to externally hosted Mercurial repos is a low risk change. It reduces maintenance overhead and lowers barriers to external contribution, both without alienating existing contributors by forcing them to change their workflows. If those repos are externally maintained, what kind of assurances will people have that they are talking to the *official* repositories of the PSF owned assets? One of the problem IMHO of the democratization of branches that a dvcs provides is knowing when you are interacting with the official code of the project. In general, more democracy is better, but that needs to be balanced with verifiable reputation. Having branches hosted on python.org gives people that immediately. Having branches hosted on other domains means there's more uncertainty. Even aside from the we should support open source question (which I feel strongly about but acknowledge others have less allegiance to), it must be absolutely clear that there are repositories which we as the Python development community, bless. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 08:55 AM, Guido van Rossum wrote: - Moving from Hg to Git is a fair amount of one-time work For anyone seriously interested in this, even experimentally, I would highly suggest looking at Eric Raymond's reposurgeon code. You can google it or find it covered in the vast discussions of his conversion of the Emacs repository from bazaar to git. I don't know for a fact that it would handle an hg to git conversion, but it's the most likely candidate for something so complex. There's a lot of similarity in the history of the Emacs and Python repositories, having gone through many previous iterations of vcs's - in Python's case, at least cvs, svn, and hg. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Nov 23, 2014, at 03:59 PM, Nick Coghlan wrote: The learning curve on git is still awful What I find so ironic is that git's model is beautifully simple, but its cli is abysmal, and its manpages are less than helpful. git-push(1) is over 650 lines and it's nearly impossible to dig out the most important bits. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ubuntu buildbot
Hi David, I noticed you run the Builder x86 Ubuntu Shared buildbot. It seems it's running a very old version of Ubuntu. Is there any chance of getting that updated? Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
git-push(1) is over 650 lines and it's nearly impossible to dig out the most important bits. I use git daily at work. I try to use it in the most simple way possible. My frustration with the man pages got to the point where I basically use Google to ask my questions, then bookmark the solutions I find (which often turn out to be on stackoverflow). Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sunday, November 23, 2014, Skip Montanaro skip.montan...@gmail.com wrote: git-push(1) is over 650 lines and it's nearly impossible to dig out the most important bits. I use git daily at work. I try to use it in the most simple way possible. My frustration with the man pages got to the point where I basically use Google to ask my questions, then bookmark the solutions I find (which often turn out to be on stackoverflow). Then there's this. http://git-man-page-generator.lokaltog.net/ -- --Guido van Rossum (on iPad) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Mon, Nov 24, 2014 at 2:42 PM, Guido van Rossum gu...@python.org wrote: Then there's this. http://git-man-page-generator.lokaltog.net/ Wow scarily accurate. http://git-man-page-generator.lokaltog.net/#2d1a13476a5f32c4db27fd7aa89a84f3 Anything to do with git submodules is virtually impossible to distinguish from an elaborate practical joke. I know this because I have tried to use them. But there is hope: the git maintainers *do* accept docs patches. https://www.kernel.org/pub/software/scm/git/docs/git-config.html#_configuration_file Formerly, this said You will find a description of non-core porcelain configuration variables in the respective porcelain documentation in the appropriate manual page.. Did you know that that means it's acceptable for a third-party git hook to make use of 'git config' to allow end users to configure it? Neither did I, till I asked on the mailing list. However, there are enough git front-ends and tutorials that mean you can generally ignore the man pages until you need something more complicated. Basically, consider the git man pages at the same time you'd consider enabling a Mercurial extension. That's how it seems to be. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ubuntu buildbot
Yeah, it definitely needs it. Historically it was intentional as my own servers were all on 8.04, but the last of those moved 12.04 last year. I think there's already a 12.04 buildbot, so perhaps 14.04 would be better? I do prefer sticking with an LTS. It'll need to move to 64-bit given the hosting environment, at least for the kernel. I could do 32-bit userspace via multiarch if keeping a 32-bit buildbot in the mix is attractive. -- David On Sun, Nov 23, 2014 at 10:18 PM, Benjamin Peterson benja...@python.org wrote: Hi David, I noticed you run the Builder x86 Ubuntu Shared buildbot. It seems it's running a very old version of Ubuntu. Is there any chance of getting that updated? Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
Brett Cannon writes: How do other projects tend to manage their bugfix vs. in-dev branches? Emacs uses a similar workflow to Python's current one, AIUI: 1. When feasible, developer decides the lowest applicable branch (in Emacs there are only two), commits and pushes there. 2. When needed, merge forward to trunk, supported by a script that helps manage already cherry-picked from trunk and not suitable for trunk patches. Is it a lot of cherrypicking? For Emacs, not that much. Most feature development occurs on trunk, and rarely are commits to trunk considered appropriate for cherrypicking to stable. Most bugfixing is in response to reports on stable, or quickly confirmed to apply to stable, is handled in stable, and then merged forward manually. Probably our long development cycles doesn't help with this as it makes cross-branch merging difficult thanks to divergence, I suspect that the time from bug injection to bug discovery, which is primarily related to length of support commitment, is more relevant than the length of the development cycle. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 24 November 2014 at 02:55, Brett Cannon br...@python.org wrote: On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan ncogh...@gmail.com wrote: Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. Are you volunteering to write a competing PEP for a migration to git and GitHub? I won't be updating PEP 474 to recommend moving to either, as I don't think that would be a good outcome for the Python ecosystem as a whole. It massively undercuts any possible confidence anyone else might have in Mercurial, BitBucket, Rhodecode, Kallithea Allura (all Python based version control, or version control hosting, systems). If we as the Python core development team don't think any of those are good enough to meet the modest version control needs of our support repos, why on earth would anyone else choose them? In reality, I think most of these services are pretty interchangeable - GitHub's just been the most effective at the venture capital powered mindshare grab business model (note how many of the arguments here stem from the fact folks like *other* things that only interoperate with GitHub, and no other repository hosting providers - that's the core of the A18z funded approach to breaking the D in DVCS and ensuring that GitHub's investors are in a position to clip the ticket when GitHub eventually turns around and takes advantage of its dominant market position to increase profit margins). That's why I consider it legitimate to treat supporting fellow Python community members as the determining factor - a number of the available options meet the good enough bar from a technical perspective, so it's reasonable to take other commercial and community factors into account when making a final decision. Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. If CPython eventually followed suit in migrating to git (as seems inevitable if all the other repos were to switch), then every buildbot will also need to be updated to have git installed (and Mercurial removed). From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. Yes, I agree any real benefit comes from the PR workflow, not from git. This is why I consider written in Python to be a valid determining factor - multiple services meet the good enough bar from a practical perspective, allowing other considerations to come to the fore. (Also note that this proposal does NOT currently cover CPython itself. Neither GitHub nor BitBucket is set up to handle maintenance branches well, and any server side merge based workflow improvements for CPython are gated on fixing the NEWS file maintenance issue. However, once you contemplate moving CPython, then the ripple effects on other systems become much larger) It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. What are the automation scripts there are that would
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On 24 November 2014 at 06:42, Brett Cannon br...@python.org wrote: On Sun Nov 23 2014 at 3:04:05 PM Georg Brandl g.bra...@gmx.net wrote: As for typo fixes, the world does not end when some typos aren't fixed. Anyway, for the docs we have an explicit offer to send anything, patch or just suggestion, to d...@python.org, and people do make use of it. No github account even required. Which is great, but I'm willing to bet clicking Edit in GitHub is easier still than creating the patch and emailing it. I have contributed doc fixes to a bunch of projects because of that Edit button (which is why I think Nick is so anxious to get it). If I had to do any checkout then I wouldn't have done them. And even having to email a diff might be too much for some changes. Yes, it's the ease of noticing a typo or other small, easy to fix error in docs, going to the source page, hitting Edit, and submitting the PR that I like with the online editing support. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
