Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 7:14 pm, Vinay Sajipwrote: > suite is a boon in this regard. Though having worked through the > tests, it doesn't seem like the DRY principle is followed as much as > it could be ... for example, the same literals being used over and > over again in copy/paste fashion, requiring patches in multiple > locations to add u() and b() wrappers, for example. I didn't have time > to rationalise this, as I was focused more on identifying and fixing > failures and errors. Ironically, I notice that I repeated myself in using "for example" twice in the above paragraph ;-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 5:36 pm, Jannis Leidelwrote: > > Ah, that makes sense, in fact your approach is much closer to what I > remember doing when pip and virtualenv was ported. Right, since I did those ports originally :-) > Honestly, I'm not sure how hard the merge is, as I'm not sure how much > changed. Martin could probably shed some light on it how he wants to deal > with it (e.g. svnmerge.py or not). Sure. > Fair enough, I just realized that's a discussion we need to have in a > separate thread (~"What's the best approach for migrating Django projects > from 2.X to3.X?") that can be handled later in the porting process. When > in doubt I would rather use a module like six that has community traction > than writing our own though. There are areas where the current code needs to do metaclass-based checks, and that involves delving into the specifics of the implementation of with_metaclass. This being the case, I made a modified version of Benjamin's which uses "_DjangoBase" as the intermediate parent class. IMO we need this to distinguish from other classes implemented using with_metaclass from the official six package. > Personally I'm fine with it, but as you say, it requires discipline > (I broke pip more than once). But it's definitely something that needs > some input from the other core devs, and probably a very good > documentation of the dos and don'ts. Having good code coverage helps to spot these potential breakages well before a release (or even a checkin), and Django's extensive test suite is a boon in this regard. Though having worked through the tests, it doesn't seem like the DRY principle is followed as much as it could be ... for example, the same literals being used over and over again in copy/paste fashion, requiring patches in multiple locations to add u() and b() wrappers, for example. I didn't have time to rationalise this, as I was focused more on identifying and fixing failures and errors. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On 28.11.2011, at 18:08, Vinay Sajip wrote: > On Nov 28, 1:04 pm, Jannis Leidelwrote: >> >> I'm a bit concerned that you didn't get in touch with us before you >> started with the work, since tracking the changes would have been >> easier. FWIW, Martin von Löwis, Alex and me would be those you can >> ask if you need any further help, usually also on IRC in #django-dev. > > Well, it's only been about a week of elapsed time. I always start > these sort of ports on an experimental-and-potentially-throwaway > basis, and I didn't know until I started how things would progress - > and well, once I was into it, I was just dipping in and out amidst > doing other things, so I suppose I just focused on the task at hand > rather than the big picture. I see, that's absolutely fine, and to be honest, getting so much progress done is definitely a good problem to have :) >> Have you worked on top of the current py3k branch in SVN (which should >> be mirrored in the Mercurial repo)? Also, did you get in touch with >> Martin von Löwis who has previously spearheaded the porting efforts >> for a while and has been granted commit privileges to the py3k branch? > > Yes, I used the BitBucket repo and the features/py3k branch within > that. I didn't contact Martin directly about this, as my approach is a > little different to what he had started, in that Martin's approach is > based on running 2to3 on the code to get new 3-friendly code, whereas > my approach is to have a single codebase with runs on 2.x and 3.x. > It's more than a small philosophical difference - I value the > information that 2to3 gives, but it just acts as the driver for > porting "by hand" with my standard development tools. Ah, that makes sense, in fact your approach is much closer to what I remember doing when pip and virtualenv was ported. In that sense, I'm definitely fine with that. How much of that approach needs to be documented for end users is probably something we can deal with later. >> I'm asking simply because I'm wondering how we should get your changes >> reviewed and committed to the official repo, it's a bit like facing a >> done deal. > > Well, I undertook the approach you mentioned in the "Python 3 and you" > thread back in September, which was to look at the features/py3k > branch and focus on test failures. Of course I didn't use the py3ktest > script (which involves the 2to3 pass), running just plain runtests.py > instead. And it's only been a little over a week, so it can't be that > hard to merge the changes assuming you agree with the approach. Even > if you start over, it's still less than a person-week of effort, > right? Honestly, I'm not sure how hard the merge is, as I'm not sure how much changed. Martin could probably shed some light on it how he wants to deal with it (e.g. svnmerge.py or not). >> Aha, meaning that you've copied over parts of the six module? Would you >> (also as a way for helping out Django users later to port their code) >> recommend putting six completely in Django? We've previously done that >> with other libraries, e.g. unittest2. > > I have no strong views on using six directly as a dependency, and I've > only used small parts of it (with_metaclass, reraise) directly. It's > certainly not needed as a dependency - you can see that the > django.utils.py3 module (which provides the functionality needed by > Django) is pretty small. Fair enough, I just realized that's a discussion we need to have in a separate thread (~"What's the best approach for migrating Django projects from 2.X to 3.X?") that can be handled later in the porting process. When in doubt I would rather use a module like six that has community traction than writing our own though. >> Indeed they are, so the next step is to review them bit by bit and align >> it with the work done before your work. Ideally committing it to SVN. > > Right, and I can provide some help with that (other work permitting). > Possibly some work could be done to look at merging the default branch > with the features/py3k branch, as a first step - presumably default > tracks SVN trunk closely. The main thing is to see whether you are > comfortable with the single codebase approach (requires some small > additional discipline from developers to consider str/bytes issues > carefully, do imports from utils.py3 where appropriate, etc. - nothing > too onerous). Personally I'm fine with it, but as you say, it requires discipline (I broke pip more than once). But it's definitely something that needs some input from the other core devs, and probably a very good documentation of the dos and don'ts. > Carl Meyer may be able to chime in with his experience > re. pip/virtualenv, which were ported using the same approach a while > ago, and have seen maintenance work and new releases since then. Also > it's worth looking at the way metaclasses have been done in the port > and to see if you can identify any issues. A lot of the changes are >
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 9:29 am, Kiril Vladimirovwrote: > My point was the social factor. I mean the GitHub community is quite > bigger. It's not big issue if we're using mercurial instead of git. Sure, I > prefer git, it's faster and stuff, but it's not a big deal, right now. Well, it's not as if there's a big team required, and presumably interested parties who can contribute will be subscribed to this list anyway. There probably isn't the kind of need for a lot of back-and- forth, given that the overall approach and some specific implementation decisions are acceptable to the core devs. > Anyway, I'm trying to get into making the tests run and if I see some > result from my work, will try to figure out some sync between github and > bitbucket. Great, we'd like to see what you find. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 1:04 pm, Jannis Leidelwrote: > > I'm a bit concerned that you didn't get in touch with us before you > started with the work, since tracking the changes would have been > easier. FWIW, Martin von Löwis, Alex and me would be those you can > ask if you need any further help, usually also on IRC in #django-dev. > Well, it's only been about a week of elapsed time. I always start these sort of ports on an experimental-and-potentially-throwaway basis, and I didn't know until I started how things would progress - and well, once I was into it, I was just dipping in and out amidst doing other things, so I suppose I just focused on the task at hand rather than the big picture. > Have you worked on top of the current py3k branch in SVN (which should > be mirrored in the Mercurial repo)? Also, did you get in touch with > Martin von Löwis who has previously spearheaded the porting efforts > for a while and has been granted commit privileges to the py3k branch? Yes, I used the BitBucket repo and the features/py3k branch within that. I didn't contact Martin directly about this, as my approach is a little different to what he had started, in that Martin's approach is based on running 2to3 on the code to get new 3-friendly code, whereas my approach is to have a single codebase with runs on 2.x and 3.x. It's more than a small philosophical difference - I value the information that 2to3 gives, but it just acts as the driver for porting "by hand" with my standard development tools. > I'm asking simply because I'm wondering how we should get your changes > reviewed and committed to the official repo, it's a bit like facing a > done deal. Well, I undertook the approach you mentioned in the "Python 3 and you" thread back in September, which was to look at the features/py3k branch and focus on test failures. Of course I didn't use the py3ktest script (which involves the 2to3 pass), running just plain runtests.py instead. And it's only been a little over a week, so it can't be that hard to merge the changes assuming you agree with the approach. Even if you start over, it's still less than a person-week of effort, right? > Aha, meaning that you've copied over parts of the six module? Would you > (also as a way for helping out Django users later to port their code) > recommend putting six completely in Django? We've previously done that > with other libraries, e.g. unittest2. I have no strong views on using six directly as a dependency, and I've only used small parts of it (with_metaclass, reraise) directly. It's certainly not needed as a dependency - you can see that the django.utils.py3 module (which provides the functionality needed by Django) is pretty small. > Yeah, we've talked about that and don't consider 3.0 or 3.1 to be sensible to > support in the future (given obvious bugs and/or lack of support from Python > core). Good, that rhymes with my thinking on it :-) > Indeed they are, so the next step is to review them bit by bit and align > it with the work done before your work. Ideally committing it to SVN. Right, and I can provide some help with that (other work permitting). Possibly some work could be done to look at merging the default branch with the features/py3k branch, as a first step - presumably default tracks SVN trunk closely. The main thing is to see whether you are comfortable with the single codebase approach (requires some small additional discipline from developers to consider str/bytes issues carefully, do imports from utils.py3 where appropriate, etc. - nothing too onerous). Carl Meyer may be able to chime in with his experience re. pip/virtualenv, which were ported using the same approach a while ago, and have seen maintenance work and new releases since then. Also it's worth looking at the way metaclasses have been done in the port and to see if you can identify any issues. A lot of the changes are pretty mechanical, wrapping literals with u() and b() - nothing too contentious there. Escape character handling esp. in regexes is another area where closer inspection would be worthwhile. The test codebase contained a lot of Unicode literals (i.e. Unicode literals in the source code, which would have been encoded in UTF-8 in the files actually stored on disk) and I have converted these to use Unicode escapes, as this is more portable. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
Hi Vinay, Many thanks for your efforts so far, as you can imagine I have a few questions, both procedural and technical. I should note though that I haven't reviewed all your changes in detail yet, since they are.. massive :) I'm a bit concerned that you didn't get in touch with us before you started with the work, since tracking the changes would have been easier. FWIW, Martin von Löwis, Alex and me would be those you can ask if you need any further help, usually also on IRC in #django-dev. On 25.11.2011, at 18:33, Vinay Sajip wrote: > I'm working on a port of Django to Python 3. I'm getting close, and in > terms of > test coverage pretty much almost there, but a few remaining test > failures are > eluding me, and I could probably use some help to speed things up. > > I started with the features/py3k branch on the BitBucket Django mirror > (the one > at https://bitbucket.org/django/django), but whereas that was > approaching from > a point of view of running 2to3 over the Django codebase, I've > followed my > preferred strategy of aiming for a single codebase for 2.x and 3.x. > (This > strategy worked well for the virtualenv and pip ports which I did a > while ago.) Have you worked on top of the current py3k branch in SVN (which should be mirrored in the Mercurial repo)? Also, did you get in touch with Martin von Löwis who has previously spearheaded the porting efforts for a while and has been granted commit privileges to the py3k branch? I'm asking simply because I'm wondering how we should get your changes reviewed and committed to the official repo, it's a bit like facing a done deal. > I've updated the django.utils.py3 module to add whatever I needed, > gratefully > borrowing ideas from Benjamin Peterson's six project as needed. Aha, meaning that you've copied over parts of the six module? Would you (also as a way for helping out Django users later to port their code) recommend putting six completely in Django? We've previously done that with other libraries, e.g. unittest2. > I run the standard test suite on the codebase using Python 2.7.2 and 3.2.2 > (I'm > not supporting 3.0 or 3.1 - 3.2 is stable, 'callable' came back and is > liberally > used in Django, and I'm not sure it's worth bothering with support for > 3.0/3.1. Yeah, we've talked about that and don't consider 3.0 or 3.1 to be sensible to support in the future (given obvious bugs and/or lack of support from Python core). > The branch is available at > > https://bitbucket.org/vinay.sajip/django (features/py3k branch) > > Latest test result summaries are as follows: > > 2.7.2: ran 4229 tests in 301.690s > OK (skipped=71, expected failures=3) > > 3.2.2: Ran 4174 tests in 303.619s > FAILED (failures=6, errors=2, skipped=78, expected failures=2, > unexpected successes=1) > > I think these results are encouraging, and I hope you agree! I added 7 > skips, mostly these are due to different representations on 2.x and 3.x e.g. > u'foo' vs. 'foo'. Indeed they are, so the next step is to review them bit by bit and align it with the work done before your work. Ideally committing it to SVN. Jannis -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Sat, Nov 26, 2011 at 1:33 AM, Vinay Sajipwrote: > I'm working on a port of Django to Python 3. I'm getting close, and in > terms of > test coverage pretty much almost there, but a few remaining test > failures are > eluding me, and I could probably use some help to speed things up. Hi Vinay, This is awesome work! Getting down to less than a dozen test failures is amazing progress. I don't have anything specific to add beyond that; I just wanted to let you know the core team (or, at least, this member of the core team) has seen your work, and is really encouraged that people are making serious inroads into making Django on Py3 a reality. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
My point was the social factor. I mean the GitHub community is quite bigger. It's not big issue if we're using mercurial instead of git. Sure, I prefer git, it's faster and stuff, but it's not a big deal, right now. Anyway, I'm trying to get into making the tests run and if I see some result from my work, will try to figure out some sync between github and bitbucket. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/RJymGD-0e68J. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
BitBucket has full git support btw. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/Ax7zncVO2FYJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 27, 2:10 pm, Kiril Vladimirovwrote: > Have you made some sort of TODO list I could use? > Or selecting some failing test and make it run fine would be fine as well? I would say: just clone the repo, try to reproduce the results, report any differences you find from my results, and select any failing test to fix. Another area to look at is to exercise other database backends, e.g. PostgreSQL. I've only looked at sqlite as the obvious backend to use first. > Also, how about moving this project to GitHub? > Django is there, too(https://github.com/django/) and I thing we could find > more participants. It just so happened that I found the BitBucket mirror first and used that, but I wouldn't expect a serious developer, who wants to pitch in, to object to using Mercurial and BitBucket instead. There's very little to choose between them (BitBucket/GitHub), compared to the real work still to be done in getting Django production-ready on 3.x. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
Have you made some sort of TODO list I could use? Or selecting some failing test and make it run fine would be fine as well? Also, how about moving this project to GitHub? Django is there, too(https://github.com/django/) and I thing we could find more participants. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/wTlyOMg5dYkJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
Hi, I'm just learning python at the moment to use Django so I don't have the knowledge to help you but keep up the good work. I'm looking forward to seeing the result of your work. Cheers. :) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/Sy0bpiSC-XgJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
Sorry the formatting of the post got mangled - not sure what happened there! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
I've made good progress in porting Django to Python 3, and could use some help!
I'm working on a port of Django to Python 3. I'm getting close, and in terms of test coverage pretty much almost there, but a few remaining test failures are eluding me, and I could probably use some help to speed things up. I started with the features/py3k branch on the BitBucket Django mirror (the one at https://bitbucket.org/django/django), but whereas that was approaching from a point of view of running 2to3 over the Django codebase, I've followed my preferred strategy of aiming for a single codebase for 2.x and 3.x. (This strategy worked well for the virtualenv and pip ports which I did a while ago.) I've updated the django.utils.py3 module to add whatever I needed, gratefully borrowing ideas from Benjamin Peterson's six project as needed. I run the standard test suite on the codebase using Python 2.7.2 and 3.2.2 (I'm not supporting 3.0 or 3.1 - 3.2 is stable, 'callable' came back and is liberally used in Django, and I'm not sure it's worth bothering with support for 3.0/3.1. The branch is available at https://bitbucket.org/vinay.sajip/django (features/py3k branch) Latest test result summaries are as follows: 2.7.2: ran 4229 tests in 301.690s OK (skipped=71, expected failures=3) 3.2.2: Ran 4174 tests in 303.619s FAILED (failures=6, errors=2, skipped=78, expected failures=2, unexpected successes=1) I think these results are encouraging, and I hope you agree! I added 7 skips, mostly these are due to different representations on 2.x and 3.x e.g. u'foo' vs. 'foo'. I'm posting the summary results to https://gist.github.com/1373553 The verbose listings are too big to post together to Github, giving me an "Entity Too Large" error. So these are posted separately to https://gist.github.com/1393994 (2.7.2) and https://gist.github.com/1394000 (3.2.2). The failures are: ERROR: test_bad_404_template (regressiontests.test_client_regress.models.TemplateExceptionTests) I think this is due to Django trying to handle an unaught exception and failing to find the 4500.html template, because the test has set the templates path to a bad value (deliberately, for the purpose of the test). ERROR: test_cull (regressiontests.cache.tests.DBCacheTests) This raises a "sqlite3.IntegrityError: datatype mismatch" but I haven't yet found the underlying cause. FAIL: test_modelform_factory_metaclass (regressiontests.model_forms_regress.tests.CustomMetaclassTestCase) This is down to the way metaclasses are implemented slightly differently. FAIL: test_lazy_objects (regressiontests.i18n.tests.TranslationTests) Proxy object comparison code differs between 2.x and 3.x, needs looking at. FAIL: test_RegexValidator_raises_error_23 (modeltests.validators.tests.TestSimpleValidators) I haven't tracked this down yet. FAIL: testParsing (regressiontests.conditional_processing.models.ETagProcessing) This is down to different representations on 2.x and 3.x: e\"t\"ag vs. e"t"ag FAIL: test_null_display_for_field (regressiontests.admin_util.tests.UtilTests) This is also down to proxy comparison logic. FAIL: test_templates (regressiontests.templates.tests.Templates) This is down to representational differences between 2.x and 3.x. I did the work and ran the tests on a VM running Ubuntu Oneiric Ocelot (x64). You can hopefully reproduce these by cloning my repository, updating to the features/py3k branch, and then running one of PYTHONPATH=.. python2.7 -u runtests.py --settings test_sqlite or PYTHONPATH=.. python3.2 -u runtests.py --settings test_sqlite in the tests directory. The work's taken about a week of elapsed time (on and off), and I haven't tracked any changes in the default branch since then. Of course, I realise there's still a lot left to do, but I hope I have broken the back of it. I've approached it just from the point of trying to get the tests to pass, and not (yet) focused on actually running realistic scenarios. Perhaps some of my fixes need changing - having other devs' eyeballs on the code can definitely help. Please let me have your comments! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.