Re: Python 3: should we apply unicode_literals everywhere?
I would also prefer Option 2, as the places where str('...') are needed are
not all that many.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/django-developers/-/WLtnInRyKyAJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 - style question
I think Option 2 is better, for the reasons you state. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port status
On Jul 24, 7:37 pm, Aymeric Augustin wrote: > Since Django's test suite isn't exhaustive, we focus on code review rather > than passing tests to validate the > changes. That's why we're doing them step by step rather than in a huge > merge. We're coordinating this effort > on the #django-dev IRC channel. Sure. The test results are just one metric, which one might class as "necessary but not sufficient". There are numerous places where things IMO aren't done correctly in the main Django repo: for example, using BytesIO in some places where StringIO should be used, and using b prefixes a little too liberally (for example, if you use a b prefix for a class name when constructing a type, it will fail on Python 3.x). If you think it would be helpful for me to be available on #django- dev, I can try to do that more often. > Of course, I'm frequently looking at your branch to see how you've resolved > the various problems. It would be > very useful to view the entire diff between your branch and master. > Unfortunately I couldn't figure out how to do > this on GitHub. The "Files changed" tab > onhttps://github.com/vsajip/django/compare/django3shows a lot of > differences in AUTHORS that I wouldn't expect to find there. Any idea? I've not intentionally made any changes to AUTHORS. I started working with Git the way one is supposed to, but I've found that for some reason, every time I do a merge with master, the same conflicts keep coming up again and again, even if they were resolved in previous merges; this makes it hard for me to use Git to keep my branch synchronised with changes in the upstream repo. I mentioned this to Claude a while ago, but I haven't been able to find out why I have so much trouble. (I've tried both fast-forward and no fast-forward options when merging). Possibly something went wrong in one of my merges, but AFAIK the AUTHORS file should be identical. I actually keep 3 repos around: django-upstream, which I never touch but just pull from the main Django repo, then my repo which I use to publish my changes, and another Mercurial repo which I use for help with merging due to the aforementioned problems with Git merging. I just checked: if I run meld to compare django-upstream and my Django repo, I see no changes in the top-level directory other than in setup.py (which IIRC are changes Martin von Löwis made a long while ago to patch sysconfig in 2.7 / 3.2). I've found in the past that diff UIs are not completely reliable due to how they parse line endings (by which I mean Windows/Mac/Linux). Also, the diff is pretty big, so it may push the diff-determining code into less well tested areas. So I use meld, which hasn't let me down :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port status
Since the update of Django to use Benjamin Peterson's six package for single code-base compatibility, I've updated my port [1] to do likewise. All tests pass [2] with the SQLite backend on Ubuntu Linux (64-bit). I added to the small section at the end of six.py [3] of additional customisations for Django's use. Some of these additions may be removable in due course if appropriate changes are made to Django. The additions are IMO uncontroversial - mostly, they are to support uniform access for APIs which return lists on 2.x and iterators on 3.x. BTW, many tests fail if six's default StringIO is used on 2.x (this is StringIO.StringIO). However, cStringIO.StringIO works in those cases. I've tackled this by using from django.utils.six.moves import StringIO wherever possible, but from django.utils.six import StringIO where necessary. The former resolves to StringIO.StringIO on 2.x, while the latter resolves to cStringIO.StringIO. On 3.x, both resolve to io.StringIO and there are no failures due to StringIO usage. I'm currently unable to test with other backends, but will try to resolve any port-related problems which are fed back to me via the GitHub issue tracker on the repo [4]. Note that my changes are in the django3 branch, and not the master branch. Regards, Vinay Sajip [1] https://github.com/vsajip/django [2] https://gist.github.com/1373553 [3] https://github.com/vsajip/django/blob/django3/django/utils/six.py#L356 [4] https://github.com/vsajip/django/issues/new -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: json vs simplejson
On Jun 11, 10:51 pm, Luke Plant wrote:
> We've switched internally from json to simplejson. Our 1.5 release notes
> say:
Do you mean the other way around?
> You can safely change any use of django.utils.simplejson to json
>
> I just found a very big difference between json and simplejson
>
> >>> simplejson.loads('{"x":"y"}')
>
> {'x': 'y'}
>
> >>> json.loads('{"x":"y"}')
>
> {u'x': u'y'}
>
> i.e. simplejson returns bytestrings if the string is ASCII (it returns
> unicode objects otherwise), while json returns unicode objects always.
>
> This was, unfortunately, a very unfortunate design decision on the part
> of simplejson - json is definitely correct here - and a very big change
> in semantics. It led to one very difficult to debug error for me already.
Right. And on Python 3, the json module (correctly) doesn't accept
byte-strings at all.
> So, this is a shout out to other people to watch out for this, and a
> call for ideas on what we can do to mitigate the impact of this. It's
> likely to crop up in all kinds of horrible places, deep in libraries
> that you can't do much about. In my case I was loading config, including
> passwords, from a config file in JSON, and the password was now
> exploding inside smtplib because it was a unicode object.
This is one place where there are limitations in the 2.x stdlib -
other places include cStringIO and cookies. For example, if you pass a
Unicode object to a cStringIO.StringIO, it doesn't complain, but does
the wrong thing:
>>> from cStringIO import StringIO; StringIO(u'abc').getvalue()
'a\x00b\x00c\x00'
>>>
Fun and games ...
I'm not sure there's any easy way out, other than comprehensive
testing.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: admin_changelist failing on Oracle
On Jun 2, 3:57 am, Russell Keith-Magee wrote: > > For auditing purposes, a ticket is always helpful, even if the ticket > is closed straight away. That way, anyone who hits the same problem > later on can see the source of the problem and the fact that a > solution should be coming in the near future. There was already a ticket for this - #17932 - which I've reopened, and added a pull request. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: admin_changelist failing on Oracle
On Jun 1, 6:39 pm, Ian Kelly wrote: > > The error is caused by one of the column names in the table. See the > second paragraph at: > > https://docs.djangoproject.com/en/1.4/ref/databases/#naming-issues > Thanks, that helped. It seems that in this case, the offending name is 'date'. I added a db_column='event_date' and all seems well. As this is a standard Django regression test, I presume a ticket should be raised? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
admin_changelist failing on Oracle
I'm getting an error when syncdb tries to execute the following SQL against Oracle 10g XE: CREATE OR REPLACE TRIGGER "ADMIN_CHANGELIST_EVENT_TR" BEFORE INSERT ON "ADMIN_CHANGELIST_EVENT" FOR EACH ROW WHEN (new."ID" IS NULL) BEGIN SELECT "ADMIN_CHANGELIST_EVENT_SQ".nextval INTO :new."ID" FROM dual; END; The error is django.db.utils.DatabaseError: ORA-06552: PL/SQL: Compilation unit analysis terminated ORA-06553: PLS-320: the declaration of the type of this expression is incomplete or malformed Can anyone suggest what the problem is? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - now available on GitHub
> The izip_longest definition was removed from itercompat in revision > b60b45a2a565 (which is fine, since it was only there for Python 2.5 > compatibility), but it means that the places that imported it need to > be changed to import it directly from py3 instead. Right, and I fixed that a couple of days ago. Anssi has kindly posted feedback to the GitHub issue tracker on my repo, so I'm following up there. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - now available on GitHub
On May 28, 8:24 pm, Anssi Kääriäinen wrote: > Is there any workaround for this? I need to install psycopg2 and cx- > oracle under python 3 to run the tests, and virtualenv is the only way > I know how to do this. However, this issue is causing a lot of noise > in the results, making testing the py3-branch somewhat hard. Just get a recent version of virtualenv.py and run that. For example, https://raw.github.com/pypa/virtualenv/1.7.1.2/virtualenv.py (I've not specifically tested this exact version, but IIRC I only had the problem with my distro's version, which was a little out of date). It's a self contained script, so just downloading it and running it should allow you to create a working virtual environment. Thanks for your input so far. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - now available on GitHub
> I found following error when trying to browse to the admin site of my django > app: > > File > ".../venv//lib/python3.2/site-packages/django/db/backends/postgresql_psycopg2/base.py", > line 8, in > from django.db import utils > ImportError: cannot import name utils Not sure why it isn't working for you. The import seems to work for me: vinay@eta-oneiric64:~/projects/django3/tests$ PYTHONPATH=.. DJANGO_SETTINGS_MODULE=test_sqlite python3.2 Python 3.2.2 (default, Sep 5 2011, 21:17:14) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from django.db import utils >>> Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - now available on GitHub
Anssi,
Thanks very much for the feedback.
> Both Oracle and MySQL fail to run because of this error (on both 2.7
> and 3.2):
> File "/home/akaariai/Programming/django/tests/django/db/backends/
> mysql/compiler.py", line 2, in
> from django.utils.itercompat import izip_longest
> ImportError: cannot import name izip_longest
I will investigate this.
> I get the following error when running the tests on any database on my
> setup (Ubuntu 12.04, Python 3.2):
> Traceback (most recent call last):
> File "/home/akaariai/Programming/django/tests/regressiontests/
> test_runner/tests.py", line 197, in
> test_option_name_and_value_separated
> self.assertNoOutput(err)
> File "/home/akaariai/Programming/django/tests/regressiontests/
> admin_scripts/tests.py", line 164, in assertNoOutput
> self.assertEqual(len(stream), 0, "Stream should be empty: actually
> contains '%s'" % stream)
> AssertionError: 335 != 0 : Stream should be empty: actually contains
> 'b'\'import warnings\' failed; traceback:\nTraceback (most recent call
> last):\n File "/home/akaariai/Programming/python3/lib/python3.2/
> warnings.py", line 6, in \n import linecache\n File "/home/
> akaariai/Programming/python3/lib/python3.2/linecache.py", line 10, in
> \n import tokenize\nImportError: No module named tokenize
> \n''
>
> I used this to install my setup:
> # sudo apt-get install python3 python3-dev virtualenv
> # virtualenv -p python3 --distribute python3
> # source python3/bin/activate
> # pip install psycopg2
> # run tests...
>
> Am I missing some dependency?
There are problems with some virtualenv versions, see this open
virtualenv issue:
https://github.com/pypa/virtualenv/issues/194
The problem with importing tokenize.py in your traceback implies you
might be running into this issue.
> PostgreSQL seems fine on 2.7 for the reduced test set ran
> (introspection transactions inspectdb fixtures queries extra_regress
> prefetch_related test_runner aggregation_regress). On 3.2 I get this
> additional error:
> Traceback (most recent call last):
> File "/home/akaariai/Programming/django/tests/django/test/utils.py",
> line 203, in inner
> return test_func(*args, **kwargs)
> File "/home/akaariai/Programming/django/tests/modeltests/
> prefetch_related/tests.py", line 378, in test_child_link_prefetch
> self.assertIn('authorwithage', connection.queries[-1]
> ['sql'].lower())
> File "/usr/lib/python3.2/unittest/case.py", line 889, in assertIn
> if member not in container:
> TypeError: Type str doesn't support the buffer API
I'll investigate this too.
I've enabled the issue tracker in the port's GitHub repo [1]
- feel free to add any further findings there.
Regards,
Vinay Sajip
[1] https://github.com/vsajip/django
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Python 3 port - now available on GitHub
The single codebase port of Django to Python 3 is now available on GitHub [1]. Recent core changes have been merged, and the test results are available at [2]. Summary: 2.7.2: Ran 4734 tests in 540.793s - OK (skipped=112, expected failures=3) 3.2.2: Ran 4688 tests in 524.807s - OK (skipped=120, expected failures=2, unexpected successes=1) Recent tests only cover the SQLite backend and were run on Linux. Help with testing other DB backends (and the GIS functionality) would be much appreciated! Regards, Vinay Sajip [1] https://github.com/vsajip/django/tree/django3 [2] https://gist.github.com/1373553 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GitHub migration done!
Hi Charles, > I setup a new mirror using hg-git: > > https://bitbucket.org/django/django-hg-git > > I work for Bitbucket and I'll make sure the mirror is maintained. Thanks very much for doing this. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GitHub migration done!
On May 1, 5:39 pm, Carl Meyer wrote: > It's just a cron job and a local repo using hgsubversion; every five > minutes it pulls the latest from Subversion and pushes it to Bitbucket. > The repo and cron job are hosted on the djangoproject.com server. If you > are able to get the conversion from git working and repeatable on a > local repo, I think Jacob can probably get you hooked up to host it on > djangoproject.com (which also has an ssh key with permission to push to > bitbucket.org/django/django). > > The tricky bit will be making the switch to sourcing from git in a way > that doesn't change all the historical commit hashes, making the new > mirror repo merge-incompatible with the current mirror (and all clones > of it). It's possible there will be no way to do that, in which case I > guess a new incompatible mirror is still better than no mirror at all. Okay, in case anyone cares, I've set up a local repo and have used hg- git to pull Django's Github repo to it, and pushed out to a Mercurial repo at https://bitbucket.org/vinay.sajip/django_mirror/ I've also set up a Git repo at https://bitbucket.org/vinay.sajip/django_reflected/ from the local Mercurial repo. On casual inspection, at least, all the hashes look the same between the official Django repo and the django_reflected repo. There were no errors reported; it's rather slow pushing to a Git repo, not sure why yet. I haven't set it up as a cron job, so it's not actually a mirror yet, but some manual pulls and pushes to the remote Mercurial repo worked fine. I'll wait to hear back from Atlassian to see if they've set anything up, before I do anything more. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GitHub migration done!
On May 1, 5:39 pm, Carl Meyer wrote: > It's just a cron job and a local repo using hgsubversion; every five > minutes it pulls the latest from Subversion and pushes it to Bitbucket. > The repo and cron job are hosted on the djangoproject.com server. If you > are able to get the conversion from git working and repeatable on a > local repo, I think Jacob can probably get you hooked up to host it on > djangoproject.com (which also has an ssh key with permission to push to > bitbucket.org/django/django). > > The tricky bit will be making the switch to sourcing from git in a way > that doesn't change all the historical commit hashes, making the new > mirror repo merge-incompatible with the current mirror (and all clones > of it). It's possible there will be no way to do that, in which case I > guess a new incompatible mirror is still better than no mirror at all. Okay, I'll take a look at it. In the meantime, I've pinged someone at Atlassian to see if they already have something/might be interested in hosting and maintaining a mirror (since BitBucket actually uses Django). I'll still try it out with a local repo to see how it works (as time permits). Thanks and regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GitHub migration done!
On May 1, 2:19 am, Carl Meyer wrote: > > Good point. I think which of those happens now depends on whether a > motivated someone steps up to figure out how to convert the mirror to > use hg-git and source from Git, and then maintain it as needed. I > originally did the current mirror, and it really hasn't needed any > maintenance over the past two years, but I no longer use Mercurial or > the Mercurial Django mirror, so that "motivated someone" is not likely > to be me this time around. I don't mind doing it, if it's sufficiently low-maintenance, and I do use Mercurial as well as Git. Did you implement it using a local repo and a cron job, or was there something else you used which was more purpose-built? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: GitHub migration done!
On Apr 28, 4:08 am, Adrian Holovaty wrote: > * We're going to keep the Subversion repository around indefinitely, > but it'll no longer be updated. That means that any mirrors using that repository as a source are also not worth using any more. There's the official BitBucket Mercurial mirror of Django: https://bitbucket.org/django/django/ Is that going to be changed to be a mirror of the GitHub repo, or will it disappear altogether? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Django use of stdlib HTMLParser "internals"
Although the Django tests all pass when run for the Python 3 port under Python 3.2, I hit some problems when testing against recent changes in the Python trunk (default branch, which will become 3.3). Looking into these problems, I found that Django subclasses HTMLParser and overrides parse_starttag, and in the overridden method uses an HTMLParser attribute, tagfind, which has recently changed. The change leads to the breakage. As tagfind isn't marked as private, it seemed possible that it was OK for Django to use it, so in the first instance I posted about it to python-dev: http://mail.python.org/pipermail/python-dev/2012-April/119074.html and raised a Python issue: http://bugs.python.org/issue14679 >From the discussion on python-dev, it seems possible that an optimal fix might require changes both in Python and in Django. Before creating a ticket about this for Django, I would like to get an opinion about this from the Django committers - so can someone please look at this python-dev thread and/or Python issue and comment? Thanks and regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Errors in tests
The errors seem to be related to Aymeric's change in r17894. If I
change
def _reset_dicts(self, value=None):
builtins = {'True': True, 'False': False, 'None': None}
if value:
builtins.update(value)
self.dicts = [builtins]
to the seemingly equivalent
def _reset_dicts(self, value=None):
value = copy(value or {})
value.update({'True': True, 'False': False, 'None': None})
self.dicts = [value]
then the errors no longer occur.
Ticket created: https://code.djangoproject.com/ticket/18103
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Errors in tests
Hi, I just upgraded to r17895, and now tests appear to have stopped
working. Here's the tail end of the test run:
==
ERROR: test_user_permission_performance
(regressiontests.admin_views.tests.UserAdminTest)
--
Traceback (most recent call last):
File "/home/vinay/projects/django/tests/regressiontests/admin_views/
tests.py", line 3244, in test_user_permission_performance
response = self.client.get('/test_admin/admin/auth/user/%s/' %
u.pk)
File "/home/vinay/projects/django/django/test/client.py", line 427,
in get
response = super(Client, self).get(path, data=data, **extra)
File "/home/vinay/projects/django/django/test/client.py", line 243,
in get
return self.request(**r)
File "/home/vinay/projects/django/django/core/handlers/base.py",
line 136, in get_response
response = response.render()
File "/home/vinay/projects/django/django/template/response.py", line
104, in render
self._set_content(self.rendered_content)
File "/home/vinay/projects/django/django/template/response.py", line
81, in rendered_content
content = template.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
140, in render
return self._render(context)
File "/home/vinay/projects/django/django/test/utils.py", line 60, in
instrumented_test_render
return self.nodelist.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
823, in render
bit = self.render_node(node, context)
File "/home/vinay/projects/django/django/template/base.py", line
837, in render_node
return node.render(context)
File "/home/vinay/projects/django/django/template/loader_tags.py",
line 123, in render
return compiled_parent._render(context)
File "/home/vinay/projects/django/django/test/utils.py", line 60, in
instrumented_test_render
return self.nodelist.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
823, in render
bit = self.render_node(node, context)
File "/home/vinay/projects/django/django/template/base.py", line
837, in render_node
return node.render(context)
File "/home/vinay/projects/django/django/template/loader_tags.py",
line 123, in render
return compiled_parent._render(context)
File "/home/vinay/projects/django/django/test/utils.py", line 60, in
instrumented_test_render
return self.nodelist.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
823, in render
bit = self.render_node(node, context)
File "/home/vinay/projects/django/django/template/base.py", line
837, in render_node
return node.render(context)
File "/home/vinay/projects/django/django/template/loader_tags.py",
line 62, in render
result = block.nodelist.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
823, in render
bit = self.render_node(node, context)
File "/home/vinay/projects/django/django/template/base.py", line
837, in render_node
return node.render(context)
File "/home/vinay/projects/django/django/template/base.py", line
1193, in render
'use_tz': context.use_tz,
File "/home/vinay/projects/django/django/template/context.py", line
96, in __init__
super(Context, self).__init__(dict_)
File "/home/vinay/projects/django/django/template/context.py", line
18, in __init__
self._reset_dicts(dict_)
File "/home/vinay/projects/django/django/template/context.py", line
23, in _reset_dicts
builtins.update(value)
ValueError: dictionary update sequence element #0 has length 1; 2 is
required
--
Ran 4698 tests in 956.978s
FAILED (errors=166, skipped=109, expected failures=2)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...
vinay@eta-oneiric64:~/projects/django/tests$
All the errors appear to have the same root cause. I'm testing with
Python 2.7.2+ (default, Oct 4 2011, 20:06:09) on Ubuntu Oneiric 64-
bit.
Can anyone shed any light on this? The value being passed to the
update method is an instance of
django.template.context.RequestContext. I can't see what I might be
doing wrong, so any help would be appreciated.
The tests were run using
PYTHONPATH=.. python runtests.py --settings test_sqlite
in the tests subdirectory.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Django and dictionary ordering
Is there any work going on with regard to making Django more resilient
in the face of changes to dictionary ordering? I know there has been
discussion about the hash collision issue last December, but I'm
unable to find a summary of the official Django position on this. Can
someone please point me to it? My Google-fu may have deserted me.
Following recent changes to the in-development Python 3.3, several
dozen Django tests are failing, apparently because of changes to
dictionary hashing and hence key order. Some examples:
==
FAIL: test_proxy_model_included
(regressiontests.fixtures_regress.tests.TestFixtures)
--
Traceback (most recent call last):
File "/Users/administrator/projects/django3/tests/regressiontests/
fixtures_regress/tests.py", line 363, in test_proxy_model_included
% widget.pk
AssertionError: '[{"model": "fixtures_regress.widget", "fields":
{"name": "grommet"}, "pk": 1}]' [truncated]... != '[{"pk": 1, "model":
"fixtures_regress.widget", "fields": {"name":
"grommet"}}]' [truncated]...
- [{"model": "fixtures_regress.widget", "fields": {"name": "grommet"},
"pk": 1}]
?
-
+ [{"pk": 1, "model": "fixtures_regress.widget", "fields": {"name":
"grommet"}}]
? +
==
FAIL: test_basic_processing_in_view
(regressiontests.forms.tests.forms.FormsTestCase)
--
Traceback (most recent call last):
File "/Users/administrator/projects/django3/tests/regressiontests/
forms/tests/forms.py", line 1527, in test_basic_processing_in_view
py3_prefix("VALID: {'username': %(_)s'adrian', 'password1': %
(_)s'secret', 'password2': %(_)s'secret'}"))
File "/Users/administrator/projects/django3/django/test/
testcases.py", line 429, in assertHTMLEqual
self.fail(self._formatMessage(msg, standardMsg))
AssertionError: VALID: {'username': 'adrian', 'password2': 'secret',
'password1': 'secret'} != VALID: {'username': 'adrian', 'password1':
'secret', 'password2': 'secret'}
- VALID: {'username': 'adrian', 'password2': 'secret', 'password1':
'secret'}
+ VALID: {'username': 'adrian', 'password1': 'secret', 'password2':
'secret'}
There are no corresponding failures with Python 3.2. The Django 3.x
branch is pretty up to date with trunk.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Python 3 port updated for 2.6 compatibility
As it seems likely that Python 2.5 compatibility will be dropped for
Django 1.5, I've updated the Python 3 port[1] to remove the hacks
needed for 2.5 compatibility. This means:
1. The u() and b() calls have been removed, and replaced with removing
the u prefixes and adding "from __future__ import unicode_literals" to
the relevant modules.
2. The use of sys.exc_info() has been removed in favour of using the
3.x exception syntax ("as e:").
There are some places where native strings are required, and for this
an "n" function has been added to django.utils.py3. This is a no-op on
Python 3.x but on Python 2.x it converts unicode to str. (Examples of
where native strings are needed - __repr__ should return native
strings, WSGI expects native strings, some cookie APIs expect native
strings and fail when passed Unicode in 2.x.)
Note that 2.x compatibility is now with Python 2.6.5 or later. Earlier
2.6 versions would not accept Unicode in kwargs keys; this is fixable,
but it's debatable whether we really need to - it would introduce
quite some noise.
The older version is still available in a branch named "2.5-3.x".
However, this does not reflect recent changes in Django trunk.
The default branch does reflect the latest Django trunk changes (up to
r17276).
Testing has been done on Linux; the default branch passes all tests
for Python 2.7.2 and 3.2.2 (and 3.3.0, the in-development version)
with the SQLite backend.
On the PostgreSQL backend, all tests pass on 2.7.2 and 3.2.2.
On the Oracle backend, one test fails on both 2.x and 3.x, but this
appears related to my Oracle NLS configuration.
If anyone has the chance to try out the latest changes, I'd be
grateful for some feedback. I haven't been able to set up working GIS
configurations, so the GIS backends remain untested.
Regards,
Vinay Sajip
[1] https://bitbucket.org/vinay.sajip/django/
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Proposal: drop Python 2.5 support in Django 1.5
On Dec 23, 3:23 pm, Jannis Leidel wrote: > > I would argue that Jaunty being a non-LTS release shouldn't be considered as > a target platform we want to support. OTOH the LTS release Ubuntu Lucid > (10.04) ships Python 2.6.5, so we're good. Sure, Jaunty was just an example. I'm not sure what versions of 2.6 are on different releases of RHEL, CentOS etc. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Proposal: drop Python 2.5 support in Django 1.5
On Dec 10, 4:56 pm, Adrian Holovaty wrote: > I think both of these proposals are great -- start merging the Python > 3 work right after we release 1.4, anddropsupport for Python2.5in > trunk after 1.4 is released. Before we do this, another decision is required - which release of 2.6 is the minimum Django should support? There was a change which occurred with 2.6.5 (IIRC) to allow Unicode in kwargs keys (earlier versions would raise an exception). If we use "from __future__ import unicode_literals" to avoid using u() and b(), then a lot of Django code will be affected. Supporting 2.6- >2.6.4 will require cleaning kwargs, and to avoid this one would need to state that Django will only support 2.6.5 onwards. Of course, that might affect quite a lot of users with 2.6 system Pythons which are < 2.6.5 (e.g. on Ubuntu Jaunty the system Python is 2.6.2). Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Django 1.4 alpha on December 22nd
On Dec 16, 12:45 am, Ian Kelly wrote: > The problem with merging it and labeling the support as "experimental" > is that the changes are of such a fundamental nature that they could > easily break things in 2.x, not just in Python 3. Agreed, it's too risky to merge in 1.4 this late in the day. As well as passing the full suite, it would need to be tested in more real- world scenarios (including with third-party apps) to be sure there were no regressions in 2.x. On the bright side, it does allow time for simplifying to allow "except X as e:", and removal of u() and b() from the port, as well as more extensive testing on both 2.x and 3.x. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all Oracle tests (bar one) now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 11, 6:02 pm, Aymeric Augustin wrote: > > There are also the GIS database backends in django.contrib.gis.db.backends: > postgis, spatialite, mysql, oracle. > Of course - well, I wouldn't want to have nothing to do ;-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python 3 port - all Oracle tests (bar one) now pass on 2.7.2 and 3.2.2 with the same codebase
I have all tests except one passing with the Oracle backend on 2.7.2
and 3.2.2 with the same codebase. The one failure I get (the same on
2.7 and 3.2) appears to be related to my Oracle configuration - I
haven't tracked it down yet.
Tests were run on Linux Mint 12 32-bit, using Oracle 10g XE (Express
Edition) using cx_Oracle 5.1.1 installed via pip into virtual envs for
2.7 and 3.2. Results:
Python 2.7
FAILED (failures=1, skipped=76, expected failures=3)
==
FAIL: test_unicode_data (modeltests.basic.tests.ModelTest)
--
Traceback (most recent call last):
File "/home/vinay/projects/django3/tests/modeltests/basic/tests.py",
line 496, in test_unicode_data
u('\u6797\u539f \u3081\u3050\u307f'))
AssertionError: u'\xbf\xbf \xbf\xbf\xbf' != u'\u6797\u539f
\u3081\u3050\u307f'
- \xbf\xbf \xbf\xbf\xbf
+ \u6797\u539f \u3081\u3050\u307f
Python 3.2
FAILED (failures=1, skipped=82, expected failures=2, unexpected
successes=1)
==
FAIL: test_unicode_data (modeltests.basic.tests.ModelTest)
--
Traceback (most recent call last):
File "/home/vinay/projects/django3/tests/modeltests/basic/tests.py",
line 496, in test_unicode_data
u('\u6797\u539f \u3081\u3050\u307f'))
AssertionError: '¿¿ ¿¿¿' != '林原 めぐみ'
- ¿¿ ¿¿¿
+ 林原 めぐみ
Any pointers as to the cause of the remaining failure (which is an old
problem[1]) would be appreciated.
So we now have good results on SQLite, PostgreSQL, MySQL and Oracle,
which I believe is all of the backends which ship with Django - have I
missed any?
Regards,
Vinay Sajip
[1]
http://groups.google.com/group/django-developers/browse_thread/thread/15b710ba6912998d/63e9559398a13660
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all PostgreSQL tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 10, 8:20 am, Aymeric Augustin wrote: > Yes, I documented the procedure > here:https://code.djangoproject.com/wiki/OracleTestSetup > where I set up the CI server. Thanks for the pointer. Because of the need to have a specific amount of swap space, I ended up installing Oracle 10g XE on a 32-bit server (Linux Mint 12). No problems were encountered, until I ran the basic test to ensure that everything is working (python 2.7, cx_Oracle 5.1.1 installed vi PyPI): FAIL: test_unicode_data (modeltests.basic.tests.ModelTest) -- Traceback (most recent call last): File "/home/vinay/projects/django3/tests/modeltests/basic/tests.py", line 496, in test_unicode_data self.assertEqual(Article.objects.get(pk=a.id).headline, AssertionError: u'\xbf\xbf \xbf\xbf\xbf' != u'\u6797\u539f \u3081\u3050\u307f' - \xbf\xbf \xbf\xbf\xbf + \u6797\u539f \u3081\u3050\u307f I realise this is an old error (I found a post from Dec 2009 about it[1], but no reason was mentioned there, nor how to get around it). I'm running with a UK locale, and after following the procedure in the Wiki, if I look in the web interface for the NLS configuration, I get: ... NLS_CHARACTERSETWE8MSWIN1252 ... NLS_NCHAR_CHARACTERSET AL16UTF16 ... As this is a Linux installation I've no idea why the 1252 character set is configured, but in any case, since NLS_LANG is set to .UTF8 by the backend, I assumed it wouldn't matter, but apparently it does :-( Can anyone enlighten me as to what to do about this? Thanks & regards, Vinay Sajip [1] http://groups.google.com/group/django-developers/browse_thread/thread/15b710ba6912998d/63e9559398a13660 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - notes for people wanting to review changes/port apps
On Dec 10, 8:49 pm, Jacob Kaplan-Moss wrote: > What do you think about using lib2to3 to write some custom > fixers to do some of this grunt work automatically? Might help people > port their Django sites/apps a bit easier, right? On the face of it, I'd say it's doable, and perhaps worth it - it would be part of a generic toolset for single-code-base porting, and not especially Django-specific. But it won't make things automatic, there will probably always be *some* manual intervention required (unless a *lot* of time was spent putting intelligence into the fixers - that might be "how long is a piece of string?" territory). But yes, it would remove a fair bit of the grunt work. Of course, you need to be sure that a single code base strategy makes sense for you. Reading some of Chris McDonough's posts on reddit and elsewhere, it looks as if Pyramid is using the same approach. Perhaps the time is ripe to develop some special 2to2plus3 fixers ... Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python 3 port - notes for people wanting to review changes/port apps
I've created a page on the Django Wiki indicating what guidelines I used when porting Django from 2.x to a single codebase for 2.x/3.x, and which can also be followed by people wanting to port apps across on the same basis. The first cut is at https://code.djangoproject.com/wiki/PortingNotesFor2To3 and I would welcome your comments - I'm sure I've missed a thing or two. I will update the page as things occur to me or in response to feedback about it. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 10, 4:47 pm, Luke Plant wrote: > > 1.4 is never going to happen. We are hoping to release a 1.4 alpha very > soon, merging this work would be a major mistake at this point. > > The patch requires lots of changes to the way things work. Many are > small, but they impose some mental overhead (like using b() etc). We > need *all* the core developers to get fully up to speed with these > before we commit, otherwise we will introduce tons of bugs as we try to > get 1.4 out of the door, even if all the tests pass at the moment. > > The timing is very good for 1.5, however, so hopefully we can merge soon > after the release of 1.4. Even then, I think 1.5 would be a "Python 3 > preview" i.e. we wouldn't promise the same level of support for Python 3 > as for Python 2.X. Sounds like a plan :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 10, 12:40 pm, Gert Van Gool wrote: > What I would love as a developer in the documentation is, what do I > need to do so my code "could" run on Python 3. > It might not work, but most obvious issues (like b() and u()) would be > out of the way. I know it's not the official documentation, but you might find my notes on the Wiki helpful: https://code.djangoproject.com/wiki/PortingNotesFor2To3 Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python 3 port - all PostgreSQL tests now pass on 2.7.2 and 3.2.2 with the same codebase
I now have Django passing the test suite with the PostgreSQL backend,
on Python 2.7.2 and Python 3.2.2. The tests were run on Ubuntu Oneiric
64-bit in a VM.
Python 3:
Python 3.2.2 from the Ubuntu package repository.
psycopg2-2.4.2 from http://initd.org/psycopg/
Python 2:
Python 2.7.2 - Oneiric's system Python.
psycopg2-2.4.2 from the Ubuntu package repository.
Results:
Python 2.x:
Ran 4478 tests in 1265.949s
OK (skipped=69, expected failures=3)
Python 3.x:
Ran 4432 tests in 1268.253s
OK (skipped=75, expected failures=2, unexpected successes=1)
The 3.x test has fewer tests because PIL is not installed in that
environment. With PIL installed, the results are
Python 3.x:
Ran 4477 tests in 1383.850s
FAILED (failures=1, errors=1, skipped=71, expected failures=2,
unexpected successes=1)
The failure/error are caused by bugs in the PIL port on Python 3
(git://github.com/grahame/pil-py3k.git). It fails to recognise
valid .png files:
==
ERROR: test_multiple_calls
(regressiontests.file_storage.tests.InconsistentGetImageDimensionsBug)
--
Traceback (most recent call last):
File "/home/vinay/projects/django3/tests/regressiontests/
file_storage/tests.py", line 538, in test_multiple_calls
image_pil = Image.open(img_path)
File "/home/vinay/.virtualenvs/py32/lib/python3.2/site-packages/PIL/
Image.py", line 1936, in open
raise IOError("cannot identify image file")
IOError: cannot identify image file
==
FAIL: test_image_field (modeltests.model_forms.tests.OldFormForXTests)
--
Traceback (most recent call last):
File "/home/vinay/projects/django3/tests/modeltests/model_forms/
tests.py", line 1253, in test_image_field
self.assertEqual(f.is_valid(), True)
AssertionError: False != True
If anyone can reproduce these results, that would be great.
Additional data point: On earlier runs, I was experiencing really long
test run times. I found I was using the same database for 'default'
and 'other' in the configuration. I got some errors in the
multiple_database tests because of this, but had to be really patient,
because the tests in general ran so slowly. I got no warning about
this, just very poor performance. My working test_psycopg2.py settings
file is:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'djangotest',
'USER': 'djangotest',
'PASSWORD': 'djangotest',
'HOST': 'eta-karmic',
},
'other': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'djangotest',
'USER': 'djangotest',
'PASSWORD': 'djangotest',
}
}
Stupidly, I had left the 'HOST' line out of the above configuration,
causing both 'default' and 'other' to point to 'localhost'. Once this
was rectified, the test times seem a lot more reasonable. So it was
nothing to do with runtime overheads like calling u() etc.
So - with Ian Clelland's post[1] from a couple of days ago, we have
successful runs on 2.x and 3.x using a single codebase and SQLite,
PostgreSQL and MySQL backends. I suppose Oracle will be the next one
to focus on: Ian Kelly was getting 17 failures and 24 errors on 7
Dec[2], but some changes have been committed since then.
Is Oracle XE (Express Edition) a valid platform for Django's Oracle
tests? Does anyone here have experience installing it on Ubuntu 64-
bit? I have an Oracle forum post[3] which gives the installation
procedure, but it appears somewhat involved ...
Regards,
Vinay Sajip
[1] http://groups.google.com/group/django-developers/t/ad4e980c8cde49f
[2] http://groups.google.com/group/django-developers/msg/c113d3df5e836b7e
[3] https://forums.oracle.com/forums/thread.jspa/threadId=2301639
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all MySQL tests now pass on 2.6.7 and 3.2.2 with the same codebase
On Dec 9, 4:36 pm, Tom Evans wrote: > I know you put the word 'unscientifically' in there, but you can draw > no conclusions from doing one run of each like that. See my reply > earlier in the week on how to simply and easily do valid statistical > testing. > > http://osdir.com/ml/django-developers/2011-12/msg00162.html > Hi Tom, You're right, and you did offer to run some benchmarks if you were pointed to some code. The Django changeset that Ian pointed to (https://bitbucket.org/vinay.sajip/django/changeset/6b1413a9901a) should be runnable without needing to set up a database backend like MySQL or PostgreSQL, is there any chance you could run those benchmarks (on the regression test runs under 2.x and 3.x) and post results? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all MySQL tests now pass on 2.6.7 and 3.2.2 with the same codebase
On Dec 8, 11:39 pm, Ian Clelland wrote: > I now have Django passing its entire unit test suite with the MySQL and > SQLite backends, on Python 2.6.7 and Python 3.2.2 Ian, Thanks for the comprehensive summary and eliminating those last few issues on the MySQL backend. One more thing which might be useful to have for comparison purposes is the test results when run on 2.x with an unmodified Django (from the mirror at https://bitbucket.org/django/django/), so that people can see what the potential performance penalty is for calling u() and b() all over the place. I realise that it's not going to be a scientific comparison, but it will probably show up if there is appreciable overhead (I haven't found that to be the case, but it might vary based on platforms and backends). I posted these from the tests (for the sqlite backend) run on my Ubuntu VM, in response to a question from Armin Ronacher: Django on 2.x unported: 4482 tests in 368.972s, skipped = 90 Django on 2.x, with u(), b(), sys.exc_info() etc: 4478 tests in 367.635s, skipped = 90 Django on 3.x, with u(), b(), sys.exc_info() etc: 4423 tests in 372.946s, skipped = 96 It would be interesting to see if a similar pattern emerges on OSX. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 8, 6:46 am, Ian Clelland wrote: > > There also seem to be a number of unicode-related errors there (mixed > collations; unrecognized characters) -- Could that be related to the lines I'm also getting the same errors on 2.x with the 2.x adapter on unpatched Django - so it might well be to do with my MySQL installation (stock 5.1.58 for Ubuntu 64-bit). It might be worth trying with my latest Django repo and my patched MySQLdb on your installation, if you have the time and inclination... BTW are you on IRC? If so, what nickname do you go by? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 8, 6:46 am, Ian Clelland wrote: > > use_unicode used to be unconditional on the database connection; now you've > removed it, but just for Python 3. > I looked into it a little further - the use_unicode flag is just used in the 2.x constructor to set default converters to return unicode instead of str. This would be implicitly always unicode in the 3.x adapter. I also looked further into one of the collation errors - test_serialize_unicode. From what I can see, the query is converted to utf-8 bytes and passed to MySQL, which returns the error. I can't see where the Swedish collation it refers to comes from. Will continue to investigate ... Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 8, 6:46 am, Ian Clelland wrote: > There also seem to be a number of unicode-related errors there (mixed > collations; unrecognized characters) -- Could that be related to the lines > in your patched mysql/base.py: > > if not PY3: > kwargs['use_unicode'] = True > > use_unicode used to be unconditional on the database connection; now you've > removed it, but just for Python 3. I removed it because the Python 3 adapter doesn't recognise the kwarg; I thought it might be that under Python 3 it would be implicitly true. Possibly I need to check the detail of where that value is used in the 2.x adapter and see what the 3.x adapter does in that area. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 7, 8:38 pm, Ian Clelland wrote: > Thoughts? A further update: of my 24 failures, 17 are in multiple_database, as are 3 failures. But I get the same errors with unpatched Django 2.x running with the standard 2.x MySQLdb adapter, so it might be something to do with my MySQL configuration. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 7, 8:38 pm, Ian Clelland wrote: > Thoughts? The Django test run using MySQLdb at https://github.com/vsajip/MySQL-for-Python-3 completed with 24 failures, 9 errors, so your PyMySQL results are better at 5 failures, 2 errors. I will dig in a little further into these issues. Details at https://gist.github.com/1446201 - total run time was 9737 secs, BTW, but that is quite possibly down to differences between our machines. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Adding Support for PyMySQL (for Python 3)
On Dec 7, 8:38 pm, Ian Clelland wrote: > > 3. Code speaks, I know -- what's the best way to share this? I've sent > Vinay a patch, but that's for his Py3k branch, and might not get the > audience that something like this needs. I'm doing it for Py3k support, but > ideally it would work just as well under Python 2.x. Should I set up a > public repository somewhere, or just open a ticket and attach patches? > > Thoughts? Flames? Ian, I have a clone of Dāvis' repo which supports MySQLdb on Python 3: https://github.com/vsajip/MySQL-for-Python-3 (forked from https://github.com/davispuh/MySQL-for-Python-3) I mentioned in a message to you that I couldn't use it because it used str.format rather than % formatting, but actually that was easy enough to patch. I didn't have to make the Django changes called for in your patch, and the test suite is running now with the MySQLdb backend. I got some warnings for utf-8 bytestrings as fixtures were loaded, but it's run the first 275 or so tests without errors or failures so I'm going to let it run to see how far it gets. I will push my changes to the above repo soon. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 7, 3:50 pm, Ian Clelland wrote: > I'm using PyMySQL (https://github.com/petehunt/PyMySQL) -- it's almost a > drop-in replacement for MySQLdb. The biggest change is in the data > converters -- they have a different function signature from MySQLdb's. > > I have had to make a couple of patches to the library, which I hope to get > applied upstream, otherwise it's working pretty well, so far. By "library" I assume you're referring to PyMySQL. Is there a repo I can clone, so I can start testing with MySQL? > I'm down to four failures and one error now -- I'll post back once those > are done, with a set of patches, hopefully. Great - thanks! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 7, 5:53 am, Ian Clelland wrote:
> On Tuesday, December 6, 2011, Ian Kelly wrote:
> > itertools.izip_longest was added in Python 2.6, though, so a
> > compatibility function is needed for Python 2.5.
>
> Ahh, that's a good catch. I'll have to make use of your solution, then.
>
I've already pulled Ian Kelly's changes from his BitBucket repo. Which
MySQL driver are you using for Python 3? I started looking at
git://github.com/davispuh/MySQL-for-Python-3.git
but it appears to expect queries formatted with {} rather than %s, so
it doesn't seem that I can use it.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 7, 5:14 am, "Joe & Anne Tennies" wrote: > So, I hg pull; hg update and reran the Python3 tests. The Egg import test > passes now. > > The attached patch fix makes the last 2 tests pass in both 2.7.2 and 3.2.1 > under postgres and sqlite. I'll be honest and say I don't know why, so > please figure out why and explain it to me. I think it's because psycopg2 > won't run upper on something that's not a unicode in python 3, but it needs > to be a bytestring later to have the "buffer interface" implemented. I think that part about "upper" is probably right - it doesn't make sense to run "upper" on anything other than text. Part of it is just bugs in my conversion - if you look at some of the other test methods in the same test case, it appears that I applied b() in some places when I shouldn't have. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 6, 11:29 pm, Ian Clelland wrote: > > Without doing anything else, I installed pymysql, and made a few minimal > modifications to use that module in the mysql backend, and fixed one > Python3 incompatibility issue in db/backends/mysql/compiler.py. which was? > First initial test run indicates that there's still some work to do :) > > Ran 4429 tests in 18128.079s > FAILED (failures=11, errors=375, skipped=114, expected failures=2, > unexpected successes=1) > > (yeah, that's 18k seconds, just a bit over 5 hours, but mysql has never > been fast on OS X; something about the way it rolls back the database > between tests) I presume you've tried the advice given here: http://www.stereoplex.com/blog/speeding-up-django-unit-test-runs-with-mysql Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 6, 3:31 pm, "Joe & Anne Tennies" wrote: > PS: I too have a slow postgres even after I upped the shared_buffers and > effective_cache_size while disabling fsync, and it is still pretty darn > slow. (info from:http://www.revsys.com/writings/postgresql-performance.html) > I'm testing w/ trunk to get a baseline w/ psycopg2. I'll admit I'm using a > slow HDD, but I waited well over an hour. > > When I get back from work, I'll try psycopg2 and MySQL again tonight. Thanks for that, and for the useful feedback on the sqlite runs, too. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 6, 4:18 am, Ian Clelland wrote: > If I replace line 25 with this block: > > resource_name = resource_string(app, pkg_name) > if hasattr(resource_name, 'decode'): > resource_name = resource_name.decode(settings.FILE_CHARSET) > return (resource_name, 'egg:%s:%s' % (app, pkg_name)) > > then all (expected) tests pass. Thanks very much for this fix, I have applied it and tested in a venv, and can confirm your findings. I have pushed this to BitBucket. I have installed a PIL Python 3 port and bugs in that are causing 1 failure and 1 error; everything else passes. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 6, 12:45 am, Ian Clelland wrote: > After a lot of troubleshooting, and fun installing Python3.2 under > virtualenv (Vinay, I think I ran into the same issue as you > here:https://github.com/pypa/virtualenv/issues/194), I almost have the > complete > test suite running under MacPorts Python 3.2.2. That's good news about the state of the tests on MacPorts 3.2.2. I hope Carl reads your mail about the virtualenv issue :-) > While installing Vinay's port (cloned freshly this morning from Bitbucket), > I had to edit four files, in order to get them to run under Python 3: [snip] > Did I miss a step here, or not see an error which should have been > corrected earlier? Nobody else has commented about Python3 syntax errors, > so I'm thinking it's my process. No, I got those today, too - but not on earlier days, for many days. I suspect that it's something to do with merging upstream changes into my port, but I haven't pinned it down. I pushed my changes this afternoon, so you could try pulling my changes and re-testing. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 7:08 pm, Alex Gaynor wrote: > It depends *why* it's slow. If it's slow as a consequence of all the bytes > calls and extra encoding/decoding, yeah I'd say that's a blocker, because > it'll never improve otherwise. On the other hand if it's slow because > psycopg happens to be slow on py3k, well, that's someone else's problem > then and I'm ok to merge it. Further to my earlier post, I do now think it's my PostgreSQL configuration. I aborted my test of unported Django running with psycopg2 - it had run 1223 tests in 4871.121 seconds, which would mean an average of 4 seconds per test! The full suite of 4475 or so tests would then take almost five hours! I will try to test with psycopg2 on another machine, but that might not be possible for a while, so I'll have to rely on the good offices of others like Anssi for some more timely results. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 7:08 pm, Alex Gaynor wrote: > It depends *why* it's slow. If it's slow as a consequence of all the bytes > calls and extra encoding/decoding, yeah I'd say that's a blocker, because > it'll never improve otherwise. On the other hand if it's slow because > psycopg happens to be slow on py3k, well, that's someone else's problem > then and I'm ok to merge it. I've just started a run on unported Django with my postgresql configuration. No hard data yet, but it seems just as slow as the dots get printed to the screen. I guess that might be good news for the port, but we'll wait and see what the final time is ... just to give an idea, I re-ran the test on the port, but without using RAMdisk for the data (but still with fsync = off), that took around 3h 15 min ... I've only got a GB of RAM, running in a VM, and have done no tuning other than fsync = off, so data from other testers might be more representative than my own. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 3:15 pm, Carl Meyer wrote: > That sounds orders of magnitude slower than running the tests under > Postgres here - and I haven't done any of the pg-config tweaking others > have mentioned (not even turning fsync off). Which leads me to think > that either there is some particularly unusual Postgres slowness on your > machine, or there's a major performance regression with the 3.x port and > Postgres (which, if it's that significant, would block merging the port). Or maybe I was just frustrated at the slowness of the tests - they finished in under 3 hours. Given that the sqlite tests don't feel slow, I assumed the problem was in the postgresql configuration. So I have removed postgresql 9.1 (in case that was a factor), installed 8.4, and will re-run the tests. As you suggested, it's a good idea to run the tests on the trunk to confirm whether it's a postgres configuration issue. Another thing that's different, of course, is the psycopg2 driver for 3.x - I'm not sure if it would be expected to contribute to the problem. I can of course run under 2.x, with the standard psycopg2 driver installed. Anyway, I don't think we should worry too much about the performance just yet, especially as Anssi is not apparently seeing the same kind of slowdown. Others can confirm or refute whether this slowdown is seen in their environments. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 12:14 pm, Anssi Kääriäinen wrote: > As for test running speed: how fast is it for you? For me, using an old > laptop, the speed is around 20-30 minutes (can't test for accurate speed > right now). So, if it is much slower, you have something strange in your > setup. Well, the test has now completed: Ran 4474 tests in 8869.701s FAILED (failures=18, errors=12, skipped=71, expected failures=2, unexpected successes=1) Not too bad for a first complete pass. Over two and a half hours to run, and that' s running from /dev/shm (IIUC effectively RAMdisk) and with fsync = off. All else set to default values. Time to investigate Jeremy's link to Frank Wiles' post ... Tests are running in a (64-bit) VM with 1GB RAM on a machine with 8GB physical RAM. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 12:14 pm, Anssi Kääriäinen wrote: > On 12/05/2011 01:41 PM, Vinay Sajip wrote: > > The test_ticket10432 should not generate any query at all. The tests in > django/db/models/sql/where.py should ensure that in the case of empty > __in condition the query will not be executed at all (or that part of > the where condition is removed if it is possible for the query to return > rows if the __in=() executes to False). > > The problem is that the test in where.py: > if hasattr(value, '__iter__') and hasattr(value, 'next') > isn't correct. It seems that in python3 the 'next' has been renamed to > '__next__' and thus the generator will not be consumed. My python3 > knowledge is basically nonexistent, so it might be I am wrong here. Aargh, yes. You mentioned this earlier, but I missed correcting this when I did the other fixes. I've dealt with it now [basically, py3 defines 'next_name' which is 'next' on 2.x and '__next__' on 3.x - I just changed it to hasattr(value, next_name)]. > As for test running speed: how fast is it for you? For me, using an old > laptop, the speed is around 20-30 minutes (can't test for accurate speed > right now). So, if it is much slower, you have something strange in your > setup. I haven't yet run it to completion, but have just now set it up to use /dev/shm as per Florian's suggestion (as well as having fsync = off). Will report the total time once the tests have all run. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 11:30 am, Florian Apolloner wrote: > > Use a tablespace which lies in a ram disk, you can't get any faster than > that I/O wise ;) I've seen posts telling how to do this, but I had hoped to avoid the need for it ... Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 12:57 am, Anssi Kääriäinen wrote: > > To speed up things, here are the tests that failed for me after the > fixes posted upthread: > - cache > - select_for_update > - queries > - admin_views > I've eliminated many of the errors here, but a few remain, at least some of which are related to the psycopg2 driver under Python 3 [e.g. test_ticket10432 leads to a SQL query apparently legitimately containing " IN ()", which the backend rejects as a syntax error]. I'm running the full suite now, but I don't expect it to finish before the end of the day :-( Regards, Vinay -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 1:16 am, Jeremy Dunck wrote: > > See also:http://www.revsys.com/writings/postgresql-performance.html > Thanks, I'll dig into that. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 12:57 am, Anssi Kääriäinen wrote: > There is one easy thing you can do for testing, set fsync to off in > postgresql.conf. The file is probably at /etc/postgresql/9.1/main/ > postgresql.conf. Then restart the server and tests should be way > faster. Note that if your machine crashes, you will likely lose all > data. But as this is testing server that should not matter. I've already done the fsync = off, but it didn't make that much difference :-( > To speed up things, here are the tests that failed for me after the > fixes posted upthread: > - cache > - select_for_update > - queries > - admin_views > > Although I ran out of patience, there might be more still. Well, thanks for that feedback. I'll run these selectively and see what's thrown up. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 5, 12:09 am, Anssi Kääriäinen wrote: > > Now for the next problems: > Okay, thanks - I'll look at these. By the way, I tried to get a PostgreSQL server set up on my system (9.1, as that's the default for Ubuntu Oneiric) and started running the tests, but they are running very very slowly. I've got a very simple settings.py that basically just has the minimum database connectivity settings (the same for 'default' and 'other', just the database name, username and password all set to 'djangotest'). Are there any other settings I need to set for a reasonable performance? I assumed the tests would use transactions by default, but I'm not sure if that's correct. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
On Dec 4, 4:32 pm, Anssi Kääriäinen wrote: > I am currently running the test suite using psycopg2 (2.4.0 onwards > support Python 3) and Python 3.2. I can report that there are at least > three things needing fixing: Anssi, thanks for posting this feedback. > A mistake in django/db/backends/util.py L154: > - hsh = hashlib.md5(name).hexdigest()[:hash_len] > + hsh = hashlib.md5(bytes(name, encoding="UTF8")).hexdigest() > [:hash_len] > > The fix is probably not correct. Try just b(name) instead of bytes(name, ...) and see if that works. > > And then as last thing: > File "/home/akaj/Django3/django/tests/django/db/backends/ > postgresql_psycopg2/base.py", line 57, in execute > reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e)), > sys.exc_info()[2]) > TypeError: 'IntegrityError' object is not iterable > > I don't know what to do to that. Replace *tuple(e) with *e.args (5 occurrences) and see how that works. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python 3 port - all tests now pass on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 with the same codebase
Progress update on the Python 3 port[1] - all tests now pass[2] on Python 2.5.4, 2.6.2, 2.7.2 and 3.2.2. Python 2.5.4 = Ran 4490 tests in 513.699s OK (skipped=91, expected failures=3) Python 2.6.2 = Ran 4490 tests in 455.615s OK (skipped=89, expected failures=3) Python 2.7.2 = Ran 4475 tests in 379.923s OK (skipped=90, expected failures=3) Python 3.2.2 = Ran 4420 tests in 389.154s OK (skipped=96, expected failures=2, unexpected successes=1) The tested code incorporates the latest Django SVN trunk changes (r17168) as well as Luke Plant's patch[3] to only get an exception object when actually needed (I missed a couple of cases). Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/django/ [2] https://gist.github.com/1412432 [3] http://groups.google.com/group/django-developers/msg/3f2dd22295ff1555 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 3, 9:18 pm, Luke Plant wrote: > > I did some checks in the py3k patch of all the cases where we actually > do this because we need the exception object. I found the following: > Thanks for the analysis and suggested patch. I've implemented this patch in my branch, and the tests are running now on 2.5.4, 2.6.2, 2.7.2 and 3.2.2 :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 3, 8:50 pm, Karen Tracey wrote: > What about Pythons 2.5 and 2.6? Okay, I've now done some more testing, and with some minor changes, I can report that good results were obtained on Pythons 2.5.4 and 2.6.2. The remaining failures are representational ones - u'foo' vs. 'foo' and order of dictionary keys in doctest output. Summaries follow. Python 2.5.4 == Ran 4490 tests in 481.852s FAILED (failures=1, skipped=91, expected failures=3) Python 2.6.2 == Ran 4490 tests in 449.179s OK (skipped=89, expected failures=3) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 3, 8:50 pm, Karen Tracey wrote: > What about Pythons 2.5 and 2.6? I did a bit more checking, and a bit more work will be required on 2.5 and 2.6: for example, on 2.5, parse_qsl needs to come from elsewhere, and on 2.6, unittest.skipIf needs to come from somewhere else. So some changes need to be made to django/utils/py3.py to accommodate this - it shouldn't be too much work, hopefully :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 3, 8:50 pm, Karen Tracey wrote: > On Fri, Dec 2, 2011 at 10:55 AM, Vinay Sajip wrote: > > > The Python 3 port now has all tests passing on 2.7.2 and 3.2.2 with > > the same codebase: > > What about Pythons 2.5 and 2.6? I've done nothing intentionally that prevents the port from working on 2.6 and 2.7 - it's just that my test machine happens to be a recent Ubuntu variant that doesn't have them installed. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 2, 9:29 pm, Ian Kelly wrote: > I'm already planning to go through the Oracle tests this weekend to > get them ready for 1.4, so I'll run this through as well and let you > know how it goes. Right now I expect failures even in 2.7. Great, I look forward to the feedback. I can't do any testing at the moment with an Oracle backend, unfortunately - but I can certainly look at test failures under 3.x which succeed under 2.x to try and eyeball fixes to problems. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 2, 7:32 pm, Tomek Paczkowski wrote: > Most important: where to send all those beer pints? Heh. I do drink beer, but my liver would probably prefer it if someone were to send me something from my Amazon wishlist, which is on http://www.amazon.co.uk/registry/wishlist/2CJGJ3I4HEIK5 But don't all rush at once ;-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 2, 6:34 pm, "Joe & Anne Tennies" wrote: > So, last time I saw, you had only run against sqlite. Do you need help > testing it against postgres, MySQL, and oracle (perhaps some unofficial > ones)? What about all the caching backends? Do we need tests Python 3 > equivalent tests for the ones that were skipped due to being Python 2-isms? > What about some sort of "porting guide" or other needed documentation? > > Basically, what's left, besides getting it merged into the official trunk > that people can help you with? Yes, you can help with one or all of the above items. As far as I know, the additional tests that are skipped are because of dependencies on PIL and setuptools (neither of which I have installed to run with Python 3: and there are some PIL ports for Python3, plus one can use distribute in place of setuptools. This is the area I am working on currently - working in a virtualenv with distribute and a PIL port installed. You can certainly try helping with a PostgreSQL backend, I believe py- postgresql can be used as a PostgreSQL driver under Python 3. Ian Kelly mentioned on this thread that he'll be looking at the Oracle backend. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
On Dec 2, 6:03 pm, Jacob Kaplan-Moss wrote: > > I'll try to spend the weekend writing/porting an app or two. If I can, > I'll let you know how it goes. That would be very useful feedback, thanks. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python 3 port - all tests now pass on 2.7.2 and 3.2.2 with the same codebase
The Python 3 port now has all tests passing on 2.7.2 and 3.2.2 with the same codebase: Python 2.7.2 = Ran 4475 tests in 373.875s OK (skipped=90, expected failures=3) Python 3.2.2 = Ran 4420 tests in 364.044s OK (skipped=97, expected failures=2, unexpected successes=1) This incorporates the very latest changes in Django SVN trunk (r17165). Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Python3 port now applied to default branch, with promising progress
As there seemed to be some hesitation about how best to proceed with my recent work on the features/py3k branch, I've merged the features/ py3k branch into the default branch of my repo at https://bitbucket.org/vinay.sajip/django/ and fixed most of the resulting issues. Current summary results are: Python 2.7.2 = Ran 4475 tests in 386.314s OK (skipped=90, expected failures=3) Python 3.2.2 = Ran 4421 tests in 362.374s FAILED (failures=3, errors=3, skipped=97, expected failures=2, unexpected successes=1) Summary results for this branch are at https://gist.github.com/1412432 I will be doing further work in the default branch from now on. I have merged into the version of the upstream repo as it was today, i.e. it's pretty up to date. The failures are more or less the same as were occurring on the features/py3k branch. Testing is still limited to the sqlite3 backend, but it's a good first step. Please see if you can reproduce these results, ideally on different platforms and backends, and post your findings. It should now be relatively straightforward for devs to review, just by comparing the default branches of my repo and the official mirror at https://bitbucket.org/django/django - happy reviewing! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 7:14 pm, Vinay Sajip wrote: > suite is a boon in this regard. Though having worked through the > tests, it doesn't seem like the DRY principle is followed as much as > it could be ... for example, the same literals being used over and > over again in copy/paste fashion, requiring patches in multiple > locations to add u() and b() wrappers, for example. I didn't have time > to rationalise this, as I was focused more on identifying and fixing > failures and errors. Ironically, I notice that I repeated myself in using "for example" twice in the above paragraph ;-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 5:36 pm, Jannis Leidel wrote: > > Ah, that makes sense, in fact your approach is much closer to what I > remember doing when pip and virtualenv was ported. Right, since I did those ports originally :-) > Honestly, I'm not sure how hard the merge is, as I'm not sure how much > changed. Martin could probably shed some light on it how he wants to deal > with it (e.g. svnmerge.py or not). Sure. > Fair enough, I just realized that's a discussion we need to have in a > separate thread (~"What's the best approach for migrating Django projects > from 2.X to3.X?") that can be handled later in the porting process. When > in doubt I would rather use a module like six that has community traction > than writing our own though. There are areas where the current code needs to do metaclass-based checks, and that involves delving into the specifics of the implementation of with_metaclass. This being the case, I made a modified version of Benjamin's which uses "_DjangoBase" as the intermediate parent class. IMO we need this to distinguish from other classes implemented using with_metaclass from the official six package. > Personally I'm fine with it, but as you say, it requires discipline > (I broke pip more than once). But it's definitely something that needs > some input from the other core devs, and probably a very good > documentation of the dos and don'ts. Having good code coverage helps to spot these potential breakages well before a release (or even a checkin), and Django's extensive test suite is a boon in this regard. Though having worked through the tests, it doesn't seem like the DRY principle is followed as much as it could be ... for example, the same literals being used over and over again in copy/paste fashion, requiring patches in multiple locations to add u() and b() wrappers, for example. I didn't have time to rationalise this, as I was focused more on identifying and fixing failures and errors. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 9:29 am, Kiril Vladimirov wrote: > My point was the social factor. I mean the GitHub community is quite > bigger. It's not big issue if we're using mercurial instead of git. Sure, I > prefer git, it's faster and stuff, but it's not a big deal, right now. Well, it's not as if there's a big team required, and presumably interested parties who can contribute will be subscribed to this list anyway. There probably isn't the kind of need for a lot of back-and- forth, given that the overall approach and some specific implementation decisions are acceptable to the core devs. > Anyway, I'm trying to get into making the tests run and if I see some > result from my work, will try to figure out some sync between github and > bitbucket. Great, we'd like to see what you find. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 28, 1:04 pm, Jannis Leidel wrote: > > I'm a bit concerned that you didn't get in touch with us before you > started with the work, since tracking the changes would have been > easier. FWIW, Martin von Löwis, Alex and me would be those you can > ask if you need any further help, usually also on IRC in #django-dev. > Well, it's only been about a week of elapsed time. I always start these sort of ports on an experimental-and-potentially-throwaway basis, and I didn't know until I started how things would progress - and well, once I was into it, I was just dipping in and out amidst doing other things, so I suppose I just focused on the task at hand rather than the big picture. > Have you worked on top of the current py3k branch in SVN (which should > be mirrored in the Mercurial repo)? Also, did you get in touch with > Martin von Löwis who has previously spearheaded the porting efforts > for a while and has been granted commit privileges to the py3k branch? Yes, I used the BitBucket repo and the features/py3k branch within that. I didn't contact Martin directly about this, as my approach is a little different to what he had started, in that Martin's approach is based on running 2to3 on the code to get new 3-friendly code, whereas my approach is to have a single codebase with runs on 2.x and 3.x. It's more than a small philosophical difference - I value the information that 2to3 gives, but it just acts as the driver for porting "by hand" with my standard development tools. > I'm asking simply because I'm wondering how we should get your changes > reviewed and committed to the official repo, it's a bit like facing a > done deal. Well, I undertook the approach you mentioned in the "Python 3 and you" thread back in September, which was to look at the features/py3k branch and focus on test failures. Of course I didn't use the py3ktest script (which involves the 2to3 pass), running just plain runtests.py instead. And it's only been a little over a week, so it can't be that hard to merge the changes assuming you agree with the approach. Even if you start over, it's still less than a person-week of effort, right? > Aha, meaning that you've copied over parts of the six module? Would you > (also as a way for helping out Django users later to port their code) > recommend putting six completely in Django? We've previously done that > with other libraries, e.g. unittest2. I have no strong views on using six directly as a dependency, and I've only used small parts of it (with_metaclass, reraise) directly. It's certainly not needed as a dependency - you can see that the django.utils.py3 module (which provides the functionality needed by Django) is pretty small. > Yeah, we've talked about that and don't consider 3.0 or 3.1 to be sensible to > support in the future (given obvious bugs and/or lack of support from Python > core). Good, that rhymes with my thinking on it :-) > Indeed they are, so the next step is to review them bit by bit and align > it with the work done before your work. Ideally committing it to SVN. Right, and I can provide some help with that (other work permitting). Possibly some work could be done to look at merging the default branch with the features/py3k branch, as a first step - presumably default tracks SVN trunk closely. The main thing is to see whether you are comfortable with the single codebase approach (requires some small additional discipline from developers to consider str/bytes issues carefully, do imports from utils.py3 where appropriate, etc. - nothing too onerous). Carl Meyer may be able to chime in with his experience re. pip/virtualenv, which were ported using the same approach a while ago, and have seen maintenance work and new releases since then. Also it's worth looking at the way metaclasses have been done in the port and to see if you can identify any issues. A lot of the changes are pretty mechanical, wrapping literals with u() and b() - nothing too contentious there. Escape character handling esp. in regexes is another area where closer inspection would be worthwhile. The test codebase contained a lot of Unicode literals (i.e. Unicode literals in the source code, which would have been encoded in UTF-8 in the files actually stored on disk) and I have converted these to use Unicode escapes, as this is more portable. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
On Nov 27, 2:10 pm, Kiril Vladimirov wrote: > Have you made some sort of TODO list I could use? > Or selecting some failing test and make it run fine would be fine as well? I would say: just clone the repo, try to reproduce the results, report any differences you find from my results, and select any failing test to fix. Another area to look at is to exercise other database backends, e.g. PostgreSQL. I've only looked at sqlite as the obvious backend to use first. > Also, how about moving this project to GitHub? > Django is there, too(https://github.com/django/) and I thing we could find > more participants. It just so happened that I found the BitBucket mirror first and used that, but I wouldn't expect a serious developer, who wants to pitch in, to object to using Mercurial and BitBucket instead. There's very little to choose between them (BitBucket/GitHub), compared to the real work still to be done in getting Django production-ready on 3.x. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: I've made good progress in porting Django to Python 3, and could use some help!
Sorry the formatting of the post got mangled - not sure what happened there! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
I've made good progress in porting Django to Python 3, and could use some help!
I'm working on a port of Django to Python 3. I'm getting close, and in terms of test coverage pretty much almost there, but a few remaining test failures are eluding me, and I could probably use some help to speed things up. I started with the features/py3k branch on the BitBucket Django mirror (the one at https://bitbucket.org/django/django), but whereas that was approaching from a point of view of running 2to3 over the Django codebase, I've followed my preferred strategy of aiming for a single codebase for 2.x and 3.x. (This strategy worked well for the virtualenv and pip ports which I did a while ago.) I've updated the django.utils.py3 module to add whatever I needed, gratefully borrowing ideas from Benjamin Peterson's six project as needed. I run the standard test suite on the codebase using Python 2.7.2 and 3.2.2 (I'm not supporting 3.0 or 3.1 - 3.2 is stable, 'callable' came back and is liberally used in Django, and I'm not sure it's worth bothering with support for 3.0/3.1. The branch is available at https://bitbucket.org/vinay.sajip/django (features/py3k branch) Latest test result summaries are as follows: 2.7.2: ran 4229 tests in 301.690s OK (skipped=71, expected failures=3) 3.2.2: Ran 4174 tests in 303.619s FAILED (failures=6, errors=2, skipped=78, expected failures=2, unexpected successes=1) I think these results are encouraging, and I hope you agree! I added 7 skips, mostly these are due to different representations on 2.x and 3.x e.g. u'foo' vs. 'foo'. I'm posting the summary results to https://gist.github.com/1373553 The verbose listings are too big to post together to Github, giving me an "Entity Too Large" error. So these are posted separately to https://gist.github.com/1393994 (2.7.2) and https://gist.github.com/1394000 (3.2.2). The failures are: ERROR: test_bad_404_template (regressiontests.test_client_regress.models.TemplateExceptionTests) I think this is due to Django trying to handle an unaught exception and failing to find the 4500.html template, because the test has set the templates path to a bad value (deliberately, for the purpose of the test). ERROR: test_cull (regressiontests.cache.tests.DBCacheTests) This raises a "sqlite3.IntegrityError: datatype mismatch" but I haven't yet found the underlying cause. FAIL: test_modelform_factory_metaclass (regressiontests.model_forms_regress.tests.CustomMetaclassTestCase) This is down to the way metaclasses are implemented slightly differently. FAIL: test_lazy_objects (regressiontests.i18n.tests.TranslationTests) Proxy object comparison code differs between 2.x and 3.x, needs looking at. FAIL: test_RegexValidator_raises_error_23 (modeltests.validators.tests.TestSimpleValidators) I haven't tracked this down yet. FAIL: testParsing (regressiontests.conditional_processing.models.ETagProcessing) This is down to different representations on 2.x and 3.x: e\"t\"ag vs. e"t"ag FAIL: test_null_display_for_field (regressiontests.admin_util.tests.UtilTests) This is also down to proxy comparison logic. FAIL: test_templates (regressiontests.templates.tests.Templates) This is down to representational differences between 2.x and 3.x. I did the work and ran the tests on a VM running Ubuntu Oneiric Ocelot (x64). You can hopefully reproduce these by cloning my repository, updating to the features/py3k branch, and then running one of PYTHONPATH=.. python2.7 -u runtests.py --settings test_sqlite or PYTHONPATH=.. python3.2 -u runtests.py --settings test_sqlite in the tests directory. The work's taken about a week of elapsed time (on and off), and I haven't tracked any changes in the default branch since then. Of course, I realise there's still a lot left to do, but I hope I have broken the back of it. I've approached it just from the point of trying to get the tests to pass, and not (yet) focused on actually running realistic scenarios. Perhaps some of my fixes need changing - having other devs' eyeballs on the code can definitely help. Please let me have your comments! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Logging configuration and handle_uncaught_exception
On Jun 17, 1:48 pm, Matt Bennett wrote:
> > This is a case for a custom Filter object [1]. The filter object
> > implementation would only be a few lines, to reject logging when DEBUG
> > is True, and can be attached to the admin email handler in the default
> > logging configuration. [2] This way the logging call can occur in all
> > code paths, and the admin email handler itself can remain
> > general-purpose, but the backwards-compatible behavior can be maintained.
>
> > Matt, if you'd be willing to open a ticket for this, that'd be helpful.
> > If you feel like putting together a patch using the Filter approach,
> > that'd be even better ;-)
>
> Here you are:https://code.djangoproject.com/ticket/16288
>
> Since logging Filters are not specific to an individual logger or
> handler, I've just called it DebugFalseFilter. It's not a pretty name,
> but I couldn't come up with anything better - I decided it should
> convey what the filter allows through, but AFAIK there's no name for
> "not debug mode".
A more general solution would be something like this:
class ConditionFilter(logging.Filter):
def __init__(self, condition):
self.condition = condition
def filter(self, record):
return self.condition
which you can then use in the logging configuration using something
like
'require_debug_false': {
'()': 'django.utils.log.ConditionFilter',
'condition': not DEBUG,
}
This will allow the filter to be used in more places, e.g. you could
use a more complex settings-time condition.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Logging configuration and handle_uncaught_exception
Matt Bennett writes:
>
> > This is a case for a custom Filter object [1]. The filter object
> > implementation would only be a few lines, to reject logging when DEBUG
> > is True, and can be attached to the admin email handler in the default
> > logging configuration. [2] This way the logging call can occur in all
> > code paths, and the admin email handler itself can remain
> > general-purpose, but the backwards-compatible behavior can be maintained.
> >
> > Matt, if you'd be willing to open a ticket for this, that'd be helpful.
> > If you feel like putting together a patch using the Filter approach,
> > that'd be even better
>
> Here you are: https://code.djangoproject.com/ticket/16288
>
> Since logging Filters are not specific to an individual logger or
> handler, I've just called it DebugFalseFilter. It's not a pretty name,
> but I couldn't come up with anything better - I decided it should
> convey what the filter allows through, but AFAIK there's no name for
> "not debug mode".
Reposting this, after Google swallowed my earlier response. Sorry if it
eventually turns up, making this a double post ...
A more general solution would be something like this:
class ConditionFilter(logging.Filter):
def __init__(self, condition):
self.condition = condition
def filter(self, record):
return self.condition
which you can then use in the logging configuration using something like
'require_debug_false': {
'()': 'django.utils.log.ConditionFilter',
'condition': not DEBUG,
}
This will allow the filter to be used in more places, e.g. you could use a more
complex settings-time condition.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Logging configuration and handle_uncaught_exception
On Jun 10, 2:05 pm, Matt Bennett wrote: > Is there a reason the call to logger.error can't come before we return > the technical_500_response when DEBUG=True? It seems to me that the > debug level should be checked in each individual handler, rather than > determining whether the error is passed to the logger at all. Aren't these two separate things? It makes sense to have the logging of errors occur in all code paths, but that isn't connected with levels on handlers. The first requires the logging call to be added to the code, whereas levels on handlers can be configured to be as verbose (or not) as you like, independently of logging calls in the code. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Problems with logging and Python 2.4
On Oct 6, 3:19 pm, Russell Keith-Magee wrote: > something to be said for Luke's position of encouraging people to get > away from older Python versions by withholding nice features :-) Amen to that! Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Problems with logging and Python 2.4
On Oct 6, 2:29 pm, Russell Keith-Magee
wrote:
> On Wed, Oct 6, 2010 at 9:23 PM, Vinay Sajip wrote:
>
> > On Oct 6, 11:53 am, Russell Keith-Magee
> > wrote:
> >> On Wed, Oct 6, 2010 at 6:46 PM, Luke Plant wrote:
> >> > On Wed, 2010-10-06 at 12:24 +0800, Russell Keith-Magee wrote:
>
> > There are other possibilities. For example, you could use a Filter to
> > add contextual information to the log, as in this example:
>
> >http://docs.python.org/dev/library/logging.html#using-filters-to-impa...
>
> > Filters can be added to loggers, but they only operate on the logger
> > they're attached to, and not ancestor loggers. So they might need to
> > be added to several Django loggers, not just the 'django' logger.
>
> I considered that. The problem I hit is that in order for a filter to
> operate, you still have to have access to the data your want to
> annotate onto the record. As the patch I've provided demonstrates, we
> can get access to the current request out of the top stack frame, but
> other details such as the CSRF error condition aren't in a context
> where we can access them (at least, I can't see a way, other than
> threadlocals ;-)
>
I haven't checked the other cases, but in the CSRF case ISTM you have
access to the request at all the places where you log; hence you could
do e.g.
request._extra = { 'status_code': 403 } # or whatever reserved
attr name you want
and use the fact that you can dig the request out of the stack frame,
hence get access to the relevant data to stuff into the LogRecord via
the Filter.
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Problems with logging and Python 2.4
On Oct 6, 11:53 am, Russell Keith-Magee wrote: > On Wed, Oct 6, 2010 at 6:46 PM, Luke Plant wrote: > > On Wed, 2010-10-06 at 12:24 +0800, Russell Keith-Magee wrote: There are other possibilities. For example, you could use a Filter to add contextual information to the log, as in this example: http://docs.python.org/dev/library/logging.html#using-filters-to-impart-contextual-information Filters can be added to loggers, but they only operate on the logger they're attached to, and not ancestor loggers. So they might need to be added to several Django loggers, not just the 'django' logger. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: Final call for comments
On Oct 2, 1:59 am, Kevin Howerton wrote: > Also didn't realize that adding a nullhandler to just the root logger > would remove those warnings (pretty cool)... ended up writing a bit > that adds a nullhandler to any logger without handlers for my > implementation in lumberjack... woops. > Yes, that's because a search is done for handlers in all ancestors of a logger (except when propagate is sent to False - ancestor scanning stops there). A very common case is for an application developer to just add handlers to the root logger, to capture events from all components/ libraries used. That's usually what you need for development, though sometimes you also have files fed by just one component (e.g. SQL queries) . For production, of course, you have e.g. email handlers for critical stuff as well, plus files just to catch errors, and so on. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #6735 -- Class based generic views: call for comment
On Oct 2, 1:01 pm, Carl Meyer wrote: > Again, arguments to __init__ are not the issue. What would have to be > checked is any assignment to self that takes place in __init__. How do > you propose to check that? > I think __slots__ would do for this: it would prevent a user of a view instance from just randomly assigning to self.X, unless X were named in __slots__. Likewise, __setattr__ can be used to check for assignments to self. Of course, both might be capable of being subverted, but could be helpful in catching inadvertent mistakes. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: Final call for comments
On Oct 1, 7:06 am, Russell Keith-Magee wrote: > > These are fairly minor modifications, so I'm still intending to commit > early next week, barring major objections. > As per my answer to Kevin - I think a NullHandler addition to the 'django' logger needs to happen internally (not under a site developer's control) and the 'null' handler should be removed from the examples (as it won't be needed any more). That should make the configuration dictionary even simpler :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: Final call for comments
On Oct 1, 10:28 pm, Kevin Howerton wrote:
> # Ensure the creation of the Django logger
> logger = logging.getLogger('django')
>
I think the comment is misleading. I agree, there'd be no need to
create the logger there as it doesn't seem to be used there. Two
points:
1. Library code should add a NullHandler instance to the top-level
logger so that in the absence of any other configuration, you don't
get spurious "No handlers found ..." message. This should be just
after the logger = logging.getLogger('django') line:
logger.addHandler(NullHandler())
This line should always be there, and the NullHandler should be
omitted from the example configurations. Essentially, it's implicitly
there in addition to the handlers configured in settings.py. If this
is not done and the user turns off configuration via LOGGING_CONFIG =
None but omits to add any other configuration, you would get that "No
handlers found ..." message.
2. Library code should generally never add handlers (other than
NullHandler) unless it's under the application (site) developer's
control. That's happening in this case because of the convention with
LOGGING_CONFIG: the site developer is effectively telling Django to
add those handlers, albeit in a declarative way.
Hope that's clear!
Regards,
Vinay Sajip
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: #6735 -- Class based generic views: call for comment
On Oct 1, 4:16 pm, Russell Keith-Magee wrote: > "Don't use instance variables on a class" isn't advice you're going to > see printed very often. Yet that would be the advice that would be > *required* in order to use stateless class-based views in the way you > describe. > > Objects have state. I don't think it's at all unreasonable to suggest > that a instances of a class based view should be able to have state. > And even if we documented the fact that they can't have state, I'm > willing to *guarantee* that people will *try* to make them stateful. > To perhaps complicate the issue a little further, I think there's a distinction between read-only state and mutable state. It seems reasonable to let view class instances have read-only state (this might be part and parcel of increased [re]usability) such as template names, output mimetype configuration etc. This would be quite safe, ISTM. For mutable state during request processing, I'd vote for sticking it all in the request. Then your method signatures need only include the request - not unreasonable - and if for some reason you couldn't do that, there's always thread locals for the requests (duck ;-)) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #6735 -- Class based generic views: call for comment
On Oct 1, 11:16 am, Johannes Dollinger wrote: > Could you (or anyone knowledgable) add a section, that explains why each > request should have its own view instance? > The thread-safety argument alone does not suffice: if all _request_ state > would be stored on request instead of the view, you wouldn't need new > instances per request. You could also pass around state explicitly - which > admittedly gets messy quickly. > So is this purely about preventing users from shooting themselves in the > foot? (hyperbole: Will there be protection from globals in django 1.4?) I think Johannes has a point. The copy-on-call may seem useful at first sight, but as it's a shallow copy, it may perhaps engender a false sense of security. So view_instance.attr = x will not result in threads stepping on each other's toes, but view_instance.container_attr.attr = x surely will? It seems better to stress thread-safety dos and don'ts in the documentation. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: Final call for comments
On Oct 1, 3:45 am, Kevin Howerton wrote: > Regarding the 'version' setting.. .is that in the python 2.7 > implementation? I ended up removing it from the included > logging-config in lumberjack as it wasn't really doing much. Yes, it's required so that we can introduce schema changes in future Python versions without breaking backwards compatibility. It's best to leave it in, it's just an extra line after all. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: request for comments
On Sep 27, 1:15 am, Russell Keith-Magee wrote: > > On top of that, there is the purely architectural argument: > threadlocals are a global variable by any other name. They increase > coupling in the systems in which they are used. If an engineer came to > you and proposed a design that relied upon the use of global > variables, they would be soundly ridiculed; but for some reason, > calling that same global variable a "threadlocal" makes it socially > acceptable. > You're right in many ways, but these things are there because they're useful. You can't even get away from globals - every module is a global, and there's even a global list of them in sys.modules. Even the global settings in Django, which are the cause of a lot of pain, have been very useful in their way. It's definitely true that in the hands of the Wrong People, abuse of thread-locals could lead to maintainability nightmares. But that's not to say that there's no place for them anywhere; it's just that you have to provide very clear do's and dont's. This problem is not specific to thread-locals; an analogous case occurs in class-based views. There's no question (IMO) that class-based views can be a real boon, but they also need a warning label about not storing state in the class instance when processing requests. Interestingly, in the use case I mentioned, the use of thread-locals leads to a decrease of coupling in the sense that some third-party library knows nothing about the thread it's being called from or the request that it's helping to process, yet its logging messages can contain that context and thus be more useful. The alternative would perhaps be to e.g. pass a request from the view layer into the lower layers of code, which (a) would lead to increased coupling for no real benefit (methods which include the request in their signatures just so it can be reflected in log messages, for example), and (b) might not even be possible if you are using third-party code to do a decent chunk of your work for you (and that's code you can't easily control, and have no say in the design of). > It is entirely possible to build a robust, testable web system without > the use of threadlocals. You just need to be thorough in your design > work. > Modulo the use of third-party code, the design of which you generally don't have much influence over. By and large, I think we're in agreement: I just think that thread- locals have their place, but must be used with care and under the guidance of experienced practitioners. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #12012 Logging: request for comments
On Sep 26, 2:16 pm, Florian Apolloner wrote: > I am aware of those; but let's imagine a 3rd party library which has > no idea what a request is. In my logs I still want to know the request > (even if it's just for formatting purposes…); so I am looking for a > way to attach request specific info to __all__ logging calls done in > one request __without__ adding it to the logging call. You might be interested in this blog post: http://plumberjack.blogspot.com/2010/09/configuring-logging-for-web.html which solves the problem you mention (attaching request-specific info, to all logging calls in a request, including calls made in 3rd party libraries you use, without having to add "extra" to logging calls). However, it requires thread-locals, which for some reason the Django core team seem to be against: http://code.djangoproject.com/wiki/CookBookThreadlocalsAndUser You might regard the rest of this post OT, but ... I haven't seen any detailed evidence for the "huge threat" mentioned in that Wiki page. In fact, Graham Dumpleton, author of mod-wsgi, in this post, seems to be saying that it's not such a security threat that you need to rule out use of thread-locals: http://groups.google.com/group/django-users/msg/eb686bf30a7e4c26 To me, Graham seems right on the money. Of course, if the core team (or any one else) can point to any known exploits which take advantage of flaws in Python's thread-locals implementation, that would settle things right away :-) Even then, the answer would be to fix the hole in Python. Also - even if Django itself doesn't use thread-locals, can you be sure that any third-party library you are using doesn't do so under the covers? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Proposal: modular django configuration
On Jun 5, 11:22 am, "burc...@gmail.com" wrote: > > It is hardly a stretch to see how this kind of code could be extended > > to try to import settings_auto or settings_default from each app in > > installed apps. > > Sorry, I'm not native speaker. Can't understand what's "hardly a > stretch" neither from context nor from dictionary. "Hardly a stretch" == "Not exactly difficult" Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Proposal: modular django configuration
On Jun 5, 8:08 am, Russell Keith-Magee wrote: > in which a user will be using your app. Every single time in my life I > have made the statement "Nobody will ever want/need to..." I have been > proven wrong. Consider it a corollary of Rule 34 :-) > That's why the YAGNI principle needs to be applied with care ;-) On this topic, though, I fully agree with you. App configuration should always be at the mercy of site-wide (project) configuration, otherwise the app's usability is severely restricted. Maybe that's fine for some of Yuri's apps, but as a general principle, hardwired connections between specific databases and apps sounds broken to me, too. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Logging in Django
al handlers, and other 'on startup' behaviors. This can't be a > signal because you need to have somewhere to install the first signal > handler. So, startup.py needs to use a different approach, and needs > to happen as early as possible in the process. Right after logging is > configured is as good a place as any. The only operational difference > between startup.py and your PRE_MODEL_CALLBACKS list is that > startup.py doesn't require any new settings in order to work -- you > just drop a startup.py in your app, and you're done. It doesn't seem to me that having one or two new settings is such a big deal (as we're talking about having a new LOGGING setting anyway), but it's your call. I see it as a taste thing, myself. It's probably the case that the startup.py approach would be an easier sell to newbies, too. > However, once this initial startup has been invoked, any other > callback that is required can be handled by signals. There is no > operational difference between the POST_MODEL_CALLBACKS and a > 'models_loaded' signal; but again, the signal approach doesn't require > any new settings. All we are arguing about is the need for this signal > at all. > > Your comment about avoiding the model loading race condition is > certainly a valid one, and that's probably enough to get me over the > line on the need for this signal. My only concern is whether there is > anything else that a "models_loaded" signal should be 'after'. If > we're going to introduce a "we've finished setup" signal, I want to > make sure that the place we emit it is a point at which we can > guarantee that we've actually finished the setup :-) Okay, and thanks for all the feedback. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: www.djangoproject.com down?
On Jun 1, 8:29 am, James Bennett wrote: > There was an issue a few days ago where Media Temple -- which provides > our hosting -- was under a DDoS attack, and so sites hosted there were > intermittently available. That's since been resolved. Right, perhaps that explains it. It's accessible now, but a lot slower than usual. Thanks for the response, sorry for the noise. Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
www.djangoproject.com down?
I've been having trouble accessing www.djangoproject.com recently, from here in the UK. Is this a known problem? Is the server down? Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Logging in Django
s a problem)." The way I've set things up, init_logging is called only once, though settings.py is imported twice. Logging is just one instance of something you want to do early in the startup process. The mechanism I've provided is more general than the minimum needed for logging, and the only compelling reason I can see for disallowing it is that you think it somehow allows people to shoot themselves in the foot, or "it's just not the Django Way". The default settings.py created by django-admin startproject could have the init_logging callback defined as in my example, so that users don't need to do anything more than set up settings.LOGGING according to their needs. IOW, for many uses, the users have no extra work to do between our two proposals. So, easy things are easy. But having the more general mechanism also makes harder things possible at almost no extra cost, and in my view that's a good thing. > startup process, I don't see that this is something that requires a > callback system independent of Django's existing signals framework. > Again, if you can provide a specific use case to prove otherwise, I'm > willing to be convinced. It's hardly a "callback system independent of Django's existing signals framework", by which I mean it's hardly worthy of calling it a "system" as it's a lot simpler. Basically, just something like if hasattr(settings, 'PRE_MODEL_CALLBACKS'): callbacks = settings.PRE_MODEL_CALLBACKS for callback in callbacks: callback() which is pretty darn simple, to my mind and hardly a competing framework to the signals stuff. If you think the names would cause confusion (e.g. BOOTSTRAP, CALLBACK, then those can easily be changed. If your objections are on aesthetic/taste grounds, I completely understand - that's a personal preference and there's no point in trying to sway anyone from their personal style preferences. But in terms of meeting the functional requirement (both logging configuration and the more general "do stuff on start-up once and once only") and bang for buck, what I've proposed is simpler, involves less code than a separate infrastructure of startup.py files, and provides for more advanced usages (e.g. importing models in a callback without worrying about race conditions) without making the simple cases harder. Regards, Vinay Sajip [1] http://groups.google.com/group/django-developers/msg/19f1ba7e4614cff6 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Logging in Django
On May 29, 1:05 am, Waylan Limberg wrote: > > Vinay has already stated that he is making his code (which is part of > python 2.7) available to be distributed with Django so it will work on > any version that Django works on. > Yes, I've develloped my Launchpad branch (which has the dictconfig code) on a machine running Ubuntu Dapper, with Python 2.4, as that's the minimum Python version requirement for Django now :-) Regards, Vinay Sajip -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
