Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
> in fact, it takes us back to the > original proposal of a SafeForm that just takes the request object as > an argument to its constructor. Well this seems much simpler, although there is still the requirement to add the csrf_fields whenever you write out the form in the template (which isn't much - I'm just looking for the shortest and most failsafe path). As for making a subclass - is there a problem having "request" as an optional field in the BaseForm constructor? It could then at least be included with as_table, etc automatically at the cost only of adding an extra constructor parameter to existing forms, and ignored without it. -rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
On Sep 23, 2008, at 18:32, [EMAIL PROTECTED] wrote: > Django's saving can trample data unintentionally, and indeed, we've > seen this in production. Trampling data is an extremely hard to find > problem since it amounts to a race condition. Since this could be > easily avoided in several different ways, I consider this > implementation incorrect and therefor wrong. From an outside > perspective it appears that Django's "mental" model o fa transactional > database is a database in serialied mode. If put in serialized mode, > Django doesn't trample data and is fine. > [repeat message ten times.] The benefit is having a known state of single objects in the database. Not a great benefit, I agree - but it's not wrong, or even unexpected. You tell a model to save itself. It does so. Even if you perceive this as misbehavior, wrong, ill-written and whatnot, keep it to yourself. From an objective point of view, it is an implementation solving the problem with storage of objects in a database. It might not do so in a way that accommodates your needs, nor mine to be quite frank - but I accept this as an implementation. It'd seem the discussion has taken a turn to semantics, and thus this message. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
So I guess it's not my brain farting. On Sep 24, 12:23 am, Simon Willison <[EMAIL PROTECTED]> wrote: > Unsurprisingly, I've been totally over-engineering this. There's no ... > I think we solve this with a setting, the default of which looks like > this: At least you haven't been writing over-engineered code for the past 2hrs :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 11:02 pm, Jan Oberst <[EMAIL PROTECTED]> wrote: > Adding a signed field with a timestamp would be a much easier way to > secure forms. But it's not nearly as as secure as having the token > signed with an additional cookie. By setting a signed cookie you can > verify that this very form was displayed to this very client. Also, > you don't want to expire a form too early for people who just type > slow. And if a token is available for too long someone can generate a > proper token and then use it for an attack for too long. The additional cookie is pointless. A single random-enough cookie is enough to differentiate between two users, and that's all you need. You can then use this cookie to include it in the signature. Here's a snippet to explain what I mean: http://www.djangosnippets.org/snippets/1082/ You start with the assumption of a secret unique user cookie (which I'll affectionately refer to as "the cookie"). The only two things that the user gets in plain text are the timestamp and the salt. The salt is probably unnecessary, but what the heck. Nonces are cool. To this information you add the signature of [timestamp, salt, form class name, the cookie], and that's your token. The form class name is probably unnecessary as well. So to CSRF the form now, you need to recreate the token -> you need to recreate/acquire the above signature. Recreate is a no-go because you don't know the site's SECRET_KEY. Acquiring would work using our site as a black box, but only if you knew the cookie -> contradicts our assumptions. Uh, come to think about it, it seems the secret cookie is enough, the rest is just superfluous. But it might be just my brain farting. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 11:51 pm, "Amit Upadhyay" <[EMAIL PROTECTED]> wrote:
> There is another option, a template tag. I would implement it as a
> middleware and a template tag. Template tag csrf_protect, will require
> CSRFMiddleware and django.core.context_processors.request, will add a
> input file containing something derived from {{ request }} and
> middleware will check and raise HttpForbidden.
Oddly enough that's exactly how ASP.NET MVC does it:
http://blog.codeville.net/2008/09/01/prevent-cross-site-request-forgery-csrf-using-aspnet-mvcs-antiforgerytoken-helper/
They use their equivalent of a view decorator rather than middleware,
which is what I'd suggest doing in this case as well (middleware in
Django is usually applied globally which isn't always what you want).
Cheers,
Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 24, 1:04 am, Simon Willison <[EMAIL PROTECTED]> wrote: > There's another option that avoids the need for any cookies at all: > generating a persistent one-use-only token when a form is saved, > storing that in the database and only allowing submissions that > include a token that was previously assigned. Scratch that - the tokens would still need to be assigned to an individual user (almost certainly keyed off a cookie) as otherwise an attacker could create their own tokens and use them to attack another user. It would work for sites protected using HTTP authentication rather than cookies though, as you'd be able to attach each token to the HTTP auth username. I don't think this is a case we need to address though. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 11:23 pm, Simon Willison <[EMAIL PROTECTED]> wrote: > CSRF attacks are a problem for systems where an action is only meant > to be available to a specific logged in user. This user is inevitably > identified by a unique cookie. This is normally a session cookie, > hence many CSRF protection mechanisms key their hidden form token off > the session cookie. There's another option that avoids the need for any cookies at all: generating a persistent one-use-only token when a form is saved, storing that in the database and only allowing submissions that include a token that was previously assigned. This avoids any need for cookies at all, but has the serious disadvantage that you end up with potentially thousands of useless tokens stored in your database. You can clear these out periodically but it's still overhead that it would be nice to avoid. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Tue, Sep 23, 2008 at 8:43 PM, Brian Beck <[EMAIL PROTECTED]> wrote:
> The problem is that any token, no matter where we generate it, isn't
> going to be submitted back with the POST request unless it's a field
> in the form that was submitted. So the only options I see are
> mangling the HTML to add these fields (CsrfMiddleware), or add them to
> the form objects (SafeForm).
There is another option, a template tag. I would implement it as a
middleware and a template tag. Template tag csrf_protect, will require
CSRFMiddleware and django.core.context_processors.request, will add a
input file containing something derived from {{ request }} and
middleware will check and raise HttpForbidden. Its so ugly that it
does not deserve a form validation error in my opinion. This will
require least amount of changes in existing sites.
--
Amit Upadhyay
Vakow! www.vakow.com
+91-9820-295-512
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Running tests.py in apps without models.py
On Wed, Sep 24, 2008 at 3:54 AM, Brian Beck <[EMAIL PROTECTED]> wrote: > > On Sep 23, 3:40 pm, "Adam J. Forster" <[EMAIL PROTECTED]> wrote: >> Hi Eric, >> >> That's what I have done at the moment, but as you say it's a bit of a >> hack and I'm not sure that I'm happy with it. > > I ran across this bug the other day too; quite annoying. It's ticket > #3310 and there appears to be a patch. Eric's suggestion is the current workaround. #3310 is the ticket tracking the problem, and I do want to close this ticket. The patch that is applied may work (I haven't tried it recently); however, there is potentially some crossover with the changes to app registration feature that have been under discussion for a while. I'm hesitant to check in anything at this point without knowing how #3310 intersects with that work. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 1:06 am, Simon Willison <[EMAIL PROTECTED]> wrote: > Since all form.protect(response) actually does is ensure that there's > a csrf cookie, you should only actually have to do it once for one of > the forms. Unsurprisingly, I've been totally over-engineering this. There's no need for form.protect(response) at all - in fact, the form doesn't need to have anything to do with the response object. This dramatically simplifies things - in fact, it takes us back to the original proposal of a SafeForm that just takes the request object as an argument to its constructor. CSRF attacks are a problem for systems where an action is only meant to be available to a specific logged in user. This user is inevitably identified by a unique cookie. This is normally a session cookie, hence many CSRF protection mechanisms key their hidden form token off the session cookie. That's also what Django's CSRF middleware does. I like to be able to avoid sessions where possible, which is why I didn't want SafeForm to use the session cookie and thought it would need to set a cookie itself. But even if an application isn't using sessions, it's still going to need to use one or more cookies that are unique to the user. I think we solve this with a setting, the default of which looks like this: CSRF_COOKIES = [SESSION_COOKIE_NAME] SafeForm creates its csrf_token based on a hash of the cookie values for the keys in that list. It's a list because some authentication cases may rely on more than one cookie to uniquely identify the user (rare but possible). By default, it will use the Django session cookie. If you're not using sessions in your app (you're using a signed cookie called 'current_user' for example) you can specify that in the CSRF_COOKIES session. So, the bit about the SafeForm needing access to the response was a red herring. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 10:17 pm, "Robert Coup" <[EMAIL PROTECTED]> wrote: > then when you get a form submission, base64-decode it, split at "/", check > the hash matches by recalculating it, then use the proximity-to-timestamp > and the remote_addr to check the form validity. Anything that relies on remote_addr is flawed, because IP addresses change all the time. I frequently load up a Google Groups thread on my laptop, compose a reply on the train to work and submit it when I get there - and since I've moved networks my IP address changes in between loading the form and submitting it. There's also the risk of proxies that load balance traffic through different IP addresses, not to mention IP addresses that are shared by many people (including a potential attacker). Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
On Tue, Sep 23, 2008 at 12:52 AM, David Cramer <[EMAIL PROTECTED]> wrote:>
> For me, personally, it would be great if this could accept callables
> as well. So you could store the username, like so, or you could store
> a choices field like:
>
>field = models.IntegerField(choices=CHOICES)
>denorm = models.DenormField('self', 'get_field_display') # which
> would rock if it was .field.display ;)
I think denormalizing with callables is a very different thing than
denormalizing with expressions that can be evaluated by the database.
Not that they both aren't worth supporting, but db-level expressions
are going to be far easier and more efficient to validate and update
in bulk - no looping in Python, just executing SQL. In this case, I
think your example would be better suited as an FK, for denorm
purposes.
Callables would be useful for something more complicated like
abstracting auto_now to not be limited to dates, that is allowing a
field value to be set by a callable on save, not just create, for any
field type.
On Tue, Sep 23, 2008 at 2:42 AM, Andrew Godwin <[EMAIL PROTECTED]> wrote:
> Still, with an
> AggregateField(Sandwiches.filter(filling="cheese").count()) it's still
> possible to work out that you want to listen on the Sandwiches model,
> and you could then fall back to re-running the count on every Sandwich
> save, even if it ends up not having a cheese filling.
I'm not sure I like the idea of accepting arbitrary QuerySets. It
could just be my point-of-view, but I see automatic denormalization
and aggregation as intimately tied, where a denormalized field is just
a declarative aggregate expression that's optionally cached in a
column. I think this makes it easy to understand and document, since
aggregation queries and calculation fields would support the same
features, and it also allows the implementation to share a lot with
aggregates. It's also better in terms of storing and updating a
calculation: you can calculate on reads or writes for N objects in one
query without N subqueries, though it may involve a lot of joins.
> So, I think the best approach would be one to replicate fields (like my
> current DenormField; perhaps call it CopyField or something) and one to
> cache aggregates (an AggregateField, like above).
I'd also be hesitant to have two separate fields for these cases,
since copying a related field value is just a simple SQL expression. I
think the same calculation field could be used for both, by assuming a
string is a field name:
name = CalculationField('user.name')
or by using F expressions:
name = CalculationField(F('user.name'))
Another approach to calculations, that doesn't necessarily involve
denormalization, would be to use views. Russell talked to me a bit
about this at djangocon, and I think the idea was that if you solve
basing models on views (Isn't it actually possible now? Maybe we need
read-only fields), and then have view creation support in the ORM,
then calculation fields can be implemented with views. I see that
un-stored calculations are re-implementing views, but I don't know
enough about views to know whether they have some performance
advantages over embedding the calculation in a query.
-Justin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Wed, Sep 24, 2008 at 5:29 AM, Brian Beck <[EMAIL PROTECTED]> wrote: > > On Sep 23, 12:13 pm, oggy <[EMAIL PROTECTED]> wrote: > > Could we just include something like a signed salt+timestamp > > +REMOTE_ADDR in a hidden field? It's not exactly bulletproof because > > of the possibility of a same-IP-CSRF (affecting people behind > > proxies), but it's dead simple and doesn't require a lot of code > > change: Form -> SafeForm + request as the first parameter to __init__. > > Heck, I'd even trust sed to do it for me ;). > > Timestamp and REMOTE_ADDR wouldn't make a proper token unless we also > included the timestamp and REMOTE_ADDR as hidden fields -- the server > needs to be able to *regenerate* the token when the form is submitted > in order to validate the POSTed token. There are still ways to use these or other identifying values without the server having to store them somewhere. csrf_token = "[timestamp]/[remote_addr]/[hash]" where hash = sha1(timestamp + remote_addr + secret_key) eg. 104129/123.123.123.123/40bd001563085fc35165329ea1ff5c5ecbdbbeef base64 encode the csrf_token if you want to make it slightly more obscure. eg.MTIyMjIwNDEyOS8xMjMuMTIzLjEyMy4xMjMvNDBiZDAwMTU2MzA4NWZjMzUxNjUzMjllYTFmZjVjNWVjYmRiYmVlZg== then when you get a form submission, base64-decode it, split at "/", check the hash matches by recalculating it, then use the proximity-to-timestamp and the remote_addr to check the form validity. Rob :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 6:13 pm, oggy <[EMAIL PROTECTED]> wrote: > Could we just include something like a signed salt+timestamp > +REMOTE_ADDR in a hidden field? It's not exactly bulletproof because > of the possibility of a same-IP-CSRF (affecting people behind > proxies), but it's dead simple and doesn't require a lot of code > change: Form -> SafeForm + request as the first parameter to __init__. > Heck, I'd even trust sed to do it for me ;). Adding a signed field with a timestamp would be a much easier way to secure forms. But it's not nearly as as secure as having the token signed with an additional cookie. By setting a signed cookie you can verify that this very form was displayed to this very client. Also, you don't want to expire a form too early for people who just type slow. And if a token is available for too long someone can generate a proper token and then use it for an attack for too long. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Running tests.py in apps without models.py
On Sep 23, 3:40 pm, "Adam J. Forster" <[EMAIL PROTECTED]> wrote: > Hi Eric, > > That's what I have done at the moment, but as you say it's a bit of a > hack and I'm not sure that I'm happy with it. I ran across this bug the other day too; quite annoying. It's ticket #3310 and there appears to be a patch. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On 23 sep, 01:27, Simon Willison <[EMAIL PROTECTED]> wrote:
> On Sep 22, 9:41 pm, Brian Beck <[EMAIL PROTECTED]> wrote:
>
> > - If there's some other way to spell form.protect(response).
>
> Here's a crazy idea that might just work:
>
> class AddArticleForm(forms.SafeForm):
> headline = forms.CharField()
> # ...
>
> def add_article(request):
> form = AddArticleForm(request)
> if form.is_valid():
> # Process the data in form.cleaned_data
> return HttpResponseRedirect('/done/')
> return form.render('add_article.html', {
> 'extra_context_args': 'Go here',
> })
>
> We're doing a bunch of interesting things here. First, it's
> AddArticleForm.__init__ itself that looks at request.method and
> decides if it's going to bind to the data (if request.method ==
> 'POST') or simply create a new blank form. Second, we're using
> form.render() instead of render_to_response. form.render is a thin
> wrapper around render_to_response that does the following:
>
> 1. Adds 'form' to the context - a "form_var='article_form'" keyword
> argument could be used to change this default behaviour
> 2. Uses a RequestContext (with the request that was passed to
> AddArticleForm.__init__) - I can't think of any reason not to
> 3. Creates the HttpResponse using render_to_response
> 4. Sets a CSRF cookie on the response, if necessary
>
> This solves all of our problems in one shot (the need to sometimes set
> a cookie, having access to the request, etc), with the added bonus of
> reducing the amount of view boilerplate needed to use a form to the
> absolute minimum.
We've implemented a similar aproach to minimize put same logic pattern
in views, and so maximize DRY philosophy:
https://tracpub.yaco.es/cmsutils/browser/trunk/forms/forms.py
In this URL you can check GenericForm, and its childs, GenericAddForm
and GenericEditForm. The only requirement is to pass request into
constructor, that maybe is a little more coupled solution that normal
Django forms (here DRY make conflict with "less coupled")
With a request passed to forms, you can do a lot of things in
reusability, like SignedForm, but it's true that is less explicit
solution ("explicit better than implicit", third philosophical
concept).
Regards,
Manuel Saelices.
>
> The significant downside is that having a render() method on a form
> that performs the same function as render_to_response feels really,
> really strange. It's convenient, but it just doesn't feel right and
> I'm not sure I can justify it.
>
> Interesting option though.
>
> Cheers,
>
> Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Running tests.py in apps without models.py
You can just put a models.py there that is empty. A slight hack, but it should work just fine. (Think of it as __init__.py's big brother :)) Eric On Tue, Sep 23, 2008 at 2:27 PM, Adam J. Forster <[EMAIL PROTECTED]>wrote: > > Firstly I'm sorry if I have posted this in the wrong place, but I > think that it belongs here and not on the django-users list. > > Here's my problem, in most of our projects at work we have an app > called 'core' which contains modules that are either used by several > other apps or are not specific/large enough to justify their being in > their own apps. > > Today I needed to write some doctests for a function in core.utils so > I created a tests.py file inside core and placed my doctest there. > When I attempted to run the tests using 'manage.py test' I found that > they were not being run, when I ran 'manage.py test core' I got the > following error: > > django.core.exceptions.ImproperlyConfigured: App with label core could > not be found > > After some digging around in the Django source code I found the > problem. django.test.simple.run_tests makes calls to either > django.db.models.get_apps or django.gb.models.get_app to identify > which app(s) to search for tests. These functions return the models > module for the app, if the app does not contain a models module it is > not considered to be a valid app and is therefore not searched for > tests. > > This is not the first time that I have had an app which does not > contain a models module, and I can foresee that I will do the same > thing in the future. These apps may not contain models but they still > have code which needs testing. > > So my question is should I file this as a bug and start working on a > patch for simple.run_tests that checks all INSTALLED_APPS for both > models.py and tests.py when searching for tests? Or is this something > which is not likely to change, in which case would I be better off > writing my own test runner instead? > > Kind regards, > -- > Adam J. Forster > > > -- Eric Holscher Web Developer at The World Company in Lawrence, Ks http://www.ericholscher.com [EMAIL PROTECTED] --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Running tests.py in apps without models.py
Firstly I'm sorry if I have posted this in the wrong place, but I think that it belongs here and not on the django-users list. Here's my problem, in most of our projects at work we have an app called 'core' which contains modules that are either used by several other apps or are not specific/large enough to justify their being in their own apps. Today I needed to write some doctests for a function in core.utils so I created a tests.py file inside core and placed my doctest there. When I attempted to run the tests using 'manage.py test' I found that they were not being run, when I ran 'manage.py test core' I got the following error: django.core.exceptions.ImproperlyConfigured: App with label core could not be found After some digging around in the Django source code I found the problem. django.test.simple.run_tests makes calls to either django.db.models.get_apps or django.gb.models.get_app to identify which app(s) to search for tests. These functions return the models module for the app, if the app does not contain a models module it is not considered to be a valid app and is therefore not searched for tests. This is not the first time that I have had an app which does not contain a models module, and I can foresee that I will do the same thing in the future. These apps may not contain models but they still have code which needs testing. So my question is should I file this as a bug and start working on a patch for simple.run_tests that checks all INSTALLED_APPS for both models.py and tests.py when searching for tests? Or is this something which is not likely to change, in which case would I be better off writing my own test runner instead? Kind regards, -- Adam J. Forster --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
On Tue, Sep 23, [EMAIL PROTECTED] wrote: > > > You completely missed my point that if there's some integrity or other > > database error in the sequence of multiple updates or deletes (and when > > Django does updates to multiple tables or deletes, it extracts the id > > values first), we can currently roll back to the start of the sequence > > and people don't have half-update or half-deleted sequences of objects. > > Straight auto-commit behaviour loses that unless there are a lot more > > changes to the code than any patch I've seen currently. > > We're all agreed htat #3460 needs to support explicit DJango > transactions. I never intended otherwise. I always thought I had > added this to the patch, but apparently that didn't happen. I never > advocated removing transactions from Django altogether. I only want > things that aren't explicitly in a transaction to really not be in a > transaction. Richard has taken up the flag here, and I'll let him run > with it. Yeah, but you propose to make autocommit mode the default mode, i.e. an incompatible change that can affect existing applications in a very subtle way that could be very hard for some users to find. > > > You say "wrong", but relating that to how Django's save() works and > > saying that therefore loading/saving is wrong is drawing an unwarranted > > conclusion. > > Django's saving can trample data unintentionally, and indeed, we've > seen this in production. Trampling data is an extremely hard to find > problem since it amounts to a race condition. Since this could be > easily avoided in several different ways, I consider this > implementation incorrect and therefor wrong. From an outside > perspective it appears that Django's "mental" model o fa transactional > database is a database in serialied mode. If put in serialized mode, > Django doesn't trample data and is fine. So, is ticket #3460 about the lost update problem, i.e. one user saves data and another user overwrites it unintentionally? Or, is it about performance? About transaction handling in general? About postgresql insisting to roll back a transaction after an error? If you want to solve the lost update problem, fine. Let's discuss it. But, please, make up your mind what issue you want to solve. And then let's find out how to solve it, and don't assume a priori that autocommit transactions are necessarily the solution (which they clearly are *not* if applied alone, and not even when save() updates only modified data). We really need that you Richard agree on an issue before it makes sense to continue this thread. Michael -- noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg - Tel +49-911-9352-0 - Fax +49-911-9352-100 http://www.noris.de - The IT-Outsourcing Company Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk - Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
> I think that we > should try to write a patch which a) makes 3460 a configuration option > as suggested, and b) supports transactions on top of the autocommit > connection when explicitly requested (3460 currently doesn't, so parts > of the test suite currently fail when it is applied). +1 from me. I do hope that this becomes the default behavior in the future. > I'd appreciate people's thoughts on the following proposal of how to > do this. Particularly Malcolm, Jack and Collin since they seem the > main contributors to date. > > 1) Mechanism to make it a configuration option > > Various people on the 3460 ticket tracker have suggested using [6064], > but that isn't yet in trunk, and also wasn't generally agreed on the > tracker (e.g. does not allow custom Python). So, let's not do that. I think using custom python shifts the problem to the user, which is not great. Considering how much debate there has been on this issue, how can we expect users to understand when this custom python would be needed? I suspect people will be loathe to change the defaults unless they know why they should. > We need two things > - a way to create the backend in autocommit mode: I suggest a flag > 'native_autocommit' for DATABASE_OPTIONS > - a way for the running Django to tell that the backend is in > autocommit mode: I suggest a flag 'uses_native_autocommit' for > BaseDatabaseFeatures > > In both cases I'm deliberately choosing mechanisms and words so that > other backends could also later be changed to support an autocommit > mode. Seems reasonable to me. > 2) Transaction support when explicitly requested > > The current 3640 patch leaves a Django which does not support any > transaction behavior other than autocommit (leading to various test > suite failures). This was an omission on my part. I made similar patches to Twisted Python's adbapi which did redo the transaction behavior as well, and I always thought my patch for #3460 had this. It was not my intention that Django should abandon transactions, and it seems that this misunderstanding was the cause of some of the grief. Regarding your fixing of transactions: This is exactly why I've been advocating for using autocommit and doing away with implicit transactions. It's far simpler to understand and requires less code (ie, transactions only happen when Django makes them happen, and only one transaction class is needed). If it must be an option, then there will likely need to be multiple transaction backends, but it has been a while since I looked at the implementation details here, and Django 1.0 has changed significantly anyway (or perhaps in this specific area not?), so I can't offer much advice here. What you've suggested sounds reasonable. jack. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
> You completely missed my point that if there's some integrity or other > database error in the sequence of multiple updates or deletes (and when > Django does updates to multiple tables or deletes, it extracts the id > values first), we can currently roll back to the start of the sequence > and people don't have half-update or half-deleted sequences of objects. > Straight auto-commit behaviour loses that unless there are a lot more > changes to the code than any patch I've seen currently. We're all agreed htat #3460 needs to support explicit DJango transactions. I never intended otherwise. I always thought I had added this to the patch, but apparently that didn't happen. I never advocated removing transactions from Django altogether. I only want things that aren't explicitly in a transaction to really not be in a transaction. Richard has taken up the flag here, and I'll let him run with it. > You say "wrong", but relating that to how Django's save() works and > saying that therefore loading/saving is wrong is drawing an unwarranted > conclusion. Django's saving can trample data unintentionally, and indeed, we've seen this in production. Trampling data is an extremely hard to find problem since it amounts to a race condition. Since this could be easily avoided in several different ways, I consider this implementation incorrect and therefor wrong. From an outside perspective it appears that Django's "mental" model o fa transactional database is a database in serialied mode. If put in serialized mode, Django doesn't trample data and is fine. > Django's saving works exactly as intended with the current > code: all the models attribute values at the time of saving are stored > into the database. Sure, you'd like the option to only have the changed > values saved, which is something the rest of us are certainly interested > in implementing as well. Again, it probably won't be the default, since > it would change behaviour in ways that are almost impossible to detect > for thousands of users, but as an option it's not a bad idea. I have a > couple of different implementations of save-only-changes code floating > around that I've been thinking about (incorporating the various attempts > in the ticket addressed to this) and I'm sure something will land before > 1.1 I don't understand this mysterious use case for rewriting unchanged values to the database. It seems to me that you are saying that a patch to fix this data trampling bug might introduce new bugs, and so it should be optional. If there is a use case for rewriting back read values that don't change, please state it. Otherwise, I think making this an optional fix is silly. I'm completely surprised that anyone on the Django team would ship a framework that can trample data when the issue has been known about for over a year. Clearly I am missing some piece of information. > In other words, the object doing the second write wrote the values it > had. Once you accept that saving a model writes out all the attribute > values, this is pretty easy to understand. I agree we could document > this more clearly, but it's hardly mysterious behaviour (If any one says > "I assumed differently", that doesn't make it mysterious. It means that > making assumptions is usually a bad idea.) This is not something you can work around. You just have to accept that data will get trampled and that arbitrary other code can undo any database changes you make. This isn't really acceptable, especially with easy solutions available. > If you look at a calendar, you shouldn't be at all surprised by this. > The feature freeze date was weeks before the Portland sprint. Again, > this is an option for alternative behaviour that you're talking about, > not a no-brainer bug in Django. Acknowledge that there's a second side > to this, please. I will acknolwedge that you are claiming there is a second side, but you have not explicitly stated what that is. I have given pretty detailed scenarios for how these bugs affect us and how they can be fixed. Others have done the same and offered similar fixes. All I've seen so far is hand waving that the current way is fine and that it doesn't need fixing, but that our fixes may be added as non-default options at some nebulous later date. So, without some supporting evidence, it seems to some of us that this is a clear and simple bug in Django with easy solutions (ours was dirty tracking, but SELECT FOR UPDATE also works). > > So here Django > > actually causes data loss due to the race condition, even though > > transactions are used. > > It's not data loss if the intended mode is that the save() call writes > out everything. That works correctly and reliably under the given > hypothesis. Since there is no way to detect that you will blow away data this is data loss. This is not something that the application programmer can work around except by serializing postgresql or having a global database lock for all Django
Re: Denormalisation, magic, and is it really that useful?
On Tue, Sep 23, 2008 at 7:44 AM, Steve Holden <[EMAIL PROTECTED]> wrote: > > This appears to be a proposal to re-implement triggers inside Django. > > I can see there are benefits if the underlying DB platform won't support > triggers, but wouldn't triggers be the preferred solution when they're > available? That way there is no chance that changes can be made outside > the scope of the denormalization, and hence no need to recompute the > denormalized values. It would be nice to have an option to make atomic changes to the normalized and denormalized tables. -Dave --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 10:56 am, oggie rob <[EMAIL PROTECTED]> wrote: > I'm sorry, I used the wrong term here. I didn't mean that CSRF > protection isn't worthwhile, just that going the route of an extended > form might not be the best way to do it. > As for suggestions, I'm not sure I have one exactly, but I'm thinking > of perhaps overriding is_valid() and maybe using the RequestContext > object.. not sure yet. The problem is that any token, no matter where we generate it, isn't going to be submitted back with the POST request unless it's a field in the form that was submitted. So the only options I see are mangling the HTML to add these fields (CsrfMiddleware), or add them to the form objects (SafeForm). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 3:26 am, Simon Willison <[EMAIL PROTECTED]> wrote: > On Sep 23, 9:00 am, oggie rob <[EMAIL PROTECTED]> wrote: >> Is it worth a gut check to make sure this is worthwhile? > > Here's a useful case in point: the admin. Django's admin should ship > with CSRF protection turned on and baked in. Right now, I'm willing to > bet 95% of the Django admin sites out there are exploitable via CSRF > because the middleware wasn't turned on. This is really bad. > I'm sorry, I used the wrong term here. I didn't mean that CSRF protection isn't worthwhile, just that going the route of an extended form might not be the best way to do it. As for suggestions, I'm not sure I have one exactly, but I'm thinking of perhaps overriding is_valid() and maybe using the RequestContext object.. not sure yet. -rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
On Sep 23, 2:00 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote: > Without some Python-based approach, all I could see is maybe adding a > cross-platform "create trigger" API (ugh) to Django, which an > application could then use to set up its triggers during syncdb. > Otherwise, something like that forum app would have to implement a > trigger for all available backends or just ship with instructions on > how to set it up yourself. Triggers also aren't supported in Drizzle ( https://launchpad.net/drizzle ), which Django will probably want to target at some point. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Add "order_by" support for extra columns on Admin Site
Hello!
Before I wrote this e-mail, I look at the tickets but I haven't found
none with this approach.
I have this models:
class ArticleManager(models.Manager):
def get_query_set(self):
return super(ArticleManager, self).get_query_set().extra(
select={'comments_count': 'select count(*) from comments
where comments.article_id = article.id'}
)
class Article(models.Model):
# Fields must be here
objects = ArticleManager() # Overriding the `models.Manager`
I wanted that extra field "comments_count" have sort and filter
functions on Admin Site. Then I modify the
"contrib.admin.templatetags.admin_list.py" and
"contrib.admin.views.main.py" files to do it. This way I can create
"calculated fields" with sort and filter facilities on Admin Site,
that it's very useful for me to manage status for many models (for
example: "late" or "due soon" would be values for the calculated field
status)
My question: There is an easier/another way to do this?
Thanks.
--
Túlio de Paiva
[EMAIL PROTECTED]
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
On Tue, Sep 23, 2008 at 8:44 AM, Steve Holden <[EMAIL PROTECTED]> wrote: > This appears to be a proposal to re-implement triggers inside Django. I suppose it is. But then, perhaps triggers were a re-implementation of application-based denormalization. ... The chicken? The egg? The world may never know. :) > I can see there are benefits if the underlying DB platform won't support > triggers, but wouldn't triggers be the preferred solution when they're > available? That way there is no chance that changes can be made outside > the scope of the denormalization, and hence no need to recompute the > denormalized values. Perhaps the biggest downside is that distributed applications can't make use of it, since triggers aren't cross-platform. Sure, you could argue that denormalization is best suited for individual projects, rather than distributed apps, but I could see uses for it. Imagine a full-blown forum app that wants to denormalize its post counts. Without some Python-based approach, all I could see is maybe adding a cross-platform "create trigger" API (ugh) to Django, which an application could then use to set up its triggers during syncdb. Otherwise, something like that forum app would have to implement a trigger for all available backends or just ship with instructions on how to set it up yourself. That said, you're right, I would think it's usually safest to put that stuff directly in the db. That way, if you have other applications using other languages or perhaps just other ORMs, everything's all intact. I'm pretty sure I've seen arguments on both sides of the issue, though, so I'm not sure there's any objective answer to it. -Gul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
Yes, Steve, it's true that triggers do have much of the same functionality as this kind of proposal, but as you say, part of the proposal is to make these hopefully work with future non-SQL databases (although, admittedly, that's only a small piece of the puzzle). To be honest, my main drive is to have a nice, easy denormalisation API for Django; whether this means implementing triggers in all the database backends and using them behind this kind of API remains to be seen. The problems I can see with this at the moment are that: 1) If you change a DenormField, there's less writing migrations/fiddling with the database than if we managed it all in Django code (where you could just resync) 2) Triggers are a whole part of SQL Django just hasn't touched yet, and implementing support for them will take a reasonable amount of work in both debugging and testing for each backend. I'd really like to go ahead and implement this stuff 'vanilla' at first, relying on the tried-and-tested field and signal code, and then if it gets popular, add in the option to run denormalisations using triggers rather than inside Django. That way, we eventually end up with the best of both worlds, and get people using more efficient schemas first, which they can then easily bake into triggers when they come along, and if their DB supports them. Andrew Steve Holden wrote: > This appears to be a proposal to re-implement triggers inside Django. > > I can see there are benefits if the underlying DB platform won't support > triggers, but wouldn't triggers be the preferred solution when they're > available? That way there is no chance that changes can be made outside > the scope of the denormalization, and hence no need to recompute the > denormalized values. > > regards > Steve > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
This appears to be a proposal to re-implement triggers inside Django.
I can see there are benefits if the underlying DB platform won't support
triggers, but wouldn't triggers be the preferred solution when they're
available? That way there is no chance that changes can be made outside
the scope of the denormalization, and hence no need to recompute the
denormalized values.
regards
Steve
Andrew Godwin wrote:
> David Cramer wrote:
>
>> If you're not doing denormalization in your database, most likely
>> you're doing something wrong. I really like the approach that is
>> offered here.
>>
>> For me, personally, it would be great if this could accept callables
>> as well. So you could store the username, like so, or you could store
>> a choices field like:
>>
>> field = models.IntegerField(choices=CHOICES)
>> denorm = models.DenormField('self', 'get_field_display') # which
>> would rock if it was .field.display ;)
>>
>> You could also make it somehow accept querysets or something similar
>> for things like count(). I see a lot of possibilities and am a bit
>> disappointed I didn't come up with something this easy for my use-
>> cases.
>>
>>
>
> The key is making sure you can listen for changes on whatever's at the
> other end of your denormalisation. With my current snippet, it listens
> for a save on the model the foreignkey points to, then checks for the
> right ID; if we start accepting random querysets, then there has to be a
> way to resolve that back to conditions the signal listener can understand.
>
> Still, with an
> AggregateField(Sandwiches.filter(filling="cheese").count()) it's still
> possible to work out that you want to listen on the Sandwiches model,
> and you could then fall back to re-running the count on every Sandwich
> save, even if it ends up not having a cheese filling.
>
> So, I think the best approach would be one to replicate fields (like my
> current DenormField; perhaps call it CopyField or something) and one to
> cache aggregates (an AggregateField, like above).
>
> Simon Willison wrote:
>
>> Just so it's on the record, I'd like any denormalisation tools in
>> Django to include a mechanism for re-syncronizing them should
>> something go awry (like direct updates being applied to the database
>> without keeping the denormalised fields in sync). This mechanism could
>> then be exposed as a ./manage.py command which could be called
>> periodically to verify and fix any incorrect data.
>>
> Yes, this I very much agree with. The reason you always layer this stuff
> on top of a pre-normalised database is because you can then rebuild the
> data after playing with it externally.
>
> Doing so shouldn't be too much of a problem; have a management command
> that loads the models, and then just executes the update method on each
> of the denormalisationalish fields.
>
> Justin's idea of lazy updating is interesting, and probably quite doable
> (as well as what most people will want by default on aggregate queries).
>
> I'm also hoping any kind of aggregate denormalisation will work with any
> future extended aggregate support, but if the field just takes a normal
> QuerySet, that might Just Work™.
>
> Andrew
>
>
> >
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
Michael Radziej wrote: > There is no way to agree about the "#3460 ticket", because the ticket does > not really state what you want to solve, but how you want to change Django. ... > Now, please, pick only one pain I certainly agree that we've been all over the place on this thread! And my thinking has changed during the course of the discussion. I now think that there are several people each feeling different pains (I was c, I think Jack was originally mostly b), and each believing that their particular pain would be solved by the same technical feature for Django. I'll call this feature 'native autocommit' - i.e. getting a connection from the database backend using the autocommit isolation level, and then creating explicit transaction blocks only when needed. What I'd like to do is step back from the pains for now, and produce a patch which adds an optional 'native autocommit' to the Django Postgresql backends. We'll see later which and how many of the pains it fixes, rather than arguing now. If it's very successful, maybe it will become the default behaviour for Postgresql and be implemented for other databases. If not, then at least the people who want it can simply turn the option on, rather than having to maintain out-of-trunk patches. Hopefully this is in the spirit of Malcolm Trediinnick's comment: > I think it'd be a good idea to make sure we expose the ability to turn on > auto-commit, but I don't really like making it the default. In any case, > providing the ability that's can be controlled via, for example, a setting > is certainly the first step here. That's pretty independent of the whatever > the default might end up being. So, to reiterate. I'm thinking in terms of a new optional 'native autocommit' feature for the Django Postgresql backends. When the user chooses this option, the database connection which Django receives from the backend should be in "raw" autocommit mode, not wrapped in a transaction by the driver/backend, and Django would create explicit transaction blocks on top of this connection only when needed. I believe that this feature will require both: - Changes to the Postgresql backends to optionally return the autocommit connection - Changes elsewhere in Django (mostly db/transaction.py, I think), which notice when an autocommit connection is in use and then create explicit transaction blocks on top of this connection only when needed Please can we drop the discussion of pains for now, and just talk about how this new optional feature should best be implemented? My proposal is at http://groups.google.com/group/django-developers/msg/48500109ac5e514d - please read it mentally replacing 'ticket: 3460' with 'feature: option for native autocommit', then let's discuss! Cheers, Richard. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
On Mon, 2008-09-22 at 07:11 -0700, [EMAIL PROTECTED] wrote: [...] I've snipped the explanation of serialization levels. That's known stuff. It's also pretty much irrelevant to this discussion, but since you insist, I've addressed why the current behaviour is also a valid possibility, below. You completely missed my point that if there's some integrity or other database error in the sequence of multiple updates or deletes (and when Django does updates to multiple tables or deletes, it extracts the id values first), we can currently roll back to the start of the sequence and people don't have half-update or half-deleted sequences of objects. Straight auto-commit behaviour loses that unless there are a lot more changes to the code than any patch I've seen currently. > This means in a > transaction like: > > BEGIN; > SELECT username, password FROM auth_user WHERE id = 1; > UPDATE auth_user SET username = 'foo', password = 'bar' WHERE id = 1; > COMMIT; > > will do the wrong thing. Note that this is basically how Django does > all its attribute updates. You say "wrong", but relating that to how Django's save() works and saying that therefore loading/saving is wrong is drawing an unwarranted conclusion. Django's saving works exactly as intended with the current code: all the models attribute values at the time of saving are stored into the database. Sure, you'd like the option to only have the changed values saved, which is something the rest of us are certainly interested in implementing as well. Again, it probably won't be the default, since it would change behaviour in ways that are almost impossible to detect for thousands of users, but as an option it's not a bad idea. I have a couple of different implementations of save-only-changes code floating around that I've been thinking about (incorporating the various attempts in the ticket addressed to this) and I'm sure something will land before 1.1 > Let's assume that the Django code that > generated this was > > user.password = 'bar' > user.save() > > The problem here is that the SELECT statement will read all > attributes, and before UPDATE is executed, another transaction may > have changed the username column. Now the UPDATE actually reverts > such a change because Django does not keep track of which attributes > have changed. In other words, the object doing the second write wrote the values it had. Once you accept that saving a model writes out all the attribute values, this is pretty easy to understand. I agree we could document this more clearly, but it's hardly mysterious behaviour (If any one says "I assumed differently", that doesn't make it mysterious. It means that making assumptions is usually a bad idea.) > (Collin advocated very strongly for this patch in > Portland, but it was also not included in 1.0). If you look at a calendar, you shouldn't be at all surprised by this. The feature freeze date was weeks before the Portland sprint. Again, this is an option for alternative behaviour that you're talking about, not a no-brainer bug in Django. Acknowledge that there's a second side to this, please. > So here Django > actually causes data loss due to the race condition, even though > transactions are used. It's not data loss if the intended mode is that the save() call writes out everything. That works correctly and reliably under the given hypothesis. > So please don't assume you're safe just because you are in a > transaction. As mentioned elsewhere in other messages in this thread, you're overloading the term "safe" here and it's taking the thread way off track. All through this conversation you are continually failing to acknowledge that there is alternate side to these technical issues, as with all non-trivial changes. At no point is anybody saying that things like auto-commit shouldn't be possible. But it's not "the wrong serialisation level" or anything like that. It's one way of working that happens to be different from what Django makes possible at the moment. Every single non-trivial decision has both advantages and disadvantages. Trade-offs are all over the place and to successfully work out what to do requires a proper examination of both sides. > READ COMMITTED mode clearly does not work as Django > database developers imagined. Really?! This is just getting silly. Your mind-reading qualification is from which university? I cannot speak for any other developer (since I sadly don't have such a such a qualification), but read committed works exactly as I understand it -- no imagination involved, since it's been pretty well documented for years in PostgreSQL. It can produce non-repeatable reads but doesn't allow uncommitted reads. You can foam all you like about how incompetent all the developers are (and that's really what this is coming down to, since nothing at all here is really new information), but that simply isn't the case. It's simply unprofessional, as maintainers and contributors (i.e.
Re: Call for comments: CommonMiddleware (and others) break streaming HttpResponse objects.
On Jul 1, 1:02 pm, Ivan Sagalaev <[EMAIL PROTECTED]> wrote: > I've implemented iterable HttpResponse > in the first place out of purely practical reason: web server times out > if a particularly long file was passed into an HttpResponse. And also it > was really slow and was consuming all the memory. Ivan, does this imply that iteration does not work out of the box, i.e. ordinary HttpResponse __iter__() and next() are not used in the expected way? Or were you just using some middleware that consumed the response and customization was required because of that? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 23, 9:00 am, oggie rob <[EMAIL PROTECTED]> wrote: > I just read this thread now, and by the end of it, things are looking > pretty complicated. Is it worth a gut check to make sure this is > worthwhile? Although the middleware might be a hack now, it seems > sensible that it fits in request/response areas rather than in forms: > you still need to go out of your way to add it anyway (so users won't > necessarily "turn it on"); it takes a lot more code; add in the > multiple forms per page question, and to me it seems like you've fixed > a problem by introducing another. Here's a useful case in point: the admin. Django's admin should ship with CSRF protection turned on and baked in. Right now, I'm willing to bet 95% of the Django admin sites out there are exploitable via CSRF because the middleware wasn't turned on. This is really bad. I'm positive we can figure out a better API for CSRF protection than what we have at the moment. At the moment I'm focused on forms, but if there's something we can do at the view level instead I'd love to see some suggestions. > Finally, it doesn't take much to make a pretty message - something > like "You are under attack, close down your browser and try again" > with images of flaming people & such - for the (lets be realistic) > very rare cases when a CSRF attack occurs. I'm worried about false positives. One example where this would happen is if you were to change your SECRET_KEY (secret management is a whole other issue we haven't addressed). That's why I like the validation error approach - it's unobtrusive and doesn't unnecessarily scare people. We should definitely log detected CSRF issues though (logging = another issue). Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
David Cramer wrote:
> If you're not doing denormalization in your database, most likely
> you're doing something wrong. I really like the approach that is
> offered here.
>
> For me, personally, it would be great if this could accept callables
> as well. So you could store the username, like so, or you could store
> a choices field like:
>
> field = models.IntegerField(choices=CHOICES)
> denorm = models.DenormField('self', 'get_field_display') # which
> would rock if it was .field.display ;)
>
> You could also make it somehow accept querysets or something similar
> for things like count(). I see a lot of possibilities and am a bit
> disappointed I didn't come up with something this easy for my use-
> cases.
>
The key is making sure you can listen for changes on whatever's at the
other end of your denormalisation. With my current snippet, it listens
for a save on the model the foreignkey points to, then checks for the
right ID; if we start accepting random querysets, then there has to be a
way to resolve that back to conditions the signal listener can understand.
Still, with an
AggregateField(Sandwiches.filter(filling="cheese").count()) it's still
possible to work out that you want to listen on the Sandwiches model,
and you could then fall back to re-running the count on every Sandwich
save, even if it ends up not having a cheese filling.
So, I think the best approach would be one to replicate fields (like my
current DenormField; perhaps call it CopyField or something) and one to
cache aggregates (an AggregateField, like above).
Simon Willison wrote:
> Just so it's on the record, I'd like any denormalisation tools in
> Django to include a mechanism for re-syncronizing them should
> something go awry (like direct updates being applied to the database
> without keeping the denormalised fields in sync). This mechanism could
> then be exposed as a ./manage.py command which could be called
> periodically to verify and fix any incorrect data.
Yes, this I very much agree with. The reason you always layer this stuff
on top of a pre-normalised database is because you can then rebuild the
data after playing with it externally.
Doing so shouldn't be too much of a problem; have a management command
that loads the models, and then just executes the update method on each
of the denormalisationalish fields.
Justin's idea of lazy updating is interesting, and probably quite doable
(as well as what most people will want by default on aggregate queries).
I'm also hoping any kind of aggregate denormalisation will work with any
future extended aggregate support, but if the field just takes a normal
QuerySet, that might Just Work™.
Andrew
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Postgresql transaction aborts, despite being in autocommit mode
On Mon, Sep 22, Richard Davies wrote: > > > Can we figure out what this thread is about, and stick to it, please? > > As the person who originally started the thread ;-) > > I'd like to agree a design for a patch which will be accepted into > Django trunk and will enable us to close the #3460 ticket. There is no way to agree about the "#3460 ticket", because the ticket does not really state what you want to solve, but how you want to change Django. It's like going to the dentist and telling him to extract 5 left top (or however this terminology works). Well ... in the role of the dentist, we're asking you: Where does it really hurt? The ticket and the following discussion clames that: (a) django used the wrong transaction isolation level (b) it's unefficient that there are so many BEGIN/COMMITs (c) it is about that postgresql does not allow any actions after an error before you finish the transaction Now, please, pick only one pain, and try not to confuse us with too many stuff at once. We can deal with the other teeth later. Michael -- noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg - Tel +49-911-9352-0 - Fax +49-911-9352-100 http://www.noris.de - The IT-Outsourcing Company Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk - Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: django.forms.SafeForm - forms with built in CSRF protection
On Sep 22, 1:25 pm, Simon Willison <[EMAIL PROTECTED]> wrote: > CSRF[1] is one of the most common web application vulnerabilities, but > continues to have very poor awareness in the developer community. > Django ships with CSRF protection in the form of middleware, but it's > off by default. I'm willing to bet most people don't turn it on. > > I don't believe middleware is the right way to approach this. It's too > "magic" - it involves code that parses and re-writes your HTML as the > response is being returned. It also means CSRF failures can't be > gracefully handled - the middleware can throw up a big ugly error > page, but ideally a CSRF failure would be treated the same way as a > regular form validation error. I just read this thread now, and by the end of it, things are looking pretty complicated. Is it worth a gut check to make sure this is worthwhile? Although the middleware might be a hack now, it seems sensible that it fits in request/response areas rather than in forms: you still need to go out of your way to add it anyway (so users won't necessarily "turn it on"); it takes a lot more code; add in the multiple forms per page question, and to me it seems like you've fixed a problem by introducing another. Finally, it doesn't take much to make a pretty message - something like "You are under attack, close down your browser and try again" with images of flaming people & such - for the (lets be realistic) very rare cases when a CSRF attack occurs. All you need is a template, right? (And I would consider sending an email to the admin notifying them that at attack was attempted, at least to get an idea of how common these issues are.) -rob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Denormalisation, magic, and is it really that useful?
If you're not doing denormalization in your database, most likely
you're doing something wrong. I really like the approach that is
offered here.
For me, personally, it would be great if this could accept callables
as well. So you could store the username, like so, or you could store
a choices field like:
field = models.IntegerField(choices=CHOICES)
denorm = models.DenormField('self', 'get_field_display') # which
would rock if it was .field.display ;)
You could also make it somehow accept querysets or something similar
for things like count(). I see a lot of possibilities and am a bit
disappointed I didn't come up with something this easy for my use-
cases.
+1 from me!
On Sep 22, 7:00 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> On Sep 23, 12:21 am, "Justin Fagnani" <[EMAIL PROTECTED]>
> wrote:
>
> > In my experience at least, denormalization occurs
> > a lot and leaves a lot of room for mistakes, so it's something a
> > framework should handle if possible.
>
> Just so it's on the record, I'd like any denormalisation tools in
> Django to include a mechanism for re-syncronizing them should
> something go awry (like direct updates being applied to the database
> without keeping the denormalised fields in sync). This mechanism could
> then be exposed as a ./manage.py command which could be called
> periodically to verify and fix any incorrect data.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
