Re: Cross-DB JSONField ready for review

[Ole Laursen]

fredag den 2. august 2019 kl. 13.46.46 UTC+2 skrev Sage M.A.:
>
> As a follow-up to this message 
> 
>  and this ticket , I have 
> completed the implementation of a cross-DB JSONField.
>

As someone who has been using the Postgres-specific JSONField extensively 
for dynamic, custom fields for the past couple of years, can I humbly 
suggest that the some more thought goes into the field lookup before the 
current approach is enshrined?

The simple .filter(field__foo="hello world") is actually fine, but it gets 
really weird if you let users define foo. What if they call foo something 
like "contains"? What if "foo" is actually "Foo the bar?". The JSON makes 
sense:

{
  "Foo the bar?": true
}

but the Django filter does not.

The current documentation says you can use __contains, but as far as I can 
tell, __contains, besides being difficult to understand, cannot be used for 
queries like __icontains. And it overwrites a built-in, otherwise useful 
operator.

My current wish list is:

- kill special contains and contained_by (use Q(field__foo="xxx") | 
Q(field__bar="yyy") instead)

- kill has_keys and has_any_keys (use Q(field__has_key="xxx") | 
Q(field__has_key="yyy") instead)

- add possibility of using something more robust than __ for path 
traversal, perhaps just a JSON extract string like 
JSONExtract("owner.other_pets[0].name")

This could perhaps also allow us to use the JSONField content in places 
where you currently can't (e.g. annotate). I'm not sure how exactly to 
combine it with field lookup and operators, but I'm personally okay with 
something like

   .filter(**{ "myfield__" + JSONExtract("owner.other_pets[0].name") + 
"__icontains": "baz" })

That's way better than what we have now.

The neat thing about JSONField is that it can store some of the things that 
are otherwise difficult to handle with traditional schemas. I think with 
some love and a set of more orthogonal primitives, we could query it easily 
from Django, too.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/92277b8e-4453-496e-b0aa-e15b79a9d0a2%40googlegroups.com.

NULLs taking up space in foreign key indexes

[Ole Laursen]

Hi!

I recently noticed that the default indexes that Django generates for 
foreign keys also index NULL values, at least for PostgreSQL. Is this on 
purpose?

>From my digging, it looks like PostgreSQL used to exclude NULL values from 
the index, but not since some years. It's relatively easy to skip them by 
appending "where ... is not null" to the index creation, like this:

  create index for_bar_id_index on foo (bar_id) where bar_id is not null;

This comes up because nullable foreign keys are for optional relations, and 
I sometimes have big tables with a bunch of mostly NULL foreign keys (NULLs 
are cheap in PostgreSQL), and the indexes can then dwarf the table.

I have always assumed that the default indexes where for reverse joining 
and deleting purposes only so keeping track of NULLs doesn't actually seem 
necessary.


Ole


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d26f0ed4-94f3-48a5-8671-a3dfa3d5440b%40googlegroups.com.

Annotate after filter joins - is this a bug?

[Ole Laursen]

Hi!

I was reading the ORM documentation about annotate and aliases. Given

  class Paper(models.Model): pass

  class Author(models.Model):
  paper = models.ForeignKey(Paper, on_delete=models.CASCADE)
  name = models.CharField(max_length=100)

and the query

  
Paper.objects.filter(author__name__contains="Foo").filter(author__name__contains="Bar").annotate(bar=F("author__pk")).filter(bar=123)

Django 2.0.3 generates the query

SELECT ... FROM "x_paper" INNER JOIN "x_author" ON ... INNER JOIN 
"x_author" T3 ON ...
 WHERE (
  "x_author"."name" LIKE %Foo% ESCAPE '\'
  AND T3."name" LIKE %Bar% ESCAPE '\'
  AND "x_author"."id" = 123)

In other words, the annotate(bar=F("author__pk")) binds bar to the first 
join and not the second T3 join. Is this a bug?


I need to find all papers with one author fulfilling a set of criteria and 
another author (another as in author1.pk != author2.pk) fulfilling a 
different set of criteria.

The only way I can think of to do that is by somehow naming something from 
the join, like this

  Paper.objects
   .filter(author__name__contains="Foo").annotate(foo_pk=F("author__pk"))
   .filter(author__name__contains="Bar").annotate(bar_pk=F("author__pk"))
   .exclude(foo_pk=F("bar_pk"))

It could also be used to solve related join queries, like "find papers with 
authors with the same last name".

The only way I can see out at the moment is generating a query without the 
exclude part, then parsing it to find the Tx alias names and putting them 
in a RawSQL expression to form the final query.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9156ca7a-dfa9-4722-9513-f78e98607912%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: QueryDict and ordering

[Ole Laursen]

2018-01-05 19:24 GMT+01:00 Tim Graham :
> Preservation of dict ordering is guaranteed in Python 3.7+ so that
> officially fixes this, correct?
> https://mail.python.org/pipermail/python-dev/2017-December/151283.html

Sure, I can wait for that. Sorry for the noise, I should read that
list more often.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANb2Ov%2BRJHiDMjYXK64wXxmom0KCEtqRARwJkwGp3Xko60%3DcPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: QueryDict and ordering

[Ole Laursen]

2018-01-05 17:12 GMT+01:00 Tim Graham :
> Hi, Did you try writing a patch? I naively tried "class
> MultiValueDict(OrderedDict):" as the only change and it doesn't pass the
> tests. Perhaps more adaptations are required.

Tried just now, and yeah it takes a little more adaptation.
MultiValueDict is relying on __setitem__ not being called by the
underlying dict implementation in a couple of places, e.g.
MultiValueDict({ "foo": [ "bar"] }). I'm happy to write a little patch
if the idea makes sense.


> As for the motivation, I'm not sure if you described the problem in enough
> detail. (i.e. what does "It makes highly dynamic form situations more
> tedious to handle" mean?)

If you are using Javascript to insert extra forms or rearrange input
fields, and the order matters, then you can't use request.POST without
some kind of bookkeeping workaround. Either by encoding and
maintaining a sequence number client-side, or by adding extra fields
to handle it.


For instance, I have a bunch of rows like this where rows can be
rearranged by dragging:

  Foo: 1___   Bar:  hello__   [x]

  Foo: 3___   Baz:  [ Choice 1 ][x]

 [Add row]

I'm using a form to validate each row - so each have a name prefix.
Ideally, I could just go through request.POST.keys(), look for each
new set of values and instantiate forms as needed.


> If Python 3.6 preserves order, is this only an issue on Python 3.5? If so,
> I'd guess there would be little motivation in making a change (if it's a
> non-trivial change) given 3.5 will be end-of-life soon enough (2020).

Unfortunately, it's only preserved as an implementation detail, if you
write code that depends on the order being preserved, then you need to
derive from OrderedDict. There was a thread on the Python list, and
you can see here

https://docs.python.org/3.6/whatsnew/3.6.html

that they're careful not to say that dict preserves the order.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANb2OvL8KxkD3ug_C0RDK0hnyQWup09OE%2BVx0%3DLCSfRcW%2B_BSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

QueryDict and ordering

[Ole Laursen]

Hi!

Would it be possible to derive QueryDict (i.e. MultiValueDict) from an 
OrderedDict instead of dict?

I'm finding it increasingly irritating that the original order is kept by 
the whole stack right until Django puts it into a dict. It makes some 
highly dynamic form situations more tedious to handle.

Now that Python 3.6 preserves the order of dicts as an implementation 
detail, there should not be any performance overhead as far as I'm aware.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5b49e5c8-68df-41c8-96b8-1ba6f36fa0cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.core.signing and safe characters

[Ole Laursen]

2017-06-21 15:26 GMT+02:00 Florian Apolloner :
> Sorry I've mixed it up with with encodeURI. encodeURIComponent takes the
> "safe" route and escapes everything (at least as far as I understand it)
> because it does not know where the component (unless "component" itself has
> a sepcific meaning that I am not aware of) ends up -- this also makes the
> function somewhat ugly to use because you escape more than you need to. On
> the other hand, if you use encodeURI:
>
> encodeURI('http://google.com?:a=:asd')
> "http://google.com?:a=:asd;
>
> it does not escape ':' since it knows that there is no need to escape it.

This is a side note, but actually encodeURI can't be used to encode
query string parameters. It won't quote = or & either, as you can see
from your example. You need to use encodeURIComponent on each of the
parts separately.

So of the pair, encodeURI is probably the one that has questionable
use as you said, I've personally never needed it.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANb2OvK9AfqBJCa35qOnkELtdTpp7SuMtqqsqJ5kYdHNha4d6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: django.core.signing and safe characters

[Ole Laursen]

Hi again!

I'm sorry if I gave the impression that I'm trying to nitpick
adherence to a standard. There was some discussion about it in the
comments in the link I provided, and it looks like there are different
interpretations, but that's not what I'm interested in.

What I'm addressing here is specifically the comment in the source and
the assumptions it is making, which I have found confusing in practice
because I bumped into the encoding issues.

Reading that comment, I expect to be able get a token and construct a
URL with it, and get a nice URL with no percent goo in it. I don't
expect URL safe strings to be encoded by standard API, neither in the
browser or server-side.

>> It is used for the scheme "https:". encodeURIComponent(":") returns "%3A".
>
> The usefulness of that function seems questionable to me.

Could you elaborate why? What other function would you use client-side
to make sure that you generate a valid query string given a set of
parameters?

Python does the same, by the way:

$ python3
>>> import urllib.parse
>>> urllib.parse.quote(":")
'%3A'
>>> urllib.parse.urlencode({ ':foo': ':bar' })
'%3Afoo=%3Abar'

It looks like urllib is more eager when it comes to escaping, * is
also escaped by default. . isn't. So it would probably have been
better if the signing code had picked . as default separator.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANb2Ov%2BNaz_cpH%2BTRYQ5FvGeCM33r6nFm%3D7sv55g6wfO6PhnGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

django.core.signing and safe characters

[Ole Laursen]

Hi!

Maybe this has no practical implications, but this has been bugging me for 
a couple of years now, ever since I started using django.core.signing to 
generate tokens: if you take a look at

  https://github.com/django/django/blob/master/django/core/signing.py

the comment at the top says

   There are 65 url-safe characters: the 64 used by url-safe base64 and the 
':'. 
   These functions make use of all of them.

Yet, : is specifically mentioned as a reserved character:

  https://perishablepress.com/stop-using-unsafe-characters-in-urls/

It is used for the scheme "https:". encodeURIComponent(":") returns "%3A".

If I do a test with a link like  in Firefox, the 
browser doesn't quote any of the colons, though. OTOH, if you put in 
"foo:bar/" as a relative link, foo: is interpreted as a scheme. So it's not 
unconditionally safe.

Furthermore, the above page lists some more characters as safe:

  $-_.+!*'(),

Of these only -_.!*'() are not quoted by encodeURIComponent and -_ (and 
perhaps .) are already taken by signing code.

But in any case, the comment, although satisfying to read, is AFAICT 
incorrect?

I don't know if it is worth it to switch to another default separator (say 
*). There would need to be a fallback to : for some years at least.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4b0fb342-53d8-4f36-90b6-d450e24bfdab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A.objects.getdefault

[Ole Laursen]

On Friday, October 12, 2012 3:35:53 PM UTC+2, Chris Wilson wrote:
>
> I'm strongly in favour of a simple, obvious way to do the common thing, 
> which is to return None if the object doesn't exist, instead of throwing 
> an exception. My preferred method names would be .nget(), .get_or_none() 
> or .first().
>

IMHO the fact that we have all these inventive solutions is ample proof 
that people do come across this problem and find it annoying. I just found 
out today that there's a somewhat related .latest().

Anyway, without a sponsor with commit access I guess this is dead, so I'll 
shut up now. Thank you for your time. 


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/xFWfDQzuQ7cJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: A.objects.getdefault

[Ole Laursen]

On Tuesday, October 9, 2012 7:15:55 PM UTC+2, ptone wrote:
>
> Earlier discussion
>
> https://groups.google.com/forum/?fromgroups=#!topic/django-developers/Saa5nbzqQ2Q
>

This was the thread I referred to. If was from 2006 and ended up being 
about something else.
 

> tickets:
>
https://code.djangoproject.com/ticket/17546
>

Closed as dupe with no further analysis.
 

> https://code.djangoproject.com/ticket/2659
>

This is about a different API which is a bit more cumbersome. And it was 
closed 6 years ago with a - "this could be feature creep".
 

> https://code.djangoproject.com/ticket/11352
>

This is a different proposal.


Just to expand slightly on my proposal - I don't actually want to get 
anything else than None out of get. I just want something that makes 
intuitive sense to a Python programmer and saves me from the annoying 
try-except pattern or the related:

objs = SomeClass.objects.filter(slug="xyz")
obj = None
if objs:
obj = objs[0]

which I often end up shortening as

obj = SomeClass.objects.filter(slug="xyz")
if obj:
obj = obj[0]

You can't get a zero or one objects out of the ORM elegantly, something I 
find I have to do relatively often in some code bases. I don't consider 
try-except intuitive for a situation where the exception case isn't 
exceptional.

default=None makes sense when you are used to dicts, .first() or 
.get_or_none() or .getnone() or .single() would be fine too.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/r8vZ2A9tIkgJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

A.objects.getdefault

[Ole Laursen]

Hi!

What do people think of

  A.objects.getdefault(slug="hello")  # returns None if slug doesn't exist
  A.objects.getdefault(slug="hello", default=A())  # returns empty object 
if slug doesn't exist

I find that in practice, most of the time it would be better to get None 
back instead of the DoesNotExist exception, but changing .get() is of 
course impossible without breaking the API.

I do think that

  A.objects.get(slug="hello", default=None)

is more elegant, but it might break some code. Although it does seem really 
unlikely anyone would add a field named "default" to their model and then 
use it to locate a unique object.

The only related info I found in the issue tracker and on this list was a 
thread from 2006.

Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/yt14RXkhAkwJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Suppressed template errors in admin

[Ole Laursen]

On 6 Sep., 09:17, Thomas Guettler  wrote:
> Ole, can you please set up a a branch at github or bitbucket?

Eh, is that not overkill?

I don't think discussing this is worth anyone's time. No need to
promise anything or elevate this to a Big Decision as long as it's
about trivial non-intrusive fixes.

I'm still hoping someone (Russ?) will just take the patch and apply
it, and I can go back to being the happy camper. :)


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Suppressed template errors in admin

[Ole Laursen]

On 25 Aug., 06:19, h3  wrote:
> I'm not sure suppressing templates errors for the admin is such a
> great idea.

The suggestion on the table is to fix the couple of places where admin
is sloppy and doesn't include all the variables it uses in the
context. Normally you don't see this because this kind of error is
suppressed by Django. However, if you turn on
TEMPLATE_STRING_IF_INVALID, they show up and break the markup, e.g.
the column headers in the overview table.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Suppressed template errors in admin

[Ole Laursen]

On 25 Aug., 01:39, Russell Keith-Magee 
wrote:
> On principle, I have no objection to the idea of making the admin
> templates more robust in the presence of TEMPLATE_STRING_IF_INVALID;
> adding dummy values in the context sounds like a reasonable approach
> -- *if* doing this doesn't undermine broader error handling in the
> templates.

Yeah, sure. I don't think this is a big problem, a couple of places
tops. Thanks for considering this! I'm checking out the SVN release
right now, but actually the one that I've been bitten by so far has a
patch here (patch against current trunk below):

https://code.djangoproject.com/ticket/12136

Note that this was merged into a monster bug that had worse patches
(as far as I can tell), adding if-tags in the templates, and
depressingly never went anywhere.

Ole


Index: django/contrib/admin/templatetags/admin_list.py
===
--- django/contrib/admin/templatetags/admin_list.py (revision 16696)
+++ django/contrib/admin/templatetags/admin_list.py (working copy)
@@ -102,7 +102,10 @@
 admin_order_field = getattr(attr, "admin_order_field",
None)
 if not admin_order_field:
 # Not sortable
-yield {"text": text}
+yield {
+"text": text,
+"class_attrib": ""
+}
 continue

 # OK, it is sortable if we got this far

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Suppressed template errors in admin

[Ole Laursen]

Hi!

I have a project where setting TEMPLATE_STRING_IF_INVALID has been
invaluable in finding problems (for various reasons). The caveat
mentioned in the docs don't kick in here because no templates are
inherited from elsewhere. Except the admin which unfortunately breaks
down in some places.

I realize there's been a big discussion before about the merits of
error suppression. No need to repeat that.

But is it really not possible to fix admin in the few places it's a
problem? Those I've seen so far have one-line fixes where you add an
empty variable to the context. Seems harmless to me. I realize
cluttering the template itself with a {% if var %} is ugly. No need to
go there.

I don't mind writing a patch for the cases that bother me, but won't
do it unless somebody is willing to apply it?


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Decision for ticket #6362 - Remove blank spaces with strip when validating the data

[Ole Laursen]

On Jul 4, 4:52 pm, Jacob Kaplan-Moss  wrote:
> On Mon, Jul 4, 2011 at 12:13 AM, Ryan  wrote:
> > Umm... How about now?
>
> Sorry, but this isn't going to happen. I left more information on the
> ticket:https://code.djangoproject.com/ticket/6362#comment:43.

It sounds like you've never been bitten by strip bugs. Here's the ones
I've been through:

1. User inputs "foo " in some kind of identifier that is matched
verbatim when doing a DB query. User can now not login/find his stuff/
whatever. Or the error is catched by a check regexp and instead the
poor sod cannot create a user and don't understand why as the space is
invisible. EmailField and date/time fields used to have this bug:

https://code.djangoproject.com/log/django/trunk/django/forms/fields.py

2. User clears a field by selecting the text and hitting space bar
(why you would do that is beyond me, but some people do it). Now the
field is submitted as " " instead of "" which to Django is not blank.
Code that kicks in if the field is not blank now kicks in even though
there's no value. Something breaks.

3. User inputs "Foo " as first name, everything looks fine in HTML
because HTML strips extraneous spaces, but once you send the person a
text confirmation, you send it to "Dear Foo ," instead of "Dear Foo,".

The truth is that almost all char form fields intended for humans
should be stripped (do you think this is incorrect?), and it's easy to
forget to do it.


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Simple suggestion (forms and fields with dynamic parameters)

[Ole Laursen]

Hi!

I have an application where I need a field for selecting between some
predefined options that are dynamic up to the point where the form is
instantiated. So I've used a TypedChoiceField and overwritten __init__
on the form (not the field) to set the choices based on a database
lookup and some massaging.

This works well, but now I'd like to use the same field on another
form.

I thought I could solve this by defining a custom field, but the
problem is that on Field.__init__, it's far to early to capture the
choices. And the next call, clean(), is after the fact. Ideally I'd be
able to get called when the form is constructed.

What do you think about adding an empty form_constructed() to Field to
be called by BaseForm.__init__() after self.fields has been set?


Then a custom field could look like this

class MyField(forms.ChoiceField):
def __init__(self, foo, **kwargs):
self.foo = foo
super(self.__class__, self).__init__(**kwargs)

def form_constructed(self, form):
if self.foo:
self.choices = ...

I had a look at ModelChoiceField, and it seems like a lot of non-
intuitive code could be deleted from it with a hook like this?


Ole

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Need a little help on #6552: admin + GA = broken upstream caching on all pages

[Ole Laursen]

Hi!

If a core developer would take a look at

http://code.djangoproject.com/ticket/6552

and indicate whether the approach is the right one, that would be
great.

Short version of the story: using django.core.context_processors.auth
is the same as saying "all pages depend on who is currently logged
in", meaning a Vary: cookie header is always emitted on all pages,
meaning no upstream caching including reverse proxies like Varnish for
any pages if visitors happen to have unique cookies, which they will
with Google Analytics. And admin has a hard-coded dependency on the
context processor.

So admin + Google Analytics = no caching.

There's a simple suggestion for fixing things in the issue, and I
don't mind writing the missing documentation if there's consensus it's
the right approach. Maybe we can get it fixed in Django 1.2.


Ole
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: FileFields and file ownership

[Ole Laursen]

On Aug 11, 1:39 pm, Ole Laursen <o...@iola.dk> wrote:
>   1) a convenient file pointer for facilitating the upload machinery
>   2) a field for storing a file, just like storing it directly in the
> database except we put the data in the file system

No conclusion?

Here are some options

1. Apply the patch in 11663 more or less as is. Pros: no API breakage.
Cons: all FileField objects are leaking unless you do something.
Deleting an object with a FileField causes full table scan (unless you
add an index yourself).

2. Apply the patch, but turn on orphan deletion by default. Pros:
FileField is leak-free out of the box. Cons: API break if you were
relying on the garbage, deleting still causes full table scan.

3. FileField owns the file it points to. Pros: no leak, simple code,
no table scans. Cons: API break if you were relying on two objects
pointing to the same file.

4. As 3, but add option to be more in line with current behaviour. So
API break, but it's easy to fix.


Ole
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: FileFields and file ownership

[Ole Laursen]

On Aug 12, 5:08 am, Malcolm Tredinnick 
wrote:
> This is an assumption that isn't universally correct and it's
> implications appear to be vastly under-appreciated. One of the reasons
> we *don't* delete files automatically is that you cannot you know for
> certain, given no other information, that no other process on the system
> or elsewhere is still needing that file. You are implicitly assuming
> that the file upload through Django means that Django instance is the
> only thing using the file. Other Django installs could be using. Other
> processes could be. It's impossible to tell.
>
> So knowing when it is safe to delete files requires domain-specific
> knowledge.

Yes, well, exactly. I agree. :)

What I suggest is that we define this, instead of making implicit
assumptions. Either to be some kind of elaboration of the current
approach in Django, or simply that FileField is owning the file.

If FileField owns the file, then the semantics are easy to document
(and implement). If something else is using the file, you'll know you
have to do something about it. In my experience, that does not happen
often, but maybe Django could help a little with some configurability
just in case.

In the end, doesn't it boil down to what's the reasonable semantics
for a field for storing uploaded files? Am I missing some important
use case?

> Realise that uploading with the same filename isn't the only way to use
> up disk space. The filename is a pretty random string to use as a save
> thing and it takes about 35 seconds to write a script to generate
> hundreds of thousands of filenames for the same file if different
> filenames are the only impediment to DOS'ing a system.

Sure, it's not specific to the naming. The use case I am thinking of
is a simple user profile where you can attach a photo of yourself. If
you code that with Django and ImageField today, you've got a disk
leak, which seems counter-intuitive when you read that it's a file-
upload field.

I must admit that I was also very surprised to see yesterday that
deleting an object with a FileField causes a full table scan.


Ole

PS: Oh, by the way I really enjoy reading your blog from time to
time. :) It makes me wish I was a better writer.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

FileFields and file ownership

[Ole Laursen]

Hi!

There are a couple of bugs open/closed about what happens when you
upload a new file to a file field that already has a file:

  http://code.djangoproject.com/ticket/11663
  http://code.djangoproject.com/ticket/2983
  http://code.djangoproject.com/ticket/4339

Progress is currently halted because a design decision is needed,
maybe the problem is conflicting visions of what FileField is.

First: what happens right now is that the old file is left behind. If
the new file has the same name as the old, it is mangled so both can
stay.

As has been pointed out in 2983, this is all else set aside a security
problem because the old file is essentially garbage that when left
behind makes you vulnerable to someone filling up the disk (say on
shared hosting with few resources) by uploading the same file over and
over. So even if you check file sizes, you're not safe.

Here are two ideas of what FileField is:

  1) a convenient file pointer for facilitating the upload machinery
  2) a field for storing a file, just like storing it directly in the
database except we put the data in the file system

I think Django is currently the first. It won't let you overwrite
files (insists on mangling), it doesn't clean up the garbage, it does
a sort of reference counting so when the object is deleted, it first
checks if other objects with the same field is pointing to the file
before whacking it.

The implication of the second idea is a one-to-one mapping between
fields and files that Django will do everything it can to maintain. I
think the difference amounts to: always delete old files, don't go
through the whole table upon delete since if you messed with the
pointers, you pay the price (consider a table unique constraint
instead), and perhaps be a bit more careful so it's possible to
reupload a file with the same name to the same object without hitting
the name mangling code.

The documentation just says it's "A file-upload field." I think a file-
upload field would be better served by the second idea because you
don't have to think about ownership - Django's got it. What do you
think?


Ole
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Absolute paths in settings.py

[Ole Laursen]

On 30 Okt., 22:55, "Valts Mazurs" <[EMAIL PROTECTED]> wrote:
> There should be significantly greater number of Django projects that are
> deployed without any fancy stuff involved.
> Anyway, if someone wants to deploy his project as egg it would be easy to
> just change "PROJECT_DIR =.." as all other settings might stay the same.

Yeah, even without project dir auto-detection, it's better to
construct the absolute paths by concatenating with a PROJECT_DIR if
things are organized that way, DRY. Maybe it's plain, but the idea
didn't occur in my mind until recently. :)

The __file__ trick is a bit of a hack, I suppose Django usually has
some kind of idea of where things are located (or maybe not?).


Ole
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Absolute paths in settings.py

[Ole Laursen]

Hi,

There are a couple of things in settings.py that require an absolute
path. I don't know about you, but I tend to put all things related to
a project in subdirectories under the project directory. Thus all the
absolute paths have the same prefix, the project directory. Rather
than going trough the hassle of maintaining this by hand, I have begun
using the following snippet:

PROJECT_DIR = os.path.abspath(os.path.dirname(__file__))

then for instance

MEDIA_ROOT = os.path.join(PROJECT_DIR, 'media')

and

TEMPLATE_DIRS = (
# Always use forward slashes, even on Windows.
os.path.join(PROJECT_DIR, 'templates/'),
)

Would it make sense to put this or something like it in Django?


Ole

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---