Re: Proposed Field API additions

[Marty Alchin]

On Fri, Jun 8, 2012 at 8:45 AM, Jacob Kaplan-Moss wrote:

> Can't this be done by auto-discovering subclasses of Field
> (Field.__subclasses__)?


Unfortunately, __subclasses__() doesn't work down through the whole
hierarchy, just one level deep. So unless we plan to walk the tree to find
them all, some other form of registration would probably be better.

-Marty

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Extending templates dynamically

[Marty Alchin]

On Fri, May 6, 2011 at 12:25 PM, Christophe Pettus  wrote:
> Steal from the best! :)  One additional feature that they added was a dynamic 
> way of doing {{ extends }}.  Rather than specifying the tag in the template 
> source, the inheritance path can be specified directly in the 
> render-equivalent call.  This has proven to be quite useful for those times 
> that an inner template is used in multiple wrapper contexts.  Is this 
> something that might be worth investigating in Django?  Looking at the Django 
> source, the implementation seems quite straight-forward.

Does the current behavior of {% extends %} not do what you want?

http://docs.djangoproject.com/en/1.2/ref/templates/builtins/#extends

In particular, note that you can supply any template variable as the
argument to {% extends %} in order to dynamically inherit from
whatever template your view decides is appropriate. That said, if you
have questions about using {% extends %} in this way, it's probably
best to take it to django-users instead.

-Marty

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Problem with ``Model.objects.create``

[Marty Alchin]

On Tue, Jan 4, 2011 at 3:10 PM, Yo-Yo Ma  wrote:
> Simple question (A or B): Would you rather A) have a
> SomeDjangoException come up where you don't expect it and a visitor
> see a 500 page, or B) have your data ruined by a developer mistake?

Developer mistakes can ruin your data. Full stop.

That has nothing to do with Django, its ORM or the create() method
being discussed here. A developer mistake *always* has the potential
to severely cripple a website or much, much worse. Sure, we could
argue all day about whether this one particular method could add some
extra protections, but please don't try to kid yourself (or any of us)
into thinking that "fixing" create() will somehow make websites safe
from mistakes.

The fact of the matter is this: create() has a particular behavior,
and that behavior is documented. Sure, the documentation *could* be
more clear, but if we wanted to call out every potentially disastrous
mistake in bold with flashing lights and sirens, the vast majority of
the documentation would look like that, and nobody would ever read any
of it. Worse yet, it still couldn't possibly cover the full range of
possible mistakes, so anybody who did bother to read it would still be
wide open to a whole range of problems. It's our responsibility as
developers to understand the way our tools work and use them
accordingly. No amount of framework design will ever substitute for a
well-informed developer using it.

> Your data is ruined when your code's logic (the view) can no
> longer assume that the data's interface (the ORM model) is still valid.

That's just it: the data's interface (the ORM model) *is* still valid.
It's as valid as it's documented to be. An ORM is a device that
communicates between a database and an object. As long as there's
enough information for those two sides to talk to each other, a
conversation can take place. Anything that needs to happen beyond that
is, as you mentioned, your code's logic. You can encapsulate some of
that logic in a full_clean() method and call it separately to make
sure your objects are valid according your code's logic, but all the
ORM itself cares about is whether objects are valid according to the
database requirements.

Unfortunately, this is one of the many issues where "X doesn't work
the way I think it should" gets confused with "X doesn't work the way
it should" or even "X doesn't work." If you have a constructive
argument for how you think it should work that can be implemented in a
way that doesn't break compatibility with other people's code, feel
free to propose that and discuss it. But spinning around about how
"wrong" the behavior is currently doesn't help anybody.

-Marty

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Porting Django to Python 3

[Marty Alchin]

On Thu, Jan 14, 2010 at 2:32 PM, Karen Tracey  wrote:
> Martin's approach was single codebase where the 3.x version for execution is
> generated by 2to3, not single source for execution across 2.x and 3.x.  Thus
> I'm wondering if this difference is accounted for by 2to3?  If yes, then it
> is not necessarily a problem that would stand in the way of maintaining
> single Django source and supporting Python 2.x and 3.x simultaneously.

Yes, I was a bit less clear than I should've been. I was responding on
an assumption that the author was expecting a single codebase to work
with 2 and 3 without going through 2to3 in between. To my knowledge,
2to3 does handle all the syntactic issues between the two, but I just
wanted to make it clear that it's definitely not "pretty much the same
as supporting old 2.x pythons."

-Gul
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Porting Django to Python 3

[Marty Alchin]

2010/1/14 Łukasz Rekucki :
> It is possible to write 3.x code that is backwards-compatible with
> python 2.6+. There are some rough edges like, names of stdlib modules,
> instance checks for strings and some introspection details. In my
> opinion, it's pretty much the same as supporting old 2.x pythons.

In many cases, this is true, but there are other scenarios (certain
forms of exception handling, for example) where there is no syntax
that's valid in both versions. That's syntax, not just libraries and
functions. There's no way to even get a file to parse in both Python 2
and Python 3 in these situations. There are certainly places in Django
that will run into these, so we really can't have a single codebase
that's completely compatible with both branches.

-Gul
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Loading Backends

[Marty Alchin]

On Mon, Dec 21, 2009 at 10:46 AM, Simon Willison
<si...@simonwillison.net> wrote:
> On Dec 21, 2:22 pm, Marty Alchin <gulop...@gmail.com> wrote:
>> Looking over Simon's patch for signed cookies, I noticed that yet
>> another feature needs to write its own "load a backend from a setting"
>> function.
>
> Yup - and as I copied and pasted it from somewhere else I thought
> exactly the same thing. Is this kind of refactoring something we can
> do after the 1.2 feature freeze? If so I think it would be worth
> cleaning up at least some of this stuff.

Absolutely, this is completely separate from anything that should
affect the feature freeze. As Russ pointed out, there are a lot of
different of this backend loading stuff, so it'll be tricky to figure
out just how much can/should be factored out, how to deal with some of
the variations and how to recommend people do it going forward. In
fact, it's not even something that needs to happen by the 1.2 release
at all, if more time is necessary to get it right. It's less than
ideal at the moment, but I'd say practicality beats purity in this
case, to make sure we can get the goods out the door.

-Marty

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Loading Backends

[Marty Alchin]

Looking over Simon's patch for signed cookies, I noticed that yet
another feature needs to write its own "load a backend from a setting"
function. A quick search of the codebase shows at least four other
instances of this behavior (shown below), and it seems it's only going
to get more common going forward. Is there a way we can factor some of
this out into a utility? As it stands now, the author of each feature
needs to figure out which one to copy from, which means figure out how
they're different, even though many of those differences are just
stylistic.

django.contrib.auth.load_backend()
django.core.files.storage.get_storage_class()
django.template.loader.find_template_loader()
django.db.load_backend()

Database loaders are a bit of an oddball, since they load an entire
module's worth of classes based on the setting, but there's repetition
even there. Unfortunately, I can't dedicate much of my own time at the
moment to look into this, but I think it'd be worth doing before we
start adding even more features with configurable backends in the
future.

-Marty

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for feedback: django.utils.signed and signed cookies

[Marty Alchin]

On Mon, Dec 21, 2009 at 8:00 AM, Luke Plant  wrote:
> Rather than use 'settings.SECRET_KEY' as the default HMAC key,
> shouldn't we add a prefix so that any usage of SECRET_KEY can't be
> (potentially) used to attack other usages?  We discussed this a while
> back.  The new messages system uses:
>
>  'django.contrib.messages' + settings.SECRET_KEY
>
> for its HMAC key, which seems like a good convention to adopt. I don't
> know how strictly necessary it is, but I don't think it can hurt.

This was my concern as well, after the previous talks. The behavior
seems to be there, in the form of the `salt` argument, but I doubt
many people will use it. It might not be strictly necessary, but I
agree that it's probably a good practice that I would personally like
to see mandated by making that argument required, rather than
optional. It may seem like overkill, but if we're offering a way for
people to secure their applications, I think we should offer the most
secure option as the default, rather than simply the easiest. Of
course, the real-world importance of salting the hash would be best
addressed in the cryptographic review, but I agree that it's something
that should be noted prominently here, to make sure it gets a
once-over in said review.

> I also think the "Cryptographic signing" documentation should clearly
> note that developers should be very careful not to expose
> functionality that would allow users to retrieve signatures for
> arbitrary strings.  If they do, it will allow any other system which
> uses the same key for signing to be subverted.

This is particularly important in the case of cookies, because it's
all to easy to simply change the name of a cookie and have it send an
authenticated value. That's why my original implementation also used
the cookie's name as an additional salt in the value. The effect is
essentially the same as using the `salt` argument explicitly, and I
think it's an acceptable shortcut for the cookie case. We're already
asking users to supply the key, which is basically a namespace anyway,
so why shouldn't we just automatically use that as salt for the hash?
That way, if `salt` does become required for the generic signing case,
it can still be left out from the cookie case, since the key can play
double-duty. If there's a concern with collision with other keys, we
could always prefix it with 'django_cookie_' or something before
applying it, but I don't think that'd be a necessary step.

-Marty

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Best place for code that processes stuff from settings.py once (for logging implementation)

[Marty Alchin]

On Mon, Oct 12, 2009 at 12:09 PM, Jeremy Dunck  wrote:
> Also, I bet Marty knows this area well from his book work.

Actually, I didn't research much on the initialization process as a
whole, if there indeed is such a beast. I started with what happens
when Python actually encounters a model definition, which occurs after
settings and INSTALLED_APPS have been taken into account, which is
pretty much where the DevModelCreation article starts as well. Like
most people, I've generally deferred to James on the startup issue
here.

http://www.b-list.org/weblog/2007/nov/05/server-startup/

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

On Fri, Sep 25, 2009 at 10:42 AM, Simon Willison
 wrote:
> On Sep 25, 3:39 pm, Simon Willison  wrote:
>> While that makes sense for caching, I couldn't say if it makes sense
>> for signatures or not - when we sign something, will we always know
>> the point at which we want that signature to expire? I don't know.
>
> Here's a good argument for signing things with the creation-timestamp
> rather than the expiration-timestamp - it leaves the door open for a
> mechanism whereby historic SECRET_KEYs are stored. When we see a
> signed string, we can use its timestamp to decide which of our
> historic keys should be used to validate it.

But that only works for signatures that do in fact use a timestamp. If
the API makes timestamps optional, then there's still the question of
what to do for signatures that aren't stamped anyawy. In those cases,
I assumed the protocol would be to simply try to most recent
SECRET_KEY and work backward if the signature failed. Once all current
and deprecated keys have been tried (which should only be a total of
two, I would think), it would raise BadSignature at that point.

If we already have that behavior for non-timestamped cookies (and
correct me if I'm wrong on that point), I don't know that it's much of
an argument in favor of which timestamp to use. As long as there's not
a huge list of valid SECRET_KEYs to choose from, the overhead of
trying them each individually should be negligible, so I'm not sure
how much of an issue it would be.

I do agree though that you do have a point about there being possible
reasons for expiring the key at a different point after the fact, but
I'd argue that in those cases, you'd want to set an explicit
expiration time. In your example, you provided expire_after=24 * 60 *
60, but that wouldn't let you expire a cookie because of an event that
happened 2 hours ago. I would think you'd want to pass in a specific
time that cookies should be considered expired instead.

Of course, that then goes back to stamping cookies with their creation
time anyway, because otherwise you couldn't really do it right. If I
say "expire as of time X" I want to expire all cookies issued prior to
that point in time, while leaving any issued after that point intact.
The only way to make that decision is to know the time it was issued,
rather than when it was originally expected to expire.

I think we're getting a bit ahead of ourselves, though. There's
nothing stopping an application timestamping its own signatures and
validating them however they like, so we're really just discussing a
reasonable default here. Heck, since the timestamp behavior is driven
entirely within an already-signed value, we don't need to provide any
behavior at all if it's not a common requirement. I think it's a good
idea, but I'd like to hear from other people before I really stand
behind including it.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

On Fri, Sep 25, 2009 at 9:26 AM, Simon Willison  wrote:
> The API would look something like this:
>
 s = signed.sign('value', timestamp=True)
 v = signed.unsign(s, expire_after=24 * 60 * 60)
>
> A SignatureExpired exception would be raised if the signature was
> older than the expire_after argument (SignatureExpired would subclass
> BadSignature)

Very true. I had considered baking the timestamp right into the value
portion as well, but I was concerned about the extra space it would
take. It looks like it would only be 8 characters max, if encoded as
base64, which could be shortened to 6 if we strip out the == at the
end. I think I remember reading somewhere that those can be reattached
programmatically on the other end if necessary. All in all, the space
usage isn't that bad, and since it would be an optional component
anyway, it wouldn't add any overhead to the common case.

Would expire_after on the unsign just automatically imply
timestamp=True? There's been a lot of concern raised about parity in
the API, and it reads a little weird with the two different arguments.
I'm not sure it's a problem, but it's just a little funny.

Even though Ben rightly pointed out that we can't autodetect whether a
value is signed or not, I wonder if we could at least autodetect the
presence of a timestamp within a signature, once we already know that
the value is supposed to be signed. Essentially the unsigning code
could look for two different types of signatures, one with a timestamp
and one without. The timestamp would then be the actual expiration
time, rather than the time it was signed, so the API can look like
this instead (with a key added per prior discussion).

>>> s = signed.sign('key', 'value')
>>> v = signed.unsign('key', s)

>>> s = signed.sign('key', 'value', expire_after=24 * 60 * 60)
>>> v = signed.unsign('key', s)

This does make some assumptions about the format of the signed value,
but once we explicitly establish that the value is signed (by way of
passing it into the unsigning code), it's safe to make certain
assumptions about the format of the value. Or am I missing something
obvious here?

In the cookie case, it might be appropriate to use the combination of
an explicit expiration and signed=True to imply that an expiration
timestamp should be added to the cookie as well. I think that would be
the desired behavior, but I'm not quite sure.

Aside from some of the specifics of the cookie implementation, I think
we're getting close to an API here. I just hope we can get some input
from a cryptographer to make sure we get a solid implementation before
we go too far with this.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

On Fri, Sep 25, 2009 at 7:02 AM, Luke Plant  wrote:
> Suppose one part of an app signs an e-mail address for the purpose of
> an account confirmation link sent in an e-mail.  The user won't be
> able to forge the link unless they know HMAC(SECRET_KEY, email).
>
> However, suppose another part of the website allows a user to set
> their e-mail address (merely for convenience), and stores it in a
> signed cookie.  That means an attacker can now easily get hold of
> HMAC(SECRET_KEY, email), and forge the link.

This was something I was concerned with when I put together my app, so
I just added the name of the cookie to the signature as well, rather
than requiring some other explicit prefix. Since the two parts of the
application would need to use different names anyway to avoid other
problems, I figured the cookie name alone would be sufficient. If we
end up with something like response.set_signed_cookie() or
response.set_cookie(..., signed=True), the cookie name would be
available to the signing code internally, without any need to add in
some other key.

Of course, it'd still be worth documenting for the case of using the
signing stuff outside of cookies, because similar problems could creep
in elsewhere. I just think if we already have a name available, we
should be able to use it without any trouble at all. I wish there was
a way to sign the expiration as well, so people couldn't artificially
extend the life of the cookie, but since that doesn't come back in the
request, there'd be no way to validate it.

> So I propose:
>
>  - we add unique prefixes to the SECRET_KEY for every different
>   place it is used.  So for the e-mail confirmation link, we use
>   HMAC("email-confirmation" + SECRET_KEY, message)

I think this is good for everywhere the raw signing API is accessed,
perhaps to the point where that API can't even be used without a key
(a namespace, really - honking great idea!). Helpers on top of that
API could do without asking for that separately, if they can retrieve
a reasonable key from other forms of input, such as a cookie name or a
query-string argument name.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

On Thu, Sep 24, 2009 at 3:22 PM, Benjamin Slavin
 wrote:
>> response.set_cookie('key', 'value', sign=True)
>> - results in a Set-Cookie: key__Xsigned=value header
>
> Unfortunately, this approach won't work.
>
> A malicious client can just send "key" rather than "key__Xsigned" and
> you'll never know that the cookie hasn't gone through validation.
> This also means that you can't look for cookie values that match a
> specific format (ex: body.signature) because a malicious client could
> just drop the signature portion.
>
> As always, we can't trust the client. :-(

And you've just added another reason my followup email is invalid,
though I sent that before reading your response. (Take that, future
me!)

> 3) COOKIES as an intelligent object -- We can overload .get so we're
> doing something like request.COOKIES.get('foo', signed=True) -- I
> think this has the best chance at an interface that keeps a consistent
> feel. It's completely backward compatible, though it breaks the
> traditional expectation of what you can do via the `get` method on a
> dictionary.

I was wondering about this option as well, after I mentioned adding a
request.COOKIES.get_unsigned() method. I actually like this idea a
lot, personally. As you mention, it's backward compatible, and I'm not
sure it completely breaks the traditional expectation. After all,
aren't subclasses expected to customize the behavior of their parents?
It doesn't change any existing behavior, but rather just adds
something extra.

The one downside to using get() directly, as opposed to an altogether
new method, is that get() doesn't raise a KeyError when a value
doesn't exist. That means if anyone's wrapping request.COOKIES[key] in
a try block and catching KeyError, changing to the new code is more
than just a one-liner. I'm personally okay with this, but it's
definitely worth noting.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

> Also, does the name of a cookie factor into the cookie length limits?
> My reading of RFC 2109 says yes, but it'd be worth verifying, since it
> would cut down on the usable value space. With your compressed base64
> stuff, that's not as big of a problem, but still something to look
> into.

Also, just to throw this out there for the sake of compleness: could
the signature be stored under a separate name, rather than being
bundled with the original cookie itself?

Set-Cookie: key=value
Set-Cookie: key__Xsignature=signature_string

It seems like this could address a couple issues at once.

* There's a clear distinction between signed and unsigned cookies, so
request.COOKIES can be populated with only valid cookies

* The key/value pair remains unchanged, so things like Google
Analytics can happily ignore the signature if it was applied
unnecessarily (middleware is back on the table!)

Since there may be an upper limit on the number of allowed cookies,
though, doubling the number of cookies could present some very real
problems. RFC 2109 recommends allowing at least 20 cookies per domain
name, and it looks like at least Microsoft takes that to be a
maximum,[1] so it could present very real problems (middleware is back
off the table!).

I'm not sure how many cookies people use on a regular basis, and this
would only be for explicitly signed cookies, so maybe it'd be okay,
but it's flirting dangerously close to being completely unworkable.
Worse yet, it doesn't look like there's any predictable way to know
which cookies would get lost in the event of having too many, so this
may end up causing some very weird application errors if things go
wrong.

At least now it's been recorded for future reference. (Hello, future
me, looking up information on why we did things the way we did! Do we
have flying cars yet?)

-Gul

[1] http://support.microsoft.com/kb/306070

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

On Thu, Sep 24, 2009 at 2:54 PM, Simon Willison  wrote:
> Hmm... I hadn't considered that. I was thinking that the unsigning
> could be transparent, so by the time you access request.COOKIES['key']
> the value had already been unsigned (and if the signature failed the
> cookie key just wouldn't be set at all, as if the cookie never
> existed). But as you point out, this doesn't work because you can't
> tell if the cookie was signed or not in the first place.

The behavior you mention here is exactly what django-signedcookies
does, but it can only do so because it can blindly assume that all
cookies are signed, which as you pointed out, causes other problems.

> We could fix this with a naming convention of some sort:
>
> response.set_cookie('key', 'value', sign=True)
> - results in a Set-Cookie: key__Xsigned=value header

That seems pretty ugly on the surface, but it does confine the
ugliness to somewhere most people won't bother to look. One potential
problem is if someone wants to use __Xsigned in the name of an
unsigned cookie, but a namespace clash like that should be extremely
rare in practice.

Also, does the name of a cookie factor into the cookie length limits?
My reading of RFC 2109 says yes, but it'd be worth verifying, since it
would cut down on the usable value space. With your compressed base64
stuff, that's not as big of a problem, but still something to look
into.

> request.unsign_cookie('key') might be an option, as at least that
> reflects the set_cookie / sign / unsign API a bit. Not ideal by a long
> shot though.

>> try:
>>     value = signed.unsign(signed_value).decode('utf-8')
>> except ValueError:
>>     # Whoops! UnicodeDecodeError winds up here as well!
>
> That's a great argument against subclassing ValueError - I hadn't
> considered the unicode case. You're right, if anything it should
> subclass SuspiciousOperation instead.

I don't know if it's completely anti-ValueError, because a ValueError
subclass does still make some sense semantically, and since you can
catch more than one exception type in a try block, it's perfectly
functional. It's just that when lazy people blindly catch ValueError
without checking for something more specific as well, things can
break.

So it really just comes down to whether we expect people to be
thorough or lazy. Hrm. Yeah, I guess that answers it. :)

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Adding signing (and signed cookies) to Django core

[Marty Alchin]

I'm obviously going to weigh in here, having authored the signed
cookies app you mentioned below, but be aware I'm not a cryptographer,
nor do I have any personal use for signed cookies at all. I can
appreciate their value, but I'm not even using my own app in anything,
so if there are problems with it, I haven't experienced them
first-hand. I won't comment on everything, because I certainly trust
your thoughts on the matter. (Your comment on the Django book is what
prompted me to write the app in the first place!) I do have a couple
thoughts, though. So, anything I don't comment on, I agree with.

On Thu, Sep 24, 2009 at 1:18 PM, Simon Willison  wrote:
> It's also something that's hard to do correctly. At the sprints Armin
> pointed out that I should be using hmac, not straight sha1, for
> generating signatures (something Django itself gets wrong in the few
> places that implement signing already). Having a cryptographer-
> approved implementation will save a lot of people from making the same
> mistakes.

I have no idea how hmac differs from straight sha1, but this was
raised in a django-signedcookies issue as well, and i've since
integrated it. I'm with you on this; if somebody who knows better
recommends something, I'm inclined to listen.

> I think signed cookies should either be a separate API from
> response.set_cookie or should be provided as an additional argument to
> that method. I'm not a fan of signing using middleware (as seen in
> http://code.google.com/p/django-signedcookies/ ) since that approach
> signs everything - some cookies, such as those used by Google
> Analytics, need to remain unsigned.

I admit, I hadn't considered third-party cookies when I put it
together as a decorator. Client-side cookie access is likely
problematic as well, but that'll always be questionable anyway. You
can't validate a cookie in the client without divulging your secret
key and you can't just ignore the signature, because that defeats the
whole purpose. My app also provides a decorator, which might help in
some rare situations, but most of the time things like analytics
cookies will be in a base template and will always be included in
whatever view happens to have the decorator. I'm very surprised that
in all this time, nobody submitted a bug about the analytics problem.

> So the API could either be:
>
>    response.set_signed_cookie(key, value)
>
> Or...
>
>    response.set_cookie(key, value, signed=True)
>
> (I prefer the latter option)

I prefer the latter as well, for an added reason. I'd personally like
to invest some time in seeing if there are any other interesting
pitfalls in set_cookie() based on it deferring to Python's
SimpleCookie implementation. When writing Pro Django, I realized that
SimpleCookie expects everything to be strings, so #6657 came up with
secure=False resulting in a secure cookie after all. I don't know if
there are other such issues, but it might be worth looking at in
detail if we already have to add in signed cookie support.

And before anyone asks, no I don't think tying the signing behavior
into secure=True would be a good idea. Secure is meant to tell the
browser it should only send the cookie back over a secure channel,
such as HTTPS. While people who need secure cookies may also want
signed cookies, they're two separate ideas that I don't think would do
well mixed together. Of course, that leaves us with a potential
response.set_cookie(key, value, secure=True, signed=True), but I think
it's worth it to be explicit.

But I do have one other concern with either of these APIs that is at
least worth discussing: the disparity between setting a signed cookie
and retrieving one.

response.set_cookie(key, value, signed=True)
value = signed.unsign(request.COOKIES[key])

Mainly, unsigning a cookie requires importing and directly using the
signing module anyway, so the benefit of having set_cookie() handle it
transparently is fairly weak (unless, of course, I'm missing something
obvious). I'd rather just see the signing code made available and
well-documented, so that at least the change in code when adding
signed cookies is equivalent on both sides.

Another option would be to have request.COOKIES be a custom
dictionary, with an extra .get_unsigned(key) method that would work
like .get(), but validates the signature along the way. That behavior
can't be added straight to __getitem__() though, because that has no
way of knowing whether the cookie was signed to begin with.

These are the types of issues that led me to just implement it as a
middleware, so the API could be as dead simple as possible. With these
new issues in mind, I don't think dead simplicity is really an option,
so I'd rather fall back to being explicit.

 signed.unsign('hello.9asVJn9dfv6qLJ_BYObzF7mmH8c')
> 'hello'
 signed.unsign('hello.badsignature')
> Traceback (most recent call last):
> ...
> BadSignature: Signature failed: badsignature
>
> BadSignature is a subclass of 

Re: FileStorage and folders

[Marty Alchin]

Alexey wrote:
> I wonder, why is there no folder creation functions in Storage and
> FileStorage classes?

Because none of Django's features really required it.

> I ask this not because I don't know how to create it mysef. I'm just
> curios, why there are only one function for working with folders --
> listdir()?

For what it's worth, that actually wasn't intended to be there. I put 
that in when I thought that the initial file storage update would also 
support FilePathField, since that would need to be able to list a 
directory's contents. It accidentally made it into the final patch, and 
people immediately started using it, so it stayed.

> As for me,  function mkdir() should be there.

In the long run, I agree. I'm not sure it's much of a priority at the 
moment, but it's probably worth discussing for 1.2. Is there already a 
ticket out there somewhere for discussion? If not, would you mind 
creating one? Even if it doesn't make it in, it's easier to keep track 
of if there's a ticket for the request.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Template Caching

[Marty Alchin]

I won't speak to most of this, since Russ and Jacob have said it all,
but I do want to respond to one point.

On Thu, Aug 6, 2009 at 3:55 AM, Russell
Keith-Magee wrote:
>> By simply checking that the template has a ``render`` method in
>> ``get_template``, we get the added benefit of allowing users to write
>> loaders that return custom Template instances, or Template instances that
>> use an alternative template language like Jinja or Mako.
>
> I'd be particularly interested in hearing Jinja and Mako users weigh
> in on this particular comment. If we can improve the accommodation for
> external template engines with a relatively simple change to our
> rendering, I'm all for it.

Mike and I talked about this a bit in IRC before he submitted the
patch, because I had looked into the topic fairly thoroughly while
working my way through Pro Django. I actually wrote up a template
loader that wraps up a Jinja template in a Django candy shell, but of
course it didn't work because the engine expected templates to be
strings. Prior to engaging in a long, beastly workaround (which I
admit was funky, but it worked and I'm fairly proud of it,
regardless), I did make a small change that allowed template loaders
to return Template objects, which worked fine for both Django
templates and (wrapped) Jinja templates.

Of course, it was getting down to the wire on Django 1.0 at the time,
so I didn't submit the patch for review or anything, I just went ahead
with the workaround. So, I won't pretend to be from the Jinja or Mako
camp, but I can speak from experience that Jinja templates can be
achieved with just the small change Mike included in the larger issue
here. In fact, the interfaces are so similar, it would *almost* work
even without the extra wrapper around Jinja templates. The only real
trouble is that their contexts work differently, so it's not a perfect
match, but with the right template loader, the conversion is fairly
straightforward.

Anyway, I hope that perspective helps, but I also look forward to
hearing from people with Jinja/Mako/etc experience, to see if this
addresses any needs I may not have considered during my testing. If it
ends up working as well as it did for me last year, I think this one
feature alone will be well worth including, even if the caching stuff
has to be delayed because of concerns over template tag state.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: DurationField, request for help and comments

[Marty Alchin]

On Tue, May 26, 2009 at 2:16 PM, Jerome Leclanche  wrote:
> Right now, removing months and years is a matter of removing two lines
> from the patch. I didn't want to write a patch just to get told "it's
> lacking months/years!"; I'm also in favour of dropping it and
> following Python's style; but up until now no one except yourself and
> Yuri have given feedback, I had no reason to drop it.

My only beef here is that Yuri did mention it, with reasonable
justification in the ticket, and you did acknowledge that it could be
removed, yet it remained. I can understand the tendency to support as
much as possible until someone reigns you in, but once someone does,
it's best to listen. But, I've spoken my peace and it sounds like
we're on the same page now, so we move on.

> As for the current syntax, I believe with a small syntax helper
> somewhere next to the field, this would be and feel completely
> natural. 01:30:10 is a bit meaningless if you're not working in
> seconds. One of my application uses standard milliseconds durations,
> would I have to write 01:30:10.5 to have another 500ms? Would I need
> 1-2 other fields for milli/microseconds - which would make us go back
> to the former problem ("Many input boxes is not a nice user
> experience")?

In my view, yes, 01:30:10.5 (or 01:30:10.50 if you prefer) would
be the better approach. I will, however, freely admit that my support
for that syntax is based on nothing more than personal preference and
a vague idea of what I believe to be more common. I don't have any
research to back it up, but I don't expect any of us do in this area.
A quick search reveals that ISO 8601 is the relevant specification
here, which would leave us with something along the lines of
PT01H30M10S. In addition to being not-so-user-friendly, it seems ISO
8601 doesn't provide for anything smaller than a second, so we're back
to that same problem anyway.

> That's the way I was going until someone came up with the idea of
> parsing a timestring. Looking back, this is much more elegant, as long
> as the user knows what to do. The need for a javascript helper is
> entirely dropped since it's extremely easy to input any kind of value,
> and the user doesn't have to convert any kind of value. If the user
> wants 17 weeks, they just input "17w" instead of having to calculate
> 119 days. If they want "1500 seconds", same thing; so on.

I'll admit, my preferred approach does drop support for weeks, and if
you were to put in more than 24 hours, 60 minutes or 60 seconds, you'd
get a big, fat error message instead of the implicit calculations that
timedelta does behind the scenes. I still disagree with the notion
that the current syntax is "extremely easy to input any kind of
value," but again, neither statement is based on very much, so it's
likely impossible to say which is "better".

We're basically debating between the two representations allowed by
ISO 8601, at least in spirit. It seems an international panel of
experts couldn't decide on one over the other either, so they allow
both. Unfortunately, that's not really in keeping with Python
philosophy ("There should be one-- and preferably only one --obvious
way to do it."), so we need to pick one or the other. Whichever way we
go, it seems some of us will be Dutch, some of us won't. Such is life.

At this point, I'd say we should table the discussion on input syntax
until after 1.1 is out the door, so we can get some input from people
who have more sway in these matters. And yes, I do realize that I'm
the one who brought it up. ;)

> Also remember not everyone has javascript enabled.

I've seen this argument before, and it doesn't hold water. A
JavaScript helper is just that: a helper. It's not required to use the
field, it just makes it a bit easier for those who need a little help.
Those without JavaScript would just get text boxes without the helper,
just like the existing date and time fields.

> As for a wrapper TimeDelta class, that's up to Yuri to explain - I
> don't see the problem with it personally.

I wasn't pointing fingers at anybody in particular, I just don't see
that it adds anything beyond Python's own datetime.timedelta. Once
years and months are removed, it basically just performs unit
conversion and provides an output format in line with the new input
syntax. Python's timedelta already does the unit conversion, so that's
just duplicated effort. As for the output format, I'd rather see that
handled at the widget level anyway, so any other Python code that
deals with timedeltas doesn't get a shock when the output isn't what
Python normally provides.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit 

Re: DurationField, request for help and comments

[Marty Alchin]

Okay, I'm finally back around looking into this situation, and I have
to say I'm not thrilled with the direction it's currently headed. I've
never liked the "1h 30m 10s" syntax in any situation I've ever
encountered it, and I highly doubt that it will be meaningful to much
of anyone. That's personal opinion, of course, but I just don't see it
being a very good solution. My preferred approach would be to simply
separate the days from everything else and let two fields handle it. A
day field would parse an integer, while a time field would parse
standard time syntax ("01:30:10"), which is much more commonly used
for durations.

I'll freely admit that this isn't perfectly intuitive either, but I
think it's much more natural than the current proposal. I have a
slightly modified version of my original patch that works a bit better
now, and I'm working on a JavaScript helper like the current TimeField
uses in the admin, which should help relieve the intuitiveness
problem.

More specifically, can someone explain to me just what problems are
solved by adding a custom TimeDelta class? I can only see two benefits
to it: it accepts years and months as units and it maps the new syntax
into meaningful values. I've already spoken on the syntax issue, but
I'd like to point out that accepting years and months is a very bad
idea.

Basically, there are two ways to deal with years and months, depending
on what situation you're trying to use them in. The first is to not
try to flatten it out to a specific value until applied to a reference
date. This is the approach used by calendar applications, for
instance, when you apply a yearly repeating rule with a start date or
set an alarm for one month in advance of a given event. There are
known reference dates to work with, so years and months can be
calculated accurately.

The other approach is to assign some approximate value to years and
months, which is especially useful when *not* dealing with reference
dates. For example, you might use this approach to describe a
university degree program that takes four years or a prison sentence
that lasts 6 months. When speaking in generic terms, these
approximations are appropriate, but only because they're abstract in
our minds. They're never given an absolute length of time that can be
collapsed to days, hours, minutes or seconds. That can only happen
once a true date can be assigned.

What this ticket is doing, however, is trying to formalize the second
approach by explicitly stating some approximate values for years and
months. The intent is admirable, but it's ultimately doomed. What
you're left with is a "solution" that passably works for the second
situation, but outright fails for the first. When a custom TimeDelta
with years and months is applied to a known date, the result will
nearly always be incorrect. All it really does is formalize something
that shouldn't be formalized anyway.

The third option, which is used by Python and many other computing
systems, is to take years and months out of the equation entirely.
Sure, it's a reduction in functionality, but the benefit is that what
remains *actually works*. Weeks are the largest unit of time that can
be accurately collapsed to seconds on its own and applied to date
calculations in all situations. Thus, Python's timedelta supports
weeks as the largest available unit.

I saw this issue raised on the ticket, with the response of, "I just
wanted to get the implementation in. It's easier to remove than to
add-on." Unfortunately, there have been two more patches from the same
author and the year and month support remains. I hope I've explained
the situation well enough now that we can forget about years and
months in future patches.

I'll address the question of input format once I have a new patch up
and running.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Marty Alchin]

On Mon, May 11, 2009 at 8:45 AM, Armin Ronacher
 wrote:
> All bugs are fixed now except for #10788.  Now the problem with
> closing that one is that this one requires a design descision.  I
> updated the ticket accordingly for jacob or anyone else to decide on
> it.  My personal opinion is that I consider it bad design for the
> application to depend on the final filename of an uploaded file on the
> filesystem.  This makes it nearly impossible to create "atomic"
> uploads for unique filenames.

I've been thinking this one over a bit, while everybody else was
cleaning up the rest of my mess (thanks, guys!), and I have to agree
with you, at least to a point. I've only been able to come up with two
general uses for having the filename available at that point, and both
of them have a better approach available.

If you're using the filename to store it somewhere else, typically for
denormalization, it'd be better to do that post-save, since then you
know the record actually got saved in the database. Otherwise, you
might be trying to access the content of the file, which would be
better using the direct file access, which can often save yourself a
round-trip to the storage backend (which can be a big win for paid
storage solutions, like S3).

I won't completely discount the possibility that there's a legitimate
use case for having the final filename available in the pre-save hook,
but I just don't see it. I'll admit that it's a change to current
behavior, but I'm not sure there's a good way to get it working
reliably. I think it's possible to do it if we register our own
pre-save hook directly when the FileField is attached to the model
class, but even then, we can't guarantee that it's the first handler
to get registered, much less should we be relying on the ordering of
signal handlers anyway.

Unless somebody sees something I don't, I'm not sure there's a good
way to get it working, nor do I see much to be gained from doing so.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Posting to the wrong list (was: Re: Need Django Developer urgent)

[Marty Alchin]

On Fri, May 8, 2009 at 7:05 AM, Jacob Kaplan-Moss
 wrote:
> """
> Discussion group for Django developers. This group is used for
> discussion of developing Django itself, not user questions; Please use
> django-users for issues regarding using the framework, questions for
> the Django user community, outreach, etc.
> """
>
> If you've got any suggestions of ways to make that more clear -- other
> than renaming the list, which isn't going to happen -- I'd love to
> hear 'em.

I'm not sure if it'll actually avoid confusion, but could we just do
away with the first sentence entirely? After all, even before reading
the description, they know it's a Google group called
django-developers. Does the first sentence actually add anything
useful? That would allow the description to lead off with the actual
purpose of the group, so hopefully it would do some good.

As for Ned's suggestion of "core", I wonder if that would be better
suited in the description of what we develop. Maybe something like
"...discussion of developing the core Django framework..." perhaps?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Marty Alchin]

On Thu, May 7, 2009 at 3:43 PM, Armin Ronacher
 wrote:
> On May 7, 5:37 pm, Karen Tracey  wrote:
>> #10249: can't create consistent MRO (method resolution order) when assigning
>> a File to a FileField.
> This is fixed.

I was reading over the patch Alex mentioned in IRC (yay for
DjangoBot's logger!), and though it looks different than what I
intended, if it works, I'm happy (mostly).

>> #10300: custom storage backend can't get length of content to save.
> This *should* be fixed.  I can't test it, no access to S3.

It should be reasonably simple to set up a dummy backend that exhibits
the same behavior with size, just to test it out at least.

>> #10404: height_field and width_field for an ImageField don't always work.
> This is not really related to that and I have a hard time
> understanding that part of the ORM, but it should be fixable.

It's not explicitly related to the MRO and method stuff you and Alex
have been working on, but it's definitely related to the r9766
discussion, since it's caused by the delayed saving. I have a clear
understanding of the problem, but I don't have time to write it up at
the moment, so I'll try to catch you guys on IRC if you're still up
when I get home.

>> #10788: actual file name not known until after model save.
> That's the problem with the new implementation.  The final filename is
> decided when the file is actually saved.  The cleaner way is to use
> the assigned file and directly read from it when it's still in the
> temporary location or memory.

That only solves the problem if you're trying to access the file's
contents, which I'll admit is the majority case. Other apps may need
the filename for other reasons, though, and it was available in 1.0,
so the current behavior is backwards-incompatible. Of course, all the
uses I can think of would actually be better as post_save handlers
anyway, but the fact is that compatibility was broken, so we should
see if we can restore it.

Thanks for all the work you guys have done on this. We're not out of
the woods yet, but we're much better off than we were.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Marty Alchin]

On Thu, May 7, 2009 at 12:05 PM, Armin Ronacher
 wrote:
> I'm working with Alex on that right now here in Prague.  We have some
> branches on github related to that.  Basically the idea is to start
> with getting rid of some of the over engineering in the abstract base
> classes and make sure the subclasses call the constructors properly.
> Once that works we should be able to fix all the "method missing on
> temporary file" tickets that are currently just worked around.

Good to not neglect those, indeed.

> The fixing of the problems caused by the changeset will be covered by that
> branch as well.

I'm a little worried about how easy you're making it sound. Either one
of us is thinking way too much on the subject or the other isn't
thinking enough. :)

> We'll report back some status when we get the
> existing testcases passing again :)

I can't be on IRC for about another 7 to 8 hours or so, when I expect
you guys will already be done for the day, so if you have any
questions about background or want to bounce ideas around, I'm
available via email, and should be able to respond fairly quickly. We
could use this thread if you want more input, but I'd rather not spam
all the devs with minute details.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: r9766-related issues

[Marty Alchin]

On Thu, May 7, 2009 at 11:37 AM, Karen Tracey  wrote:
> I noticed the question of what to do about the r9766-related issues came up
> in the 1.1 thread so figured, in case it's helpful, I'll lay out my
> understanding of what/where these are.

You've been doing some great working tracking these issues down, so
your understanding of their current status is immensely helpful.
Especially since I've been slacking on these for quite some time and
could use the refresher.

> So far as I know there 4 open ticks remaining related to r9766.  Three are
> regressions so I believe something really needs to be done about them before
> 1.1; one I think is just a bug in the new function.  Personally I'd rather
> not revert the new function at this point (it's been available in the
> alpha/beta releases, it would be nasty to pull it at this point).  However,
> reverting the part of r9766 that delayed saving the file to the backend
> until model save (in the case where a ModelForm is used to create a model)
> may be necessary to fix the last of the four problems, unless we want to
> document that one as a backwards-incompatible change.

To be clear, I would definitely rather not revert the change if these
can be fixed in a reasonable time frame. I just didn't want 1.1 to
block for ages while we sort this stuff out, given that all the
problems were caused by a new feature (that is, r9766 didn't actually
fix any bugs, the ticket reference was really just a formality).

> #10249: can't create consistent MRO (method resolution order) when assigning
> a File to a FileField.  This is not a regression as I don't think you could
> do this at all before r9766. I added a patch to #10249 that fixes it, but it
> needs review from someone more knowledgeable about the implications of
> messing with class bases than me, particularly since I noticed it introduced
> another problem when combined with the patch for the next problem.  There's
> a note about that in the last comment on #10249 with possible fixes for the
> subsequent problem.  It's fixable, I just don't know what the right fix is,
> and I don't know what other side-effects there might be to this patch.

Yeah, this is the big one, in my opinion, because of the potential for
side effects. r9766 went in with too little thought on the subject
(entirely my fault, not passing blame), and I don't want to make that
same mistake twice. It's not a regression in and of itself, but the
feature is broken, all the same.

> (Marty also mentioned at one point he had some other ideas on how to fix
> this one but I have no idea what they are.)

I apologize for not writing my ideas down at the time, as you probably
would've been able to help refine them by now, but here's the gist of
what I remember. Right now, FieldFile is a direct subclass of File,
overriding some methods and adding others. What I'm planning to try is
to break that stuff out into some kind of FileFieldMixin that actually
provides and overrides methods, then have FieldFile inherit from File
and that new mixin. Then, in the code introduced in r9766, it would
just add the mixin to the class, hopefully avoiding any MRO
consistency problems.

It's still a bit more complicated than that, because #10300 below
highlights the problems with overriding methods on a base class that
may have behavior you weren't expecting. I think it's a good start,
though, and it's where I'll be focusing my efforts this weekend.

> #10300: custom storage backend can't get length of content to save.  This
> one is a regression.  It's also possibly easily fixed -- it's got a patch
> that works for the reporter of the problem, but it needs review.  I also
> don't remember if I added a test, I don't think I did.

Yeah, this definitely needs review, but mostly to see if there are any
similar problems caused by any of the other overridden methods
provided by r9766. Tests are definitely essential, but this one should
be relatively easy to test.

> #10404: height_field and width_field for an ImageField don't always work.
> Specifically, they don't work if they are declared in the model before the
> associated ImageField.  This is a regression,  resulting from the 2nd part
> of r9766, the delaying of the save to storage.  Possibly fixable without
> moving the save back to where it used to be, but I haven't investigated that
> at all due to the next problem.

I remember having a small bit of discussion on this in IRC a while
back, where someone suggested getting the image's width and height
immediately upon assignment. I'm not a big fan of that kind of side
effect when simply assigning to an attribute, but it's certainly less
strain than saving the entire file on assignment, which is essentially
what we were requiring before. I was just spelled .save(), which at
least made it clear that more was going on than simple assignment.

> #10788: actual file name not known until after model save.  This was
> reported by someone who has been relying on 

Re: Django 1.1 update

[Marty Alchin]

On Thu, May 7, 2009 at 8:34 AM, Jacob Kaplan-Moss
 wrote:
> Ugh, I really hate not being able to just assign files to fields. It
> just feels hacky and wrong to call instance.file_field.save(). It'll
> also break a bunch of code folks have written over the last few
> months. I know, no backwards-compat guarantee, but that's super
> annoying if we don't have to.

Yeah, I realize that. Just yesterday, somebody told me they already
refactored some code to take the new behavior into account, so I fully
appreciate that it'd be breaking an expectation that's already been
set for a few months.

> I've reviewed most of the open issues and it looks like they all have
> patches, so can't we just fix the problems and keep the feature?

I'm a bit concerned about the likelihood of getting them all addressed
in reasonable ways in time to not severely block the 1.1 release, but
I'll set aside some time this weekend to look at it all in more
detail. Karen's already done a good bit of legwork on several of
these, so hopefully the existing patches are better in combination
than I fear.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Django 1.1 update

[Marty Alchin]

On Thu, May 7, 2009 at 6:43 AM, Jacob Kaplan-Moss  wrote:
> Once this is done we'll be down to blockers for 1.1; many of us at the
> sprint are focusing on these. More help will be appreciated!

I just wanted to add a note here that may have some impact on which
tickets get punted vs. fixed in 1.1. I've been looking over the many
file-related tickets that are in the 1.1 milestone, and several of
them were caused by r9766, where I introduced support for assigning
directly to file fields, deferring the actual save until later.

While I still think that's a valuable feature, and will likely be
required in order to complete Honza's model validation work for GSOC,
it's really a new feature that has so far caused far more bugs than
it's worth. I'd like to recommend that r9766 be reverted, so we can
take the time necessary to get it right without causing all the bugs.

I'll be working for the next couple days on getting a patch together
for not only reverting r9766, but also adding tests for all the
tickets that are currently failing, to make sure not only that
reverting r9766 does the trick, but also that we don't make the same
mistakes next time the feature comes up for implementation.

I'll get the patch ready, regardless, but I wanted to bring this up
now, since it will likely impact how those tickets are dealt with
during this process. I'll need a committer to agree with me before it
can be done, obviously, but I think it's the right call at this point.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Bug or feature: pre_save used on FileField/ImageField doesn't work as in Django 1.0.2

[Marty Alchin]

On Mon, Apr 6, 2009 at 1:16 PM, Karen Tracey  wrote:
> We cannot know for sure what the file name is until it is saved to disk, as
> the save operation may tack on underscores when handling race conditions.
> Thus we cannot delay file save to a field pre_save routine and report the
> guaranteed correct file name in a pre_save signal handler, since the latter
> runs before the former.

Yes, very true. I wrote my reply in a hurry to get you some more
information, and didn't really think all the way through that one.

> I feel like I'm going around in circles thinking about this one -- is there
> a way out that someone else sees that I'm blind to?

Welcome to my world! :( I spent a long time on issues like this prior
to getting the new file storage system in place at first, and
apparently skipped much of that step when doing r9766. At this point,
I think the most useful approach is to take a step back and just look
at the high-level options we have available. Trying to determine the
name in advance is a no-go, because that would just make the existing
known race conditions much more prominent. I see two options left:

1. Write something to disk (maybe the whole file, as it was before,
maybe just a placeholder, as the _save() does now) when the file is
first assigned, so that we have a full, proper filename early on.
Then, if the model doesn't get saved, somehow roll that back so we
don't leave stuff lingering on the filesystem. This is currently
something of a "then some magic happens" approach, since I'm not yet
sure if there's a reasonable, reliable way to roll back a file save.
But I'd like to keep an open mind, so there it is. This would also be
preferable if we can detect transaction rollbacks, because there is
also #6456 to consider.

2. Save the file all at once, as soon as possible, and blatantly
document this behavior when model validation goes in. Then, if people
need to validate a model that has a file on it, they have two options:
validate the model *before* saving the file to it, so the model is
known to be valid and can all be saved at once, or (if they need to
validate something in the file, such as its name or contents), add
their own code to delete the file if validation fails. Or, I suppose a
third option is to ignore the lingering files until they suck up too
much space, requiring a manual purge.

I obviously hate to go with number 2, but if we can't come up with
something solid, I think it's the better approach, at least for now.
Documenting unfortunate behavior is certainly preferable to coding
even more unfortunate behavior.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Bug or feature: pre_save used on FileField/ImageField doesn't work as in Django 1.0.2

[Marty Alchin]

On Thu, Apr 2, 2009 at 11:45 AM, Karen Tracey  wrote:

> I think it's a bug, and I'm pretty sure it was introduced by r9766.  The
> setting of the actual file name that will be used (which pulls in the
> upload_to path and possibly tacks on some underscores if the uploaded name
> conflicts with already-present file(s) in the upload directory) is done by
> django.db.models.fields.files.FieldFile save().  This used to be called
> fairly early on from the django.db.models.fields.files.FileField
> save_form_data() method but r9766 moved that to much later, namely a
> newly-introduced pre_save() routine in FileField.  Since (from a brief look
> at django.db.models.base.Model save_base()) the field pre_save routine is
> called after the pre_save signal is sent, the real file name has now not yet
> been set when the pre_save signal is sent.

Ugh. Thanks, Karen, for looking into this and so many others that came
up after r9766. I've been fairly negligent on these, and it's starting
to cause some pretty big problems. r9766 may well mark the last time I
try to be clever. :(

> There are a couple of still-open tickets for other side-effects of r9766
> (#10249, #10300) but I don't believe either of the potential fixes for those
> would fix this issue, so this probably needs a ticket with the details you
> provided (an integrated testcase would be even better, though I don't know
> how much signal testing is done by the test suite).  Not sure what the right
> fix would be -- can just the setting of the name be pulled out of save() and
> done earlier by save_form_data()?  Possibly save() would have to also still
> ensure it's done, in cases where save_form_data() is not involved?  Can
> anyone with a better handle on r9766 comment on this (and #10249, #10300)?

For the record, #10044 (which was "fixed" by r9766) was entered as a
precursor to model validation. After some discussions with Honza, we
discovered that validating a model with a file would cause the file to
get saved to storage even if the model itself would never get stored
in the database. Our solution was to allow files to be assigned to
models outside the save() logic, and only store the file on disk when
the model actually hits the database.

Unfortunately, that proved problematic, because if you assigned an
UploadedFile to a model attribute, it didn't have the FieldFile
methods necessary to actually store it in the backend. I briefly
considered leaving that alone, and not relying on FieldFile when
finally saving the delayed file, but that created an inconsistent API.
After all, if you have a function that acts on models with files
attached, but you pass it a model with an *unsaved* file, it would
have a different set of methods than if the file had been saved.

Thus, r9766 includes the task of creating a new hybrid class whenever
a new file is assigned to a FileField attribute. This implementation
detail is the cause of #10249 and #10300 that you mentioned, but this
one stems from the simple fact that file saving is delayed as long as
possible. A related, thus-far-unreported (I think) issue comes up when
attempting to access width_field and height_field attributes on a
model prior to saving the new file.

I have a couple ideas to try out with regard to #10249 and #10300, so
feel free to hit me up on IRC if you'd like to discuss them, but I'm
not as certain how to approach this new one yet. There are certainly
band-aids that would allow the filename to be generated prior to
storing the file, but that wouldn't address the
width_field/height_field concern, nor any other potential future
issues that arise from delaying the file. I'd like to get to a point
where we're confident this sort of thing won't happen again, but if
the only option is to just keep putting out fires and they flare up,
it'll have to do.

As for addressing this one in particular, I think we'll have to add
something similar to the _committed flag, so we can identify whether
the filename has been processed or not. If it has, return it;
otherwise, generate it using upload_to when it's requested. The
trouble with just moving it up into save_form_data() is that if
someone assigns a file manually, save_form_data() won't get the chance
to generate a proper filename and we'll have a separate issue on our
hands. I hate having to add all these flags to determine what part of
the process we're in, but at a glance, that seems the most reasonable
approach.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: (Discussion) Ticket 9986: CACHE_BACKEND argument name doesn't match code

[Marty Alchin]

On Mon, Mar 30, 2009 at 1:25 PM, Kurt Grandis  wrote:
> cull_interval or cull_period would absolutely be the most accurate
> description in terms of the algorithm's operation. My contention is that
> parameter name would be confusing to all but those who have seen the actual
> underlying culling algorithm; what if we convert to a random cache culler?.
> All the user should have to worry about is, what fraction of cache do they
> want to have culled. cull_fraction seems descriptive enough working with
> some documentation clarification.

I ran into this same problem when trying to document cull_frequency in
Pro Django. I ended up adding a separate paragraph to explain the
actual behavior, so readers could better understand what value to
supply. I've brought it up in the past on IRC, and there seemed to be
some agreement that it wasn't optimal, but no better alternatives were
found at the time.

Personally, I'd rather see an argument name that tries to tell users
what to supply, rather than solely trying to be true to the algorithm.
Obviously, the goals should coincide, but focusing on the supplied
value brings another option to my mind: cull_passes (or something
similar, like cull_sweeps). Essentially document it as being the
number of times the cull operation would need to be performed to
completely empty the cache.

That way, we could describe what the cull operation actually does,
rather than how it works. If you want it to cull a lot, set it to a
low number so that fewer passes are required to clear the whole thing.
Setting it to a higher number would require more passes to do the same
job. Does that make sense to anyone else?

Of course, I realize that now is quite the worst time to suggest a
change, so I'm not suggesting it be changed now. Changing the name
doesn't fix any bugs in the behavior, nor is it backward-compatible,
so it's well out of scope for 1.1, and possibly the rest of the 1.x
line, but I'll leave that decision up to those who are responsible for
such matters.

I'll also warn that this type of thing easily spins into a bikeshed
thread, so don't be surprised (or upset) if somebody comes along and
shuts it down to focus our efforts on more pressing matters.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Non-primary key AutoField

[Marty Alchin]

On Sat, Feb 28, 2009 at 11:27 PM, NewSpire  wrote:
> First, this restriction is not there in the database.  Auto-increment
> fields do not have to be primary keys.  Imposing this restriction
> could cause a problem when trying to use legacy tables in a Django
> environment.  Is there a compelling reason for Django to impose this
> restriction?

Django is optimized for the common case. Sure, it may well be that
it's easy to accommodate uncommon cases as well, and when it's
reasonable, we should definitely do so, but the reason it hasn't been
up to this point is simply that not enough people need it. I highly
doubt any of the core developers were actively seeking to restrict
people from using non-primary-key auto-incrementing fields; Django was
simply written to address the vastly most common need.

> Second, is the use case where the primary key is not an AutoField but
> the record insert order needs to be tracked.  For instance, if the
> primary key is a phone number or UUIDField it would be useful to also
> have an AutoField so that records can be sorted in the order they were
> inserted into the table.  This is a real world problem I am faced with
> right now.

Call me crazy, but why would an AutoField help you with that? Since
you later explain that the records are created in satellite systems,
then aggregated, wouldn't you still run into problems ordering by
something that's generated in isolation? It sounds like what you
really want is a DateTimeField(default=datetime.datetime.now). That
way, you get orderable fields that can be combined from any number of
sources without worrying about collisions. Plus, you get the added
bonus of having a real-world value that you can use for more than just
ordering.

Mind you, I'm not trying to say that a non-pk AutoField is a bad idea
in itself. I'm just not convinced that this use case is a very good
justification.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: MarkupField

[Marty Alchin]

I haven't looked too much at the patch, so I can't comment much on the
implementation yet, but reading your description and the ticket, I'd
to offer some thoughts, based on some of my own thoughts I've had on
this in the past.

First, I don't think you actually addressed the question mentioned in
the ticket regarding the 3 fields. It seems the question was whether
there should be three attributes on the Python model instance,
regardless of how many columns are stored in the database. On this
note, though, I do have a thought: specify the markup type as an
argument to the MarkupField. You already do this with a
default_markup_type, but I don't see much use in having users specify
their markup type at the time they enter the text.

Essentially, it comes down to the developer choosing a markup type
during development, rather than a user choosing it in a form. Like it
or not, fewer choices generally makes for a better user experience,
especially since offering just one choice means you can supply some
help text alongside it. Trying to supply useful information for all
the various markup options, while also helping the user decide which
one is best for the (usually brief) text they want to enter ... well,
it just doesn't seem it would scale well.

Besides, specifying the markup type as an attribute of the model means
that you can do away with one of the three database fields. Plus, it
means you can specify any markup type, simply by supplying a callable
instead of a string. We make this same type of differentiation when
specifying the upload_to argument of FileFields. The implementation
becomes both simpler and more powerful, all in one swift stroke.

As for the issue Alex really did bring up in the ticket, I think
there's a question as to whether the different field types can be
contained by a single complex object, rather than individual
attributes on the model. Basically, if you had a model like this:

class InfoModel(models.Model):
title = models.CharField(max_length=255)
description = markup.MarkupField(formatter='markdown')

I would personally rather see something like this, when it comes time
to access that field's content:

>>> info = InfoModel.objects.get(id=4)
>>> info.title
u'Django'
>>> info.description

>>> info.description.raw
u'*The* web framework for perfectionists with deadlines'
>>>info.description.formatted
u'The web framework for perfectionists with deadlines'
>>> unicode(info.description)
u'The web framework for perfectionists with deadlines'

So essentially, one attribute contains both types of content, with the
default Unicode representation being the formatted output. Since
templates call __unicode__() by default, all you'd have to do is use
{{ info.description }} in a template to get it right. But you could
still use {{ info.description.raw }} to get the original content if
necessary. Or, optionally, you can pass that through a different
processor at display time if you wanted to, using {{
info.description.raw|textile }}

That's just one man's opinion, but hopefully that helps the discussion
a bit anyway.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: reason for missing readline() in uploaded files?

[Marty Alchin]

I suppose I should probably weigh in over here. For those who don't
know, I worked on much of Django's file handling, refactoring
FileField, adding storage options and standardizing the File interface
across models, forms, storage and uploads. If there are methods
missing, I'm almost certainly the man to blame.

First, I want to point out one thing: adding readline() is different
from supporting a catch-all for any methods we might have missed. The
arguments for each are quite different, and this thread has
transitioned from one to the other, making things a bit murky. With
that in mind, I'll slpit the rest of this email into two parts,
addressing each issue.

readline()

I'm not opposed to adding readline(). It's part of the standard file
protocol, and it can be added even without the underlying file needing
to support it. After all, iterating over the file already yields one
line at a time, so we could just stop after the first line is
retrieved. That said, you can also do that yourself, currently.

for line in file:
is_valid = validate_csv_format(line)
break

It's not the prettiest, but it's a reasonable workaround in the
absence of an explicit readline(). In fact, I'd much rather using a
Python approach to readline() that does something similar, rather than
simply deferring to the _file attribute. That way, we're more able to
make sure it's there for other objects that might not implement it
directly. As long as the file is iterable, and the iterator works the
way the protocol describes, a simple Python method will work just
fine.

def readline(self):
for line in self:
return line

Other methods

First, I'll be clear. Not all the missing methods were forgotten.
readline() probably was overlooked, but that's simply because nobody
had expressed a need for it at the time, but I want to make it clear
that Malcolm was right: not all methods make sense in all situations.
We had a lot of back and forth with seek() and tell(), for instance,
because our chunking mechanism for file access assumes content will
always only be read from start to finish. This caused problems with
the zipfile module, because ZIP files don't work that way.

One of the main things we're trying to keep in mind is consistency.
Much line Django's model lookup API should support all features (or as
many as humanly possible) across all available databases, I'd like to
make sure that the provided file methods work on as many file-like
objects as possible. That's made harder by the fact that there's no
finite list of supported file objects like there is for supported
databases. In the end, it may prove to be a losing battle, but I'd
rather not give up on it quite yet, and that's exactly what deferring
to the wrapped file object feels like to me.

In general, I'd like to see any new methods receive a "yes" to each of
the following questions before being implemented in core. I'm not a
core developer, so I don't have any power to enforce this, but here
they are, all the same.

1. Have people actually expressed interest in making it available?
2. Does it make sense in the context of a web application?
3. Can it be implemented in a way that's equally supported across true
files, storage-backed files and anonymous (StringIO-type) files?

As you can see, readline() passes all of these tests. Others, like
truncate(), don't. Just for reference, consider implementing
truncate() on files backed by a storage backend. In order to work
properly for things like S3, you'd need to actually read the specified
number of bytes from the beginning of the file (over the web), then
write them back out (again, over the web) in order to achieve the
documented behavior. That's hardly what I'd call "equally supported"
alongside native filesystem files which have OS functions to do that
directly.

Anyway, those are my thoughts on the subject.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model-validation: call for discussions

[Marty Alchin]

I haven't been following everything, but I do have a couple comments
to make here.

On Fri, Jan 23, 2009 at 8:04 AM, mrts  wrote:
> works both for forms and models with the proposed approach (all_values
> is model_instance.__dict__.copy() in model field validation case).

One thing to consider is that not all data is always available via the
dictionary, particularly with regard to models. Anything that uses a
descriptor (such as relations to other models) only stores a bare
minimum of information in the instance dict. The rest is retrieved
when accessed *as an attribute*, a fact which is entirely lost in your
example. I understand you're trying not to bundle it too much with the
way models work, so maybe this was intentional, but I do think it's
worth noting, just in case it's not intentional.

> Handling this specially for model instances would be really ugly:
>
>   def __call__(self, value, all_values={}, instance=None):
>   if instance is not None:
>   if hasattr(self.other_field, instance) and getattr
> (self.other_field, instance) == self.other_value and not value:
>   raise ValidationError(self.error_message, REQUIRED)
>   elif self.other_field in all_values and all_values
> [self.other_field] == self.other_value and not value:
>   raise ValidationError(self.error_message, REQUIRED)
>
> and entirely unnecessary under the all_values approach.

I agree that it's a bit ugly, but I'd hardly call it unnecessary,
given the possibly of using descriptors on models. Forms aren't
susceptible to the same problem, because their values are stored in a
true dict regardless. Offhand, I can only think of one way to get a
uniform syntax that works the way you seem to want it to, and that's
by bundling forms and models in some other objects with a unified
interface, designed solely for validation. Something like the
following (untested, coding off the top of my head):

class ValidatableModel(object):
def __init__(self, instance):
self._instance = instance
def __getitem__(self, name):
if hasattr(self._instance, name):
return getattr(self._instance, name)
raise KeyError(name)

class ValidatableForm(object):
def __init__(self, instance):
self._instance = instance
def __getitem__(self, name):
return self._instance.cleaned_data[name]

There could be more methods to be able to iterate over fields and
whatever other niceties you would want to include, but that way you
have a uniform dict interface to work with, regardless of which object
type gets passed in, and it still preserves the ability to use
descriptors on model attributes.

That said, I'm not particularly fond of that solution myself, because
I personally feel that models and forms *should* be treated a bit
differently; the field types for each are just too different in their
capabilities. I haven't been involved with this validation stuff
except for improving file support for it, but I'm not sure a unified
approach would be as big of a win as you claim.

-Gul

(and if I'm missing some obvious bit of history here, feel free to correct me)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Unicode filenames

[Marty Alchin]

On Thu, Jan 15, 2009 at 10:33 AM, Marty Alchin <gulop...@gamemusic.org> wrote:
> While working again with files for model validation, I realized (and
> confirmed with Russ, Honza and Alex in IRC) that the tests put in as a
> fix for #6009[1] don't actually prove all the behavior that ticket
> refers to. They prove that Unicode filenames come through fine from
> uploads, but that test doesn't actually save the file at all. The
> changes I'm working on for validation cause it to save the file now,
> and the second part of that ticket (dropping too many characters from
> the filename) comes back to bite us.

Correction: it was actually an error in the test that was causing the
failure; the problem just wasn't evident until the changes I put
through. I've put in ticket #10041 to address the problem with the
test.

> I'm not sure if get_valid_filename() needs to be as aggressive as it
> currently is, because a bit of light reading suggests that we should
> be able to work with Unicode filenames, now that Django passes around
> Unicode strings properly throughout. Alas, I'm not an expert in
> Unicode or file systems, so I don't know if I'm overlooking some
> obvious problem with opening the range up to include Unicode
> characters.

This is still a problem, though it's not the cause of any test
failures. The second part of #6009 is still a potential issue, and I'm
still not sure how best to address it. I'm not going to reopen that
ticket, though, nor will I open a new one until I know a bit more
about how to continue.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Unicode filenames

[Marty Alchin]

While working again with files for model validation, I realized (and
confirmed with Russ, Honza and Alex in IRC) that the tests put in as a
fix for #6009[1] don't actually prove all the behavior that ticket
refers to. They prove that Unicode filenames come through fine from
uploads, but that test doesn't actually save the file at all. The
changes I'm working on for validation cause it to save the file now,
and the second part of that ticket (dropping too many characters from
the filename) comes back to bite us.

I'm not sure if get_valid_filename() needs to be as aggressive as it
currently is, because a bit of light reading suggests that we should
be able to work with Unicode filenames, now that Django passes around
Unicode strings properly throughout. Alas, I'm not an expert in
Unicode or file systems, so I don't know if I'm overlooking some
obvious problem with opening the range up to include Unicode
characters.

I'd like to get this fixed one way or another before putting my file
patch into model validation, because it introduces a mostly unrelated
failure that I'd rather avoid. Does anybody with more experience in
this area have any thoughts on how we can go about it? For reference,
you can spot the failure by applying a simple patch to the test.[2]

-Gul

[1] http://code.djangoproject.com/ticket/6009
[2] http://media.martyalchin.com/6009-test.diff

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Signed Cookies Status

[Marty Alchin]

In light of the upcoming feature completion deadline for Django 1.1,
I'd like to take a few minutes to mention where the signed cookies
application stands right now and some of my thoughts about its
inclusion in django.contrib. The code itself is done. I made a few
minor updates to it last week to encourage people to use it as a
decorator if they don't need the full middleware, rather than
providing a separate utils package that would be too easy to
accidentally get things wrong. I think it's in as polished a state it
can be short of adding some major new feature I'm currently unaware
of.

That said, there's still one major concern: security. The whole point
is to try to guarantee that users can't tamper with their cookies and
use them on the site, so security's of primary concern. Unfortunately,
I'm not a security expert. The app was born from a comment Simon
Willison made on djangobook.com during the initial review phase,
mentioning that he uses an MD5 hash to sign cookies for OpenID
authentication without hitting the database. I basically just took
that description and made a middleware to do the job transparently.

However, Simon's been active on Twitter recently, questioning the
security of various hash digest algorithms, specifically with regard
to signing cookies. Again, I'm no security expert, but from what I've
read, MD5 is pretty much at the bottom of the barrel in terms of sheer
security, so I have definite concerns here. Simon, if you're
listening, what have you found out with regard to recommendations on
hashing algorithms?

The app can very easily be altered to use a different algorithm, but
with greater security comes greater hash lengths. And since cookies
are already limited in length by browsers, longer hashes present a
serious threat to the real-world usefulness of the app. If being
completely secure means only short values can be stored in cookies,
I'm not sure anybody would call it a worthwhile endeavor.

Perhaps most importantly, I just read over Jacob's recent post about
what it takes to be a contrib app, and I feel the signed cookies app
fails on two out of three points. It's certainly optional, which is
the only one it gets right. But even though I believe mine to be the
only Django-based signed cookies application, there are WSGI
middleware projects available as well that do the same job without
relying on Django at all, so I think it fails the "de facto standard"
test. The approach taken by the app is (I believe) a best practice,
but the app itself hasn't proven itself as the preferred way to do it.

But it's the other failure that concerns me most: it's hardly a common
pattern. Perhaps it would be more common if included in Django proper,
but it seems like nearly every time I mention the app, I get little
more response than a flurry of questions along the lines of, "but when
would you actually want to sign your cookies?" If the majority of
people can't even find reasons to sign their cookies at all, there's
little hope of it becoming a common pattern anytime soon.

So, I'm all for improving the app as much as I possibly can, and I'll
definitely continue to make it available, but unless it gets a major
boost in both security and popularity in the next few days, I don't
think it's prudent to include it in django.contrib. If others
disagree, I'll be glad to make the minor changes necessary to make it
a proper contrib app and submit it for review, but I personally don't
think it's the best idea at the moment. I'd much rather revisit it in
a future release once it's proven itself a bit better.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Custom Fields API Cleanup

[Marty Alchin]

On Tue, Dec 16, 2008 at 2:02 PM, David Cramer  wrote:
> Could we just use a lazy proxy for the value value? When it's executed, it
> could then just call the get_value_from_db method (or whatever).

I expect that the lazy approach is what most people would really want,
so I'd be fine with that. I didn't want to assume that's what you
wanted, though, since you may have a good reason to want your data
processed immediately upon loading, rather than on first use.

Personally, I'd recommend that if we go this route, we go with full-on
lazy loading. I'd recommend against updating ForeignKey and GIS code
to use it, though, since they currently use two different approaches.
Merging them would likely introduce some interesting gotchas for
people who are relying on the specific technique used behind the
scenes.

I'm still reserving a bit of judgment, though, until we get some code
together that proves useful in a good half-dozen different situations
(at least). The only thing worse than not implementing as much of the
feature as people want would be to implement all of it but make them
reimplement it because it's not flexible enough to suit their needs.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Custom Fields API Cleanup

[Marty Alchin]

On Tue, Dec 16, 2008 at 1:03 PM, Jacob Kaplan-Moss
 wrote:
> This seems like a good idea to me, and an easy one to implement; I'm
> all for it. Are there any backwards-compatibility concerns here? It
> doesn't look like it -- we're just talking about adding a new hook to
> SubFieldBase, right?

My only concern with this proposal so far is that I fear it'll fall
short of what many people will really be looking to do here. Every
time I've done this dance, I ended up something a bit more
complicated, because I also needed to delay the processing of the db
value until it's actually used. Basically, the same idea behind
implicit ForeignKey lookups and lazy geometries in GIS.

While I do realize that there are use cases where you'd want the
processing to take place up front, when first loaded from the
database, it seems like a slippery slope, since there are also many
cases where you'd want to delay that processing. Since David's
proposal doesn't address delayed processing, we'd have to make sure to
manage expectations so people don't cry foul when it still doesn't do
what they're looking for.

I've been all for expanding how data conversion in fields is handled,
and I think SubfieldBase is a great first step for common cases. I
just wonder how many of the less-common cases we plan to address
before we start getting into support code that's bigger and harder to
maintain than custom solutions. I'm fine with the line being pushed
back somewhere so we can get more support in core, but I do want to
make sure a line gets drawn somewhere.

Given that I've had a few encounters with this type of thing before,
I'll gladly help out where I can, but I'd also want to make sure
Malcolm weighs in, since I think he was intentionally conservative in
determining just how much SubfieldBase was responsible for.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Final(ish) 1.1 feature list

[Marty Alchin]

*sigh* I was afraid someone would say that. I didn't think I had to
state that I was indeed logged in at the time. Perhaps you have
WIKI_ADMIN priveleges? :) I can edit other articles on the wiki, just
not that one.

-Gul

On Tue, Nov 25, 2008 at 11:49 AM, Jeremy Dunck <[EMAIL PROTECTED]> wrote:
> On Tue, Nov 25, 2008 at 10:34 AM, Marty Alchin <[EMAIL PROTECTED]> wrote:
>>
>> The biggest issue is that we can't edit anything. There's no "Edit
>> Page" button, and when I manually add "?action=edit" to the URL, it
>> gives me an editor form, but upon submission, I see the culprit: 403
>> Forbidden (WIKI_ADMIN privileges are required to perform this
>> operation). Any hope of getting it opened up for the rest of us?
>
> If you log in, you can get an edit button at the bottom.
>  (Or at least, I could.)
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

<>

Re: Final(ish) 1.1 feature list

[Marty Alchin]

The biggest issue is that we can't edit anything. There's no "Edit
Page" button, and when I manually add "?action=edit" to the URL, it
gives me an editor form, but upon submission, I see the culprit: 403
Forbidden (WIKI_ADMIN privileges are required to perform this
operation). Any hope of getting it opened up for the rest of us?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Lazy-loading model fields

[Marty Alchin]

On Tue, Oct 14, 2008 at 12:34 PM, Jesse Young <[EMAIL PROTECTED]> wrote:
> A LazyTextField that manages a dynamic model behind the scenes sounds
> like a neat idea. That would certainly make the Django side of this
> easier to manage. (But of course I have a slight bias towards code
> that exists already -- has anyone actually implemented this?)

I don't think it's out there anywhere, but if you like I could whip up
the code for it in about 15 minutes (probably less). I just haven't
bothered because I don't personally need it.

> Anyway, I'm only trying to point out that inline lazy fields are
> *sometimes* useful, and that Django shouldn't force people to do it by
> splitting out fields into separate tables.

Damn it. Now you've gotten me thinking of ways to possibly manage a
field the way you want (one table, lazy loading, no data migration, no
patches to django, one-line change to the model). I'm 98% sure it's
possible, and I know of many of the things that would need to be
addressed to do so, but I really shouldn't be spending my time on it.
I probably won't be able to get it out of my head until I get it
working though, unfortunately. If I do write it down, I'll let you
know.

Of course, it would also have to come with big biohazard signs warning
people that there could (and likely would) be unforeseen side-effects,
particularly in the admin, serializers, modelforms ... actually,
pretty much anything that introspects your model would never see that
field. I'm not sure if that's suitable for your case, but it certainly
wouldn't be good enough for wide distribution.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Lazy-loading model fields

[Marty Alchin]

I won't pretend to know the whole story here, but I will take issue
with some of your points. I'll leave others to people who have history
on the subject.

On Tue, Oct 14, 2008 at 3:14 AM, Jesse Young <[EMAIL PROTECTED]> wrote:
> * If you split them out, you have to manually create one model every
> time you create the other model.

If by "manually create one model" you mean "type out a brand new model
definition in Python" you're flat-out wrong. I've done quite a bit of
work with generating models dynamically[1] and what you're asking for
could be made very simple using that approach. Rather than this:

thesis = models.TextField(lazy=True)

you could use this instead:

thesis = lazy_utils.LazyTextField()

And LazyTextField could create the associated model behind the scenes,
set up a descriptor for loading its contents only when necessary,
cache those contents so they only get queried once, etc. And all this
without a single patch to Django's core and without you having to
manually maintain another model.

> * If a fraction of your queries actually need those large fields, then
> you have to do a join, which can be expensive.

It doesn't address this, of course, but like I said, I'll leave that
to the professionals.

> * If your data access pattern changes after your application has
> actual data, it is much harder to migrate your data than to make a one
> line change to the application code (i.e., by adding lazy=True to the
> field definition).

Admittedly, the relational approach would require a data migration,
but that's life sometimes. There are plenty of tools now to make
changes like this relatively easy to deal with, and I expect a custom
LazyTextField could be written to give extra hints to any of them.

> * Maintaining more tables is more work than maintaining fewer tables.

Again, this depends on what you mean by "maintaining". At a raw
database level, yes, that's true, but if you're talking about the
Python level, you can make that extra table almost non-existent and
work with it as a regular field.

Anyway, I don't expect this information to solve your problem, but I
hope I've cleared up a few inaccuracies in your justification. Maybe
it'll help you with whatever solution you do come up with.

-Gul

[1] http://code.djangoproject.com/wiki/DynamicModels

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Python 2.6 abstract base classes

[Marty Alchin]

On Fri, Oct 3, 2008 at 10:39 PM, Eduardo O. Padoan
<[EMAIL PROTECTED]> wrote:
> IIRC, the file-ABCs are in the new `io` library.

Ah, indeed. I was looking for something called "File" or similar, not IOBase. :)

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Python 2.6 abstract base classes

[Marty Alchin]

On Fri, Oct 3, 2008 at 9:48 PM, Malcolm Tredinnick
<[EMAIL PROTECTED]> wrote:
> Think "interfaces". Given that you wrote the file storage stuff and are
> no doubt aware that we still have lots of fuzzy issues where something
> may or may not be able to be used where a Python file object is used,
> you can probably see that there are some uses for this.

Yeah well, I notice a conspicuous lack of any File ABC. ;) I have a
feeling they're still working out how many flavors to have
(ReadableFile, WriteableFile, SeekableFile, etc.) before including it,
but I don't know for sure, obviously.

> There may well be a plan. I, personally, don't have a plan right this
> minute because Python 2.6 is too new and is just one of many Python
> versions we support. Trying to predict the future in open source is
> generally going to be hard. All you can really do is put in a note
> saying "this is accurate as of this date" (something that's been
> unfortunately lacking in previous Django books, so please convince your
> editor to put something in of that nature).

Yeah, understood. Having 1.0 out the door helps tremendously, because
I can target a very stable version with a clear number to work from.
I'll make sure it's plainly labeled though. :)

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Python 2.6 abstract base classes

[Marty Alchin]

Looking over the new things in Python 2.6, I wonder if there are plans
to inherit from any of the new abstract base classes in Django. It
looks (to me) like it could be backwards-compatible, using an import
like so:

try:
from collections import Sequence
except ImportError:
# Fallback for Python 2.5
Sequence = type('Sequence', (object,), {})

Django obviously works just fine without these at the moment, so
there's no pressing need to do so, but if they catch on and 2.6+
libraries start checking for them, it might be beneficial if some of
our custom types, like QuerySets, inherited the appropriate classes.

I'm not really in favor of using or not using them, but I'd just like
to know if they're in the plan, in case I should add some notes in my
... documentation.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Denormalisation, magic, and is it really that useful?

[Marty Alchin]

On Tue, Sep 23, 2008 at 8:44 AM, Steve Holden <[EMAIL PROTECTED]> wrote:
> This appears to be a proposal to re-implement triggers inside Django.

I suppose it is. But then, perhaps triggers were a re-implementation
of application-based denormalization. ... The chicken? The egg? The
world may never know. :)

> I can see there are benefits if the underlying DB platform won't support
> triggers, but wouldn't triggers be the preferred solution when they're
> available? That way there is no chance that changes can be made outside
> the scope of the denormalization, and hence no need to recompute the
> denormalized values.

Perhaps the biggest downside is that distributed applications can't
make use of it, since triggers aren't cross-platform. Sure, you could
argue that denormalization is best suited for individual projects,
rather than distributed apps, but I could see uses for it. Imagine a
full-blown forum app that wants to denormalize its post counts.

Without some Python-based approach, all I could see is maybe adding a
cross-platform "create trigger" API (ugh) to Django, which an
application could then use to set up its triggers during syncdb.
Otherwise, something like that forum app would have to implement a
trigger for all available backends or just ship with instructions on
how to set it up yourself.

That said, you're right, I would think it's usually safest to put that
stuff directly in the db. That way, if you have other applications
using other languages or perhaps just other ORMs, everything's all
intact. I'm pretty sure I've seen arguments on both sides of the
issue, though, so I'm not sure there's any objective answer to it.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Denormalisation, magic, and is it really that useful?

[Marty Alchin]

On Mon, Sep 22, 2008 at 4:03 PM, Andrew Godwin <[EMAIL PROTECTED]> wrote:
> So, I'd love people's opinions about where I should take this - to a
> dark corner and hide it, to a separate app like django-extensions (which
> involves keeping the horrid hacks in there), into a separate app but
> with patches to core to make it less hackish (i.e. more signals), or add
> it as my 1.1 pony with the proviso that I'm happy to write all the code.

Personally, I think it's best suited for a django-d13n (that's d13n
for denormali(s|z)ation) app that stands on its own. That way, you can
provide a host of other options as you see fit, have other people get
onboard and provide whatever they use, and so on.

I think there's definite merit to what you're doing, but I just don't
see it ever being in core, unless it requires some freakish internal
changes. If you do need changes to core, or you can at least frame
them in a way that would provide wider benefits, you'll definitely get
a fruitful discussion on it, at least.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Model method decorators

[Marty Alchin]

On Thu, Jul 24, 2008 at 9:54 AM, zvoase <[EMAIL PROTECTED]> wrote:
> Developers,
> I've just looked at the latest revision (r8068) for Django and there
> are a couple of things I'd like to suggest.

I've already replied to part of this email, but I'd like to take a
minute to also reply to the other portion, now that I've taken the
time to read through it.

> I propose the addition of 4 decorator functions, to
> ``django.contrib.admin`` (or a relevant submodule), which could be
> used to decorate functions so that these attributes could be set
> automatically, like so::
>
>class MyModel(models.Model):
>...
>
>@short_description('Custom method...')
>def custom_method(self):
>return something()
>
> I feel that this way is far more Pythonic, and makes code look a lot
> nicer. I've put some Python source at the end of this message which
> demonstrates the idea, and defines a generic attribute-setting
> decorator-making function which allows more decorators to be created
> if the admin changes in the future.

This sounds awfully familiar, and I don't know why. Oh wait, yeah I
do! I proposed this very thing about two years ago. At the time I was
more focused with just the alters_data attribute, but the idea is the
same. Please take a quick read through the comments on the ticket[1]
to see why this has been rejected in the past.

-Gul

[1] http://code.djangoproject.com/ticket/3009

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Last call for 1.0 alpha

[Marty Alchin]

On Mon, Jul 21, 2008 at 11:37 AM, Tom Tobin <[EMAIL PROTECTED]> wrote:
> Gah, I really haven't had time to do the
> move-code-out-of-__init__-modules dance.  I'm fairly certain that
> it'll be completely backwards compatible (via re-importing back into
> __init__), though, so I'm not too worried; I got a tad anxious about
> it a couple of weeks back, but I can't come up with a scenario that
> would break existing code.

I certainly hope 1.0 alpha isn't about hitting the
backwards-compatible limit yet, because if it is, #5361 has failed to
make the cut in time. The way I understand it, we still have some time
to introduce mild incompatibilities in 1.0 beta, when it'll hit a hard
freeze. jacob, is this correct?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: API change in set_cookie for 1.0?

[Marty Alchin]

On Tue, Jul 15, 2008 at 7:49 PM, Jeremy Dunck <[EMAIL PROTECTED]> wrote:
> I was using HttpResponse.set_cookie for the first time and annoyed
> that expires has to be a properly-formatted string.
>
> Why not take a datetime and do the formatting in the function?
>
> Obviously this is a breaking change, unless we have an isinstance for
> backwards incompatibility.

When I looked into set_cookie a while back, I was surprised how much
was just being passed verbatim to SimpleCookie, which ended up causing
some API confusion with secure cookies as well.[1] I don't know if
it's worth looking at other attributes, but there are probably some
other areas the API can be improved as well.

-Gul

[1] http://code.djangoproject.com/ticket/6657

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Add a helper function in UploadedFile's API for writing files on disk?

[Marty Alchin]

On Fri, Jul 11, 2008 at 4:14 AM, Julien Phalip <[EMAIL PROTECTED]> wrote:
> Not sure if we're talking about the same thing here. What I ask is if
> it'd be a good idea to have a generic method for UploadedFile which
> lets copy/move the file to where ever you want. These method would
> have to be overriden by InMemoryUploadedFile or TemporaryUploadedFile,
> but would allow to have something as simple as:
>
> uploaded_file.move_to("some/path/")
>
> I agree with Ivan that it is a common operation and that having it
> core in the API would avoid some boilerplate code, and as result that
> would prevent potential mistakes by the developer.

Okay, two things:

First, Django does now contain the code you're asking for, at
django.core.files.move, which performs memory-friendly file moves in a
platform-independent way.

Second, file uploads are really just half of the equation. I'm
currently working on a patch for file storage, which will handle the
other side of it. Right now, yes, the focus has been on getting the
file uploaded in a sane manner, but once the other half lands (I'm
nearly done with a new patch for it), it'll be dead simple to save
uploaded files without wasting memory *and* without caring what
backend you're using.

Here's an example of what the API would probably look like:

from django.core.files.storage import default_storage

default_storage.save('path/to/files/%s' % uploaded_file.name, uploaded_file)

That API may change slightly before it gets committed, but behind the
scenes, that would use shutils or os.rename, or chunk directly to S3
or whatever is most efficient for what you're trying to do.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Test Suite on a mac

[Marty Alchin]

On Thu, Jul 10, 2008 at 3:23 PM, Jeff Anderson <[EMAIL PROTECTED]> wrote:
> So I'm a linux user, but I have a MBP. I was playing with the django test
> suite a while back, and some test failed on my mac but would succeed on any
> of my linux machines. After discovering the django buildbot, I thought it
> might be useful to have a buildbot that runs on a mac.

I've wondered if it would be possible to set up a buildbot on a
Windows machine as well, since I recently found a bug[1] in our tests
that would've been caught earlier if we had one. I don't have access
to a box that could be dedicated to it, though. And of course, the
real trick is whether separate buildbots can all report their results
to the main instance, so they can all show up in the same place.

-Gul

[1] http://code.djangoproject.com/ticket/7696

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: One more issue with file storage

[Marty Alchin]

On Thu, Jul 10, 2008 at 2:52 PM, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> Sorry for the late reply. I meant to look at this awhile ago but it
> slipped off my plate for a bit. Looking at the latest patch
> (filestorage.16.diff) it does look like a similar approach the main
> differences coming from my need to monkey patch this functionality
> over a stock Django install. It looks to fit the bill though and
> anything I don't have to support myself is a bonus :). Can you tell me
> if your implementation is on schedule to be completed by the 1.0
> release?

At the moment, it looks good. Now that file uploads have made there
way in, I've been working on updating the patch to take those changes
into account, and just last night I got a working test suite. There
are still a few things I need to do to make it more consistent with
how uploaded files work, so it's not really ready for a new patch just
yet, but I hope to have one ready this weekend or early next week.

The issue mentioned in this ticket is the only outstanding question,
but like I've said before, I'm not considering this a blocker to
inclusion in 1.0. Adding support for per-field storage configurations
would be just adding an option, so it'd be backwards-compatible and
can be held off until the big push is over and things settle down a
bit. I'll see what I can come up with before then, but it's not
critical.

> The one main difference with my approach, and a reason for the
> metadata dict, was the use of UUIDs for filenames. After playing
> around automatically renaming files (which truthfully leaves a bad
> taste in my mouth) and allowing for overwrites, etc. I decided to
> identify files with a globally unique "key", or UUID (which would
> serve as the filename when saved to disk) and store the original
> filename along with additional meta data separate from the raw "data".

I guess I can understand this, allowing duplicate filenames without
adding numbers or underscores on the end of them. I'll admit that it'd
be difficult to pull this off as a default behavior, and bolting it on
aftermarket would probably look pretty ugly, but I expect it'd still
be possible. Once I get the new patch ready, I encourage you to take
it for a spin and see if what you're doing gets any easier.

> This was also influenced by the Amazon S3 API which was going to be
> implemented as an alternate strategy for this project.

There's also work being done on an S3 storage system that'll be dead
simple to drop in once my file work lands. I don't have a link for it
offhand; David, if you're listening, can you point the way to your
repo? It's gonna be some good stuff.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Will oldforms completely go after 1.0?

[Marty Alchin]

On Thu, Jul 10, 2008 at 12:35 PM, Adrian Holovaty <[EMAIL PROTECTED]> wrote:
> It'll be a badge of honor/experience. "I used Django back when forms
> was called newforms!"

Yeah, and everybody at DjangoCon can have a little icon on their name
badges that indicates how many backwards-incompatible changes they've
gone through ... and for those of contributing, how many of those they
caused.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: UploadedFile in cleaned_data

[Marty Alchin]

On Thu, Jul 10, 2008 at 4:46 AM, Ivan Sagalaev
<[EMAIL PROTECTED]> wrote:
> I've seen already two people highly confused with this thing. After
> successful validation of a form with file upload
> form.cleaned_data['fieldname'] contains an instance of a wrapper --
> newforms.fields.UploadedFile. People confuse it with the real
> core.files.uploadedfile.UploadedFile which is accessible through
> form.cleaned_data['fieldname'].data
>
> If I understand right the only point of a wrapper is holding an original
> name of a file that was uploaded. I think it would be better to get rid
> of the wrapper, put a read UploadedFile in cleaned_data and add
> "original_filename" as an optional parameter to it. Thoughts?

This was reported in #7614, and fixed in [7859], so if you get a new
copy of trunk, it should be all set.

-Gul

http://code.djangoproject.com/ticket/7614
http://code.djangoproject.com/changeset/7859

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: UploadedFile API changes (was: Is it UploadedFile.chunks() or .chunk() ?)

[Marty Alchin]

On Thu, Jul 3, 2008 at 12:22 PM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
> I just talked about this with Mike on IRC; I think chunk() is
> non-obvious and we're gonna change it to chunks(). Hopefully nobody's
> written so much code in the last week to be pissed.
>
> Also, we're going to change UploadedFile.file_name to
> UploadedFile.name to more closely match the built-in file API.
>
> Sorry about not thinking this through fully before committing, but
> it's gonna be better this way.

Well, as long as the floor's open, I'd like to make sure file storage
code is consistent. Currently, it uses filename instead of file_name
or just name, and it also uses size instead of file_size. I assume I
should go with name instead of filename, but should I use file_size,
or should the upload stuff use size?

On a related note, when merging #5361 with the new trunk, should the
file-related code, such as moving files and whatnot, be moved into
django.core.filestorage.filesystem? Also, should the code in
django.core.filestorage be moved over to the new django.core.files
package instead?

So many bikesheds, so little time.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Ticket #7591: Authenticate By Email Support

[Marty Alchin]

On Wed, Jul 2, 2008 at 2:56 PM, George Vilches <[EMAIL PROTECTED]> wrote:
> On Jul 2, 2008, at 1:24 PM, Paul Kenjora wrote:
>
>> I understand the resistance but you've got demand, you've got a
>> willing developer, and you've got a clean fix that significantly
>> improves the adaptability of the framework.  What better reason
>> would you need?
>
> Someone who has a proven history of contributions and maintenance of
> the core framework for a significant period of time.
>
> 
>
> Providing the addition as a 3rd party framework allows the community
> to vet your work and decide whether 1) it's worth inclusion, and 2)
> that you have the stamina to maintain it in Django for a long while to
> come.

I'll also add that appealing to the community is often a much better
way to get feedback on improvements that can be made. My first
contribution, signed cookies,[1] started as a patch in Trac, that
ended up moving to its own application, at which point I got a much
better response of people using it, finding bugs, suggesting
enhancements, etc.

My second, dbsettings,[2] started as its own project, then had the
possibility of being included in trunk, so I immediately started
making changes to satisfy the immediate onslaught of suggestions. As
time went on, I came to regret that, as sitting back and riding the
normal community support wave would've made for a much better, more
usable application.

I think the main issue is that when something's sitting in trunk, or
even in Trac, there's something of a bureaucracy, whether real or
perceived, where the core developers are the gatekeepers of all
decisions. People are quick to identify bugs, but given how long some
decisions can take in trunk, people are less likely to speak up about
suggestions or improvements.

On the other hand, third-party applications in general have a better
track record for acknowledging and responding to feedback much more
quickly, and by more people. This helps get everything refined and
polished fairly quickly, and with a high level of quality.

And, as has been said before, if you write a good application, support
it well, make it consistent with how Django itself works, and help a
lot of people with it, there's a good chance it could be considered
for inclusion into Django itself later on. Certainly a much better
chance than if you just throw it in a ticket and ask why it hasn't
been committed yet, anyway.

Jonathan Buchanan's django-tagging[3] is a prime example of this, and
will probably be one of the first considered for django.contrib after
Django 1.0 hits store shelves. It's already on the shortlist of
applications that many developers install in every new environment
they work with, and once Django itself settles down, it'll probably
work its way in fairly quickly. Mind, though, that I'm not one of the
people who makes those decisions, I've just been observing
conversations for quite a while.

-Gul

[1] http://code.google.com/p/django-signedcookies/
[2] http://code.google.com/p/django-values/
[3] http://code.google.com/p/django-tagging/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Ticket #7591: Authenticate By Email Support

[Marty Alchin]

On Wed, Jul 2, 2008 at 1:24 PM, Paul Kenjora <[EMAIL PROTECTED]> wrote:
> Reasons why an alternate backend or view is not good:
> 3. Maintaining my own back end, means others are doing the same, again poor
> re-use.

Not if you:

1. Write it well
2. Distribute it freely
3. Make its presence known

You'll find people in the Django world are quite happy to use
third-party applications.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Community representation, or, #django user "Magus-" needs to go far away

[Marty Alchin]

On Wed, Jul 2, 2008 at 12:55 AM, Kenneth Gonsalves
<[EMAIL PROTECTED]> wrote:
> actually this debate belongs on django-users (which is why I put in
> my 2 paise) - I think if taken there we can get a real measure of the
> pros of his help against the cons of his curtness at times (I
> wouldn't call it being rude)

Or we could all just stop trying to deal publicly with what should be
a private issue.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: More secure user password reset

[Marty Alchin]

On Fri, Jun 27, 2008 at 8:31 PM, Simon Willison <[EMAIL PROTECTED]> wrote:
> I've got code for this lying round which I'd be happy to donate if
> people agree this is the right approach.

I personally much prefer this approach. I've worked in a couple
communities where personal attacks were quite frequent, and a common
tactic was to claim a password was lost on someone else's account. It
didn't give them access to the account in question, but it would
adequately lock the person out if they happened to visit the site
prior to checking their email.

Of course, sites like that also tended to have password change forms
that accepted GET requests, and didn't have sufficient XSS protection.
As you can imagine, wackiness ensued.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Call for testing: streaming uploads (#2070)

[Marty Alchin]

On Thu, Jun 26, 2008 at 3:14 PM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
>
> Hi folks --
>
> As far as I'm concerned, #2070, adding large streaming uploads, is
> done. I'd like to get some public kicking-of-the-tires before I push
> the change to trunk (which won't happen before Tuesday: I'm taking a
> long weekend off).

Yay!

I can't help too much with most of your needs, except possibly coming
up with a custom upload filter just to figure out how it works.
Instead, I'll work on getting the file storage patch to play nicely
with it, so that once it's merged, we're already a step ahead on
another item in the list.

I don't expect it to take much work, but I've been holding off until
2070 either merged or hit a stage like this, so I wasn't working with
too much of a moving target. I'll stay in the loop on any changes that
are made as a result of this testing, and get us in better shape once
it lands.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Django book seeking Tech Reviewer

[Marty Alchin]

Hey all,

First, I apologize for posting this on django-dev, since I know it's
not development *of* Django, but I didn't want to add to the noise on
django-users, and most of the people I'm hoping to reach are frequent
readers of this list. No need to flame, I got it.

As most of you are no doubt aware, I'm writing Pro Django for Apress,
and I'm desperately in need of a technical reviewer. I had one, now I
don't. I don't want to get into it.

It's a paying gig (though I don't personally know how much; Apress
writes the checks), and I need someone who knows Django pretty well,
or at least isn't afraid of diving in. A big part of the job is making
sure I'm writing what's actually happening in Django, and to verify
that all my code samples run correctly.

I can't speak for exactly what kind of time commitment it will
require, but I can confidently say you'll need plenty of time. I know
it's tough this summer, with the lead-up to 1.0, and I don't want to
detract from that, since it's so important to all of us, but if
anybody can spare the time to help out, it may well mean the
difference between the book getting published and ... well, I'd rather
not consider the alternative.

If you're not available, but you know someone who is or who might be,
I'm accepting recommendations as well.

Please contact me off-list, as I don't want to clutter this up any
more than I already have.

http://prodjango.com/

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Django 1.0 roadmap

[Marty Alchin]

On Mon, Jun 16, 2008 at 2:17 PM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
>
> On Mon, Jun 16, 2008 at 12:43 PM, Marty Alchin <[EMAIL PROTECTED]> wrote:
>> * Leave them as they are, and just tell whoever commits #5361 to
>> reference them in the commit message.
>>
>> * Move all of the to the beta milestone, since they are indeed being
>> addressed, and also reference them in the comment message
>
> Do both of these, and also tag with with a common tag (see, e.g., the
> qsrf-fixed or nfa-fixed tag). I'd suggest "fstorage-fixed" or
> something. Actually, if you tag them first it's easy to move them all
> to the milestone -- I'd like the milestones to reflect the actually
> ticket count (even for dups of this nature).

Tags were already done, but I've now moved them all into the beta
milestone as well. I hope I didn't overstep my boundaries here, but I
also added a note on the VersionOneRoadmap to explain the related
tickets and how the tags should be used.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Django 1.0 roadmap

[Marty Alchin]

On Mon, Jun 16, 2008 at 11:15 AM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
> As requested, I've also added milestones for 1.0 alpha, beta, and
> final (as well as a "post-1.0" catch-all). Triagers, feel free to use
> these milestones thusly:
>
> * Any *feature* tickets related to the maybe list get put in the beta 
> milestone.

One question about this, since I'm probably best positioned to manage
the tickets related to file storage. #5361 is already added to the
beta milestone, but what should be done about the 15 other tickets
that are related to it[1]? Given how much code #5361 already has to
move, modify and replace, I've been wrapping some of the fixes up in
that one patch (keyword fs-rf-fixed), and the others will be addressed
by the documentation (keyword fs-rf-docs).

This might be considered hijacking the thread, but I can see three ways to go:

* Leave them as they are, and just tell whoever commits #5361 to
reference them in the commit message.

* Move all of the to the beta milestone, since they are indeed being
addressed, and also reference them in the comment message

* Close the fixes as duplicates now, and close the doc tickets as
wontfix once it lands (which will have to be done regardless).

Personally, I'd like to avoid the third option, since I think it's
valid to keep them going, as there's not a proper branch to checkout,
like newforms-admin has. The difference between the first two is
purely bookkeeping though, and I have no preference either way.

-Gul

[1] http://tinyurl.com/6rxpyp

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Djangochatter [WAS: Django community aggregator and non-English posts]

[Marty Alchin]

On Fri, Jun 13, 2008 at 1:38 PM, Tom Tobin <[EMAIL PROTECTED]> wrote:
> Hey, it helps me segment my incoming mail.  ;)

It's always about *you* isn't it? ;)

> Google Groups makes new mailing lists trivial.  Although this may all
> be moot if that djangosearch site pans out as useful.

Well, I'd still like to be able to get feeds for keywords, too. For
instance, with all the file stuff I've been doing, I'm curious to know
what kinds of FileField problems people are dealing with. If I could
just set up a feed for FileField, knowing it would give me a daily
digest of *everything* the community is saying about it, I'd be in
heaven. Other people would be interested to hear chatter about
"newforms-admin" (and "nfa"), "1.0", "streaming uploads", etc.

I've got an email out to Alex Aster, the guy behind djangosearch. If
he's interested in adding some features, this may well be moot. If
not, I think it might still be worth pursuing.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Djangochatter [WAS: Django community aggregator and non-English posts]

[Marty Alchin]

On Fri, Jun 13, 2008 at 1:23 PM, Tom Tobin <[EMAIL PROTECTED]> wrote:
> Ugh, time.  I have a bad habit of assuming I have more of that
> resource available than I ultimately do.  :)  I'd definitely like to
> help, but I don't think I could assume a driving role at this point.
> (Chicago Djangonauts, languishing taghelpers patch, proper multiple
> time zone support ... meh!)  Maybe a mailing list for now?

So ... a new mailing list to discuss how to address the
ever-increasing number of communication outlets? Sounds like a great
idea. :( I don't know what it would take to manage something like that
though, so whatever works.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Django community aggregator and non-English posts

[Marty Alchin]

On Fri, Jun 13, 2008 at 1:23 PM, Alex Koshelev <[EMAIL PROTECTED]> wrote:
> There is existing multi-lingual aggregator http://djangosearch.com/

I didn't realize djangosearch broke articles down by language,
complete with individual feeds! Looks like that's one problem down.
Combine that with Jacob's quick-and-dirty custom search, and maybe
we've already got all the bases covered.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Django community aggregator and non-English posts

[Marty Alchin]

This whole thing reminds me of something I've had rattling around in
my head for a while, and maybe now's the time to bring it to the
group. In addition to the language of blog posts, I've often had a
hard time tracking down information from past conversations that have
happened in the community.

There's an aggregator, IRC log, various mailing list archives, wiki
articles, ticket comments, localized community sites, the list goes on
and on. In the spirit of community-oriented sites (like djangosites,
djangosnippets, etc), I'd like to propose djangochatter, a site
dedicated to aggregated, managing and distributing communication
taking place in the Django community.

Essentially, it could pull together information from *all* those
various sources, index it for searching, provide feeds for various
tags and keywords (users would have to tag entries manually). In light
of this discussion, I expect it could also reasonably auto-detect the
text's language, so users could sign up for a feed for whatever
language(s) they can understand.

Ideally it would also have a complete (or as near as possible) archive
of *past* chatter as well. Pull together all the existing feed
history, mailing list discussions, IRC logs, wiki history, existing
ticket comments, and make them all available in a single place all at
once.

Basically, while I feel the existing aggregator is quite useful, and
has served the community well, I think there's more that can be done,
and there's no need to require the "official" site to handle it. In
fact, I would think that once djangochatter.(com|net|org) is up and
running, the existing aggregator could just redirect over to the
appropriate page and feed URLs and be done with it.

I don't have time to head this up, but I think it would be extremely
useful for everybody. I know I've spoken with a few people about this
in the past, but is anybody interested in heading this up? I'd love to
see it happen, and I'm more than willing to take part in discussions,
but I just can't dedicate myself too much to it at the moment.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: One more issue with file storage

[Marty Alchin]

On Fri, Jun 13, 2008 at 8:48 AM, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> I've been implementing a flexible file storage system for Django for
> my own use (hacked in on top of the existing file field) and would be
> happy to make this code available if you have any interest in it. I
> honestly didn't know someone was working in this area or I would have
> gotten involved sooner.

Yeah, I haven't exactly tried to hide it, but I haven't been terribly
loud about it over the past several months either.

> The code allows you specify file storage "strategies" (which are just
> classes that conform to an "interface") and specify them in the field
> constructor. A feature of my implementation that I think works out
> well is that a special descriptor is added to the model objects to
> provide access to the file.

Hrm, that sounds incredibly familiar. If you haven't looked at it
already, check out the patch at #5361. It does exactly what you're
describing, except that it also allows for a default storage class to
be used, without having to specify it on the FileField directly
(though you can if you like). That way, you can switch from one
storage mechanism in development to another in production, by just
changing the settings file.

http://code.djangoproject.com/ticket/5361

> For example, with the following code:
>
> http://dpaste.com/hold/56439/

For what it's worth, I'm not thrilled with having to specify the name
of the descriptor. In #5361, the name of the attribute is the name of
the descriptor that'll be used, just like how ForeignKey works. Beyond
that, your example looks quite a bit like what I'm doing as well.

> Instances of this model will be augmented with an additional field
> "file", as specified in the field constructor, that returns a
> "FileWithMetaData" wrapper containing the filename, file data and a
> dict of addition meta data while The file_id field on the model stores
> a "key" or UUID used to retrieve the file from whatever strategy is
> being used. This descriptor also allows setting (and deleting).

Wouldn't this introduce backwards-incompatibility with any data that
already exists in FileField instances? I suppose that's not a concern,
since you're introducing a new field type, though.

> # returns a FileWithMetaData object
> file = my_object.file
>
> # get the original filename
> orig_name  = file.filename
>
> # replaces the original file and updates "file_id"
> my_object.file = new FileWithMetaData("document.pdf", fp.read(),
> metadata={"owner": "John Q. Public"})
>
> # deletes the file from storage
> my_object.file = None

I kinda like the metadata idea, but to me, that seems better suited
for separate model fields, so it's more easier searchable and
everything. Plus, that helps keep the code changes to a minimum, since
then the file-handling code doesn't have to do anything with it at
all.

> As I said, the current field implementation itself is a little hackish
> because it was designed to integrate with the existing oldforms admin
> without modification and overrides "_save_FIELD_file", etc.

Yeah, it's the "without modification" bit that'll get you. I didn't
have to modify the admin at all (oldforms or newforms) to get things
working, but it required significant changes to Django itself. I
suppose I could've done most of it without any changes, but the idea
from the start was to enable this stuff for everybody, so some changes
to the codebase are acceptable. I've fixed several bugs in the process
too, and also deprecated save_FOO_file(), get_FOO_filename(), and the
whole gang.

Check out the documentation in the patch for #5361 when you have a
chance, and let me know what you think.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: RFC: Django 1.0 roadmap and timeline

[Marty Alchin]

On Thu, Jun 12, 2008 at 7:06 AM, Forest Bond <[EMAIL PROTECTED]> wrote:
> I think that this is a must-have:
>
>  #285 -- WSGI SCRIPT_NAME and PATH_INFO stuff

Then you'll be glad to know that it's #3 the list of "Must-have
features" in Jacob's email, just a bit below the portion you quoted.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: RFC: Django 1.0 roadmap and timeline

[Marty Alchin]

I like it, but mainly that's because I'm not the "maybe" list, and I'm
sure I can get it done in time. I do have one suggestion, though.

On Wed, Jun 11, 2008 at 10:03 PM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
> * "Maybe" features: things that *should* be in 1.0 and should be worked on
>  in the run up to the release. If, however, features on this list aren't
>  completed, they will be dropped.

I think I know what you mean here, but "dropped" sounds an awful lot
like an all or nothing deal: either it makes it into 1.0 or it never
makes it at all. It's probably best to clarify that "dropped" just
means "dropped from 1.0, to be added in the next release after it's
completed." Yeah, it might be obvious to some of us, but a lot of
people could read it the wrong way.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

One more issue with file storage

[Marty Alchin]

Hey all,

I've been fairly quiet on the file storage front for a while, since
it's basically done now, and is just waiting on the streaming upload
ticket to hit trunk first. Since I got to that point, however, I've
had two different people, working on two different storage backends,
both approach me with the same question. It doesn't seem like a
showstopper to me, but I'd definitely like to get it addressed. I've
been mulling it over for a while now and can't come up with a
universally "good" solution, so I'd like to bring it to the group for
discussion.

For a little background, #5361 splits Django's file handling code is
split into two separate groups:

* Models deal with just the file's name and its content. For the most
part, the content just passes through untouched, but certain field
types may inspect the file's contents to get information about it,
such as ImageField extracting width and height information.
* Storage systems take the name that the model knows about and maps it
to an actual storage mechanism. This way, each site is free to decide
how and where its files are stored and retrieved, independent of any
specific model.

Of course, there's always been some overlap here, but it's been kept
to a minimum: models need to know what storage systems to use when
committing the file, but they shouldn't know (or care) about *how* the
storage system deals with the file. It should just be able to pass a
name and a string of content, and blissfully allow the storage system
to do its thing. Normally, a default storage system (specified with a
new setting) is used for all FileFields, though individual fields can
also accept a "storage" argument in order to use a different one. This
helps keep models generic enough that different storage systems can be
used in development vs. production, just by altering a setting, same
as it is for many other features.

Unfortunately, we don't live in that ideal utopia where a single
setting can control everything. Different model fields may want to
specify different locations for their files, different ways of naming
incoming files, or possibly different backends entirely. It's possible
to work around some of these issues by importing
django.core.filestorage.storage, which is automatically an instance of
the default storage system, and extend its class to create a new
storage system, without relying on which one is used under the covers.
However, that only works when the options being changed don't deal
with storing the file directly (for example, file naming works just
fine this way, and is probably the most common use case for it).

Sometimes, a backend may take other options when storing a file, which
map nicely to Django's field-level description of files. For example,
MogileFS taks a "class" option that controls how the file is
distributed among the various machines in the system, while S3 can
accept HTTP headers that should be sent with the file, such as
mime-types or cache configurations. These are clearly specific to each
backend, and not anything Django cares about, so the natural first
response is to simply handle them in backend methods, supplying the
options as arguments to __init__().

The big problem now is that this breaks the separation between
FileField and file storage. Now you're specifying backend-specific
options to individual fields, which doesn't work when you want to use
FileSystemStorage in development, but S3Storage in production. One
proposal to allow this was to accept additional arguments to
FileField, which just get passed to the backend, but that seems even
worse to me, since they'd likely get passed to backends that don't
support them. Fixing that might invovle having each backend explicitly
report which arguments it accepts, and that's not a path I'd like to
take.

So far, I've only found one way to reasonably solve this. I'm not 100%
happy with it, and I'm sure others will have their issues with it as
well, but it works with the current patch, and seems (to me) to solve
the problems at hand. It's just not as clean as I'd like. Say you're
looking to switch between filesystem in development and MogileFS in
production, while including a class of "avatar" for all files using a
specific FileField. It might look like http://dpaste.com/hold/55681/,
which works, but has a few bits of wackiness.

* It relies on the fact that if a backend doesn't know about an
option, it simply won't look for that option. This is a departure from
how options normally works, but it does allow for two different
"default" backends to share a subclass in this way. FileSystemStorage
doesn't care if it has a file_class attribute, it never looks for it
anyway.
* It assumes that the backend is wired to work this way. If someone
else comes along with a backend that doesn't look for options as
attributes (since it's not really the normal way of doing things),
there may be features that can't be used without sacrificing
flexibility. This could be just a documentation 

Re: Improving the truncatewords filter (and a +1 vote for filters with multiple arguments)

[Marty Alchin]

On Wed, Jun 4, 2008 at 1:21 PM, Aral Balkan <[EMAIL PROTECTED]> wrote:
> Unfortunately, I don't think your ticketing system likes me. I'm
> getting: 500 Internal Server Error (Submission rejected as potential
> spam).

Right on the new ticket screen, under the big heading labeled "Read
this first" you'll find the following line:

"If you're getting rejected by the spam filter, we apologize! The best
way to avoid that is to register for an account and make sure you're
logged in."

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Rethinking silent failures in templates

[Marty Alchin]

On Wed, May 14, 2008 at 10:07 AM, Simon Willison
<[EMAIL PROTECTED]> wrote:
>> I suspect that a lot of people actually rely on this behavior, and it
>> would be devastating to them.
>
> Thinking about it some more you're right - I'm sure there are lots of
> cases where people are relying on things like the following:
>
> {{ article.something.title }} - outputs text if article is there,
> fails silently otherwise

I'm not expert on the history of this or anything, but I always
thought this example was the basis for silent errors. That makes me
wonder, then, if there's some other kind of separation we can have.
Perhaps variable lookups continue to fail silently, regardless of
environment, while actual {% ... %} tags raise errors by default.

Of course, there's still something to be said for using DEBUG to show
errors in development, while not in production, and individual tags
would obviously be able to suppress their own errors if they choose
to, but I think it might work.

And at the risk of supplying too many possibilities for one email, the
magic_quotes problem could be avoided similarly to autoescaping.
Essentially, provide an extra argument to all the template rendering
functions, with a default based on the DEBUG (or other) setting.
Something like def render(self, context, silent_errors=not
settings.DEBUG). Then, individual apps can specify their behavior
explicitly if they need to, while relying on the setting if they
don't.

I'm probably way off-base on these ideas, though, because it would
likely lead to a lot of "hey, why is this template erroring out, while
this other one isn't?" I dunno. This email was probably too
stream-of-consciousness to be of any use. But there it is.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: API question for model saving

[Marty Alchin]

On Thu, May 1, 2008 at 10:49 AM, Karen Tracey <[EMAIL PROTECTED]> wrote:
> On Thu, May 1, 2008 at 10:31 AM, David Cramer <[EMAIL PROTECTED]> wrote:
>
> > I'm still not quite sure why we need any additional methods, or flags, or
> anything. Can someone explain to me where the underlying API is having
> issues?
>
> Malcolm's initial note creating this thread described an example scenario
> where additional control over save() behavior is desired:

Maybe I'm smoking something, but Patryk is making the most to me,
anyway. Wouldn't it be possible to just override save() if you really
need this for a particular model? Then you can split it into
create()/update() if you want, or add any-colored argument you like,
or whatever. And if you need it on more than one, you can just put
that on an abstract base class and inherit from that instead of
models.Model.

Or am I missing something glaringly obvious? I can understand the
desire to allow this option in trunk directly, but is there anything
that's actively preventing it from being possible now? And since
Django generally caters to the common case, which this clearly isn't,
does trunk really need to have it at all?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: QSRF Related

[Marty Alchin]

On Tue, Apr 29, 2008 at 2:02 PM, James Bennett <[EMAIL PROTECTED]> wrote:
>  >  3) This is a pretty major change to Django, and there's nothing on the
>  >  page that says "HEY WE JUST CHANGED THE ENTIRE DBAPI ALL YOUR HACKS
>  >  WILL BREAK" (I didn't even know QSRF was released until someone
>  >  pointed it out to me)
>
>  Er, no, the public, documented API didn't really change much at all.
>  And people who were doing work deep under the hood presumably were
>  smart enough to be paying attention and so don't need a big screaming
>  warning that queryset-refactor was taking place.

Also, the few changes that are noteworthy are linked to from the usual
place,[1] where everyone who's updating SVN is regularly informed to
look anyway. Any new features use the usual "New in Django development
version" as well.

-Gul

[1] http://code.djangoproject.com/wiki/BackwardsIncompatibleChanges

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Models created from querysets aren't constructed with constructor

[Marty Alchin]

On Mon, Apr 28, 2008 at 4:03 PM, Ivan Sagalaev
<[EMAIL PROTECTED]> wrote:
>  I've just hit a wall with a behavior that I find a bit strange. When a
>  model is instantiated from a queryset result it's not created with it's
>  own constructor but instead uses Model.from_sequence that does creates
>  an Empty() instance and then changes its __class__ to a Model.

I'm glad I'm not the only one to have noticed this, but I thought it
was something on my end until your email prompted me to look at it
again. I'm pretty sure I remember the discussion that led to
from_sequence(), so I understand why it's there and everything, but I
do agree that we need to have some way to trigger __init__() in that
case.

For those of you keeping score at home, the bug639 test relies on the
__init__() functionality, and the fact that it's not failing in trunk
is really just a fluke, owing to the fact that the tearDown() method
of the test case doesn't actually save the object again after
retrieving it. Since it currently throws a DeprecationWarning in my
filestorage code, I updated it to use the new API, which saves by
default when you call p.image.delete(). This, then, fails miserably
because _savecount was never set. That caused me no end of confusion
trying to figure out what I did wrong, since it works on trunk. I can
use p.image.delete(save=False) and get around it for the bug639 case,
but I do think it merits some discussion.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: API question for model saving

[Marty Alchin]

On Sun, Apr 27, 2008 at 10:17 PM, Grégoire Cachet
<[EMAIL PROTECTED]> wrote:
>  Why could you not include a parameter in the Meta class, like
>  no_update = True ?
>  By default, this parameter would be False.

The problem is, in Malcolm's example of an employee database, you
might well sometimes need to make changes to a specific employee's
records. For example, you might need to update their salary, the
department they work for, maybe even their name or date of birth if
they were entered incorrectly (or for legitimate reasons, such as a
woman getting married). Simply supplying no_update doesn't give the
application the chance to decide when an update is allowed, and when
it's not.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: API question for model saving

[Marty Alchin]

I'm a bit on the other side of the situation here. To me, save() seems
adequately ambiguous, either inserting or updating depending on the
situation. As an alternative, I'd think having separate methods, like
create() and update() would make things clearer still.

In addition to avoiding the whole argument-naming issue, it seems like
this would make subclassing even better. If you create a subclass and
override create(), that would be used in the explicit create() case as
well as the create side of the save() case.

I don't know, my mind's not really on this topic enough yet, so I'm
sure this makes less sense than it seems to me. I'll think it over
some more and see.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: How expensive are tests?

[Marty Alchin]

On Sun, Apr 27, 2008 at 1:37 AM, Russell Keith-Magee
<[EMAIL PROTECTED]> wrote:
>  - The database reset is only used by django.test.TestCase. If you are
>  using raw unittest.TestCase or doctests, the database isn't flushed,
>  and so they can run quite quickly.

This addresses the root of my question. I planned on doing everything
in doctests, so I don't feel bad having two separate tests modules.

>  At the top level, you have regression tests for file uploads, so it
>  makes sense that they are all in a single package. However, that
>  doesn't preclude some internal organization. Having multiple test
>  classes in a single tests.py file is one way to do this (and there are
>  already a few examples of this in the regression tests for the
>  fixtures and test cases).

Well, my concern now is the notion of separate tests for separate
things. We have modeltests and regressiontests, and it seems (to me)
like the file storage does a little of both.

On one hand, there's the file storage code itself, which solely deals
with storing and retrieving files. This has absolutely nothing to do
with the database or any models, and has nothing to do with the upload
process itself, or any of the form handling for it. It just solely
deals with mapping filenames to actual stored content. I was thinking
tests for this would be best placed in regressiontests.

On the other hand, the file storage patch also includes a number of
changes and improvements to FileField and how its various operations
work. As model fields, I figured tests for this portion of the patch
would be best placed in modeltests.

So, I guess it's less of a technical question and more a philosophical
one. Which tests should go where?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

How expensive are tests?

[Marty Alchin]

I had mentioned this a while back, in passing, but I'd like to bring
it up again now that the filestorage patch is getting close to making
it into trunk.

In particular, I thought I had remembered some discussion a while back
about how expensive different test packages were, since each package
requires a setup/teardown of the database. Normally this is an
acceptable inconvenience, but it becomes a burdern for tests that
don't actually use the database.

This ties in with file storage because there are actually two
distinctly separate things to test: model-related file stuffs and
those features that don't care about models. I already have a number
of tests for the model-related work: FileField upload_to, etc. That's
all well and good, and it essentially tests the underlying file
storage code by virtue of relying on it.

However, I'm noticing that there are a number of features that exist
solely in the storage aspect, without any relation to FileField. I'd
rather test these in a separate test package, so it's obvious where
the line is drawn. For instance, it doesn't make sense for modeltests
to include a test of whether a given file path is actually beneath the
storage system's base location. I'd rather have a separate package for
those types of things.

So, my question is this: how expensive is it, really, to setup and
teardown the database and whatnot, with two separate packages? Is it
enough to merit rolling all these tests into one package, or would it
be worth it to maintain the separation of concerns?

If it's best to have two separate packages, I'll make that change and
upload a new patch to be looked over while we wait on the streaming
upload stuff to finish up.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Model Inheritance in qsrf and User?

[Marty Alchin]

On Fri, Apr 25, 2008 at 3:26 PM, Ian Kelly <[EMAIL PROTECTED]> wrote:
>  Purely in terms of OO design, because it's cleaner.  Object
>  composition is usually a more appropriate paradigm than class
>  inheritance.  To take the example from the post that started this
>  thread, the relationship between users and user profiles is "has-a",
>  not "is-a".  So what would be the design benefit of applying
>  inheritance here in the first place?

Not to get too much into this discussion, but I tend to be slow to
understand design patterns and other paradigms. What then would be the
appropriate way to handle an app that managed authors? I can see two
possibilities:

class Author(User):
pen_name = models.CharField(max_length=255)

or

class AuthorProfile(models.Model):
user = models.OneToOneField(User)
pen_name = models.CharField(max_length=255)

Essentially, whether it has the semantics of an "is-a" or a "has-a"
relationship seems to depend solely on what name I give the class.
Clearly a user could "be an" author, while he/she could instead "have
an" author profile.

How would I proceed in a situation like this?

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: SVN Milestones

[Marty Alchin]

On Thu, Apr 24, 2008 at 8:19 PM, James Bennett <[EMAIL PROTECTED]> wrote:
>  * Marty's file backend work needs to land, because that drastically
>   improves both the ease of file handling in general and the ability
>   to use popular storage solutions like Amazon S3.

I know you said your list isn't necessarily exhaustive, but I thought
I should pass along a conversation Jacob and I had not too long ago.
He'd like to get Mike Axiak's work on #2070 merged prior to my
filestorage patch, since it's a bit easier to modify mine to work with
his than the other way around. The fix for #2070 looks like it may
have slight backwards-incompatibilities on its own, so it should
probably be considered part of the 1.0 list anyway, but it's
definitely a prerequisite for mine to go in, and since mine's on the
list...

Of course, you guys with the commit bit can hash it out amongst
yourselves as to how you want to do it, I'm fine with it either way. I
just wanted to make sure I made that conversation known, since I don't
think it got passed along to the rest of the crew.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Middleware class vs decorators (proposal?)

[Marty Alchin]

On Sun, Apr 13, 2008 at 7:05 AM, Amit Upadhyay <[EMAIL PROTECTED]> wrote:
> I was wondering about the reason that middleware classes were used instead
> of decorators to implement middleware functionality. One of the use cases
> that lead me into thinking about it is that I was looking for a way to have
> middleware apply only to views of one particular app [facebook app for
> example]. The MiddlewareNotUsed exception that is used by DebugMiddleware is
> very limited, vs if it was a decorator, the decorator would have the view
> passed and it could have taken a decision based on the module path of the
> view.
>
> The other use case I thought of was: on my site, there are some background
> processes running intermittently, and the  load my mysql server, and abt
> 20-30 times a day on my site we get an error abt mysql server not
> responding. I imagined writing a middleware that will catch this exception,
> and sleep for a few secs and then call the view again, but I realized the
> current design of middleware does not allow that, and the only way to do
> this would be to send a http302 or something, which would be useless for
> POST requests and would be dangerous if website is loaded due to some reason
> [it will lead to recursive 302, adding up lot of load on the website]. With
> middleware I would have passed the actual view, and I would have caught the
> exception and in excpetion handler trying to call the view one more time
> before giving up.

Call me crazy, but I'm not seeing anything here that can't be done in
the process_view() stage of middleware. It receives the incoming
request, the view that's about to be executed (so you can get its
module path), as well as the arguments that will be applied to it.

The code you specified could actually be run inside there, executing
the view directly if you like, then returning the response directly
from process_view(), bypassing the rest of the middleware phase. It
doesn't sound like a pleasant situation to me, but neither does what
you're trying to accomplish.

I really think you're looking for a middleware that implements
process_view(). Look into it,[1] if you haven't already.

-Gul

[1] http://www.djangoproject.com/documentation/middleware/#process-view

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Template.render() should accept a dictionary, not just a Context object

[Marty Alchin]

On Fri, Apr 11, 2008 at 12:02 PM, Patryk Zawadzki <[EMAIL PROTECTED]> wrote:
>  Why not use the opposite test and check for instances of dict?

Because then you have to pass something that subclasses dict instead
of something that implements the interface of dict. We could go on and
on about this all day.

The usual way to approach something like this would be to check for
some attribute of contexts that aren't part of the standard dictionary
protocol. Since contexts work mostly like dicts, the only real
differences are the push() and pop() methods, which dicts don't
support. Welcome to duck typing.

def render(self, context):
"Display stage -- can be called many times"
if not hasattr(context, 'push'):
context = Context(context)
return self.nodelist.render(context)

Of course, I should note that I actually don't have an opinion on this
subject. I'm merely offering an option to suit both sides of the
equation.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: An Awesome Proposal [You won't like]

[Marty Alchin]

On Thu, Apr 10, 2008 at 1:30 PM, Marty Alchin <[EMAIL PROTECTED]> wrote:
>  Unfortunately, there are still a number of situations where existing
>  code (and most likely will) break. Some of these situations are easily
>  solvable, others *could* be solved easily by doing a bit of weird
>  stuff that would probably flirt too close to "magic", and others
>  aren't solvable at all.

I should note that #5361 only solves the easy ones. Thankfully, files
aren't a situation where we're likely to encounter as many of the
other situations, so I think it's a negligible incompatibility. Doing
it for anything that can have choices... well, that'd require a good
bit more discussion, I think.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: An Awesome Proposal [You won't like]

[Marty Alchin]

On Thu, Apr 10, 2008 at 1:08 PM, Jeremy Dunck <[EMAIL PROTECTED]> wrote:
>  I think with field subclassing, descriptors are a good way to add this
>  sort of thing, but it needn't be backwards imcompatible, because any
>  get_XXX_display could just translate to model.XXX.display, etc.

Unfortunately, there are still a number of situations where existing
code (and most likely will) break. Some of these situations are easily
solvable, others *could* be solved easily by doing a bit of weird
stuff that would probably flirt too close to "magic", and others
aren't solvable at all. I can itemize them if you'd like, but for the
sake of brevity, I'll just say that since I did one of these
translations already for #5361, I've done a lot of thinking on the
subject.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: An Awesome Proposal [You won't like]

[Marty Alchin]

On Thu, Apr 10, 2008 at 1:05 PM, David Cramer <[EMAIL PROTECTED]> wrote:
>  mymodel.filedfield.save_file()

For what it's worth, ticket #5361 does exactly this.

http://code.djangoproject.com/ticket/5361

>  - Clutters the model namespace

Agreed

>  - [Assumption] Performance gain by removing them as it creates extra
>  lambda functions to attach them.

[Counterassumption] Because of some details associated with how #5361
has to work, it has its own potential performance limitations that may
negate any gains offered by getting rid of lambda functions. I find it
best not to worry about this kinds of micro-optimizations unless
there's evidence they're causing problems.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Apps engine

[Marty Alchin]

On Tue, Apr 8, 2008 at 11:30 AM, konryd <[EMAIL PROTECTED]> wrote:
>  A question I'd like to ask experienced developers:

That's not what django-developers is for. It's for the development of
Django itself. There are plenty of experienced developers reading
django-users too.

>  Is it possible to write a django db backend that uses google's db api?
>  (this one: http://code.google.com/appengine/docs/datastore/)

There's already some discussion going on about this on the django-users list.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Streaming Uploads Discussion

[Marty Alchin]

On Fri, Apr 4, 2008 at 3:43 PM, Jacob Kaplan-Moss
<[EMAIL PROTECTED]> wrote:
>  Yeah, one thing we'll need to figure out PDQ is what's appropriate for
>  an upload handler, and what's appropriate for a storage backend.
>  Hopefully the two of you can work out the breakdown.

I'll read over the patch and the docs and see if I can get a better
handle on how it works, so I can be of more use there. Also, Mike and
I put our heads together in IRC sometimes, so we should be able to get
it sorted out soon.

>  FYI, I plan to merge 2070 first (sorry, Marty!) since I think it works
>  a bit better that way from a dependancy POV.

No worries. I still have another of David's suggestions to integrate
before I can have it "finished" anyway. Plus, there's a 3291-ticket
age difference. The youngest always gets the shaft. :) I can live with
that.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Streaming Uploads Discussion

[Marty Alchin]

On Fri, Apr 4, 2008 at 2:04 PM, Mike Axiak <[EMAIL PROTECTED]> wrote:
>  However, the instant they pass the file to some remote location
>  (think: S3UploadHandler) or alter the content (think:
>  GZipUploadHandler) they will need their own way of defining what is
>  content and how it should be "saved".

Maybe I'm missing something obvious, but why would there ever be an
S3UploadHandler? Shouldn't that be handled by a file storage backend?

As for GZipUploadHandler, if you're talking about gzipping it as part
of the save process, shouldn't that also be a file storage backend? Of
course, if you're talking about *un*gzipping it during upload, so it
can be processed by Python, I withdraw the question.

I admit I haven't been following this terribly closely, but now that
both #5361 and #2070 are nearing completion, I'm trying to get a good
handle on all of this in case there are any interactions between the
two that I can help with.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: New documentation outline (was: Refactoring the documentation)

[Marty Alchin]

On Wed, Apr 2, 2008 at 7:05 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>  One thing that struck me while looking over this list is that it might
>  be a good idea to add a section on best practices for writing reusable
>  apps.  James Bennett's presentation at PyCon hit on some really good
>  main points that the section could cover.  Maybe it could go under the
>  "django.contrib add-ons" section.  Thoughts?

While I agree with the importance of having reusability tips, I would
certainly argue against putting it anywhere near django.contrib. When
I started working on my first app, I had unnecessary delusions of
getting into django.contrib when it was ready. I wouldn't want to
cause more people to expect that any and every third-party app has a
good shot at making it into django.contrib.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Refactoring the documentation

[Marty Alchin]

On Wed, Apr 2, 2008 at 12:22 PM, James Bennett <[EMAIL PROTECTED]> wrote:
>  I most sincerely hope not; unless you're like me and write short-form
>  novels in your docstrings, they tend to be an absolutely horrible
>  source of end-user documentation.

def get_object_from_cache(self, ...):
"""
Get an object from the cache and return it.

...

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Refactoring the documentation

[Marty Alchin]

I won't bother responding to anybody in particular, because I'm in
agreement with so many on this thread so far. I'm glad to see this
refactoring happen, because a doc improvement can only make things
better for everybody.

I know you're not getting into details yet, but once you do get down
to it, I'll gladly volunteer to write, or at least edit, API reference
and internals docs. I already delve into all that anyway, so
documenting it sounds like a blast!

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Customizable QuerySets

[Marty Alchin]

On Mon, Mar 31, 2008 at 5:27 AM, Malcolm Tredinnick
<[EMAIL PROTECTED]> wrote:
>  If you want to be able to specify a different default manager for some
>  particular use of an existing model, you're sort of after a third type
>  of model subclassing that I've thought about but haven't implemented:
>  subclassing an existing model and explicitly telling Django that this is
>  only Python-level (and not database-/ORM-level) inheritance. So all the
>  database interactions are part of the parent class(es) and the child
>  class simply adds extra functional pieces (such as a new default
>  manager). Nothing existing so far rules out adding this, so it's not
>  something that has to be resolved now, which is why I haven't wasted any
>  brain cells on it so far. This might be fairly easy to add once we work
>  out how to spell it, since it's only saying "create absolutely no fields
>  for this model, not even links back to the parent model and definitely
>  don't create a database table."

I've invested a few brain cells on this already, though not enough for
a complete solution. It'd be off-topic for this discussion, but
if/when you get around to it, feel free to hit me up if you're looking
for another opinion on how it could be done.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: GSoC2008 Proposal

[Marty Alchin]

On Fri, Mar 28, 2008 at 9:05 AM, Waylan Limberg <[EMAIL PROTECTED]> wrote:
>  Have you seen the dbsettings [1] app? If I'm understanding you
>  correctly, it does everything you want.
>
>  [1]: http://code.google.com/p/django-values/

Well, not exactly everything, since dbsettings doesn't export to
templates automagically (nor would I want it to). I do, however,
wonder if there's merit to the notion of a SoC project to formalize
and standardize something like dbsettings.

I wrote it to do something I needed, then modified it to suit the
desires of others, without any real design goals in mind. Since then,
it's been forked at least a half-dozen times, some of which have
landed in high-profile projects (like Satchmo). My original
(optimistic) goal was to have a system that could be generic and
extensible enough that it could be adopted by any app or project,
without the need to fork it.

I don't know if that's even possible, much less whether it'd be a good
goal to work toward, or whether it would be suitable for a SoC
project. I just wanted to throw it out there, given the original
proposal and the mention of dbsettings.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reorganising management/validation.py

[Marty Alchin]

On 2/24/08, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:
>  A few things I'm working on now end up requiring additions to this list
>  and every time I open up the file I want to reorganise it to makes
>  things easier to maintain. The main point is that almost all of these
>  validation checks are actually Field-specific or Options-specific, so
>  could just as easily be contained on the appropriate Field, etc. Adding
>  this path (django-admin.py asks each field to verify itself) also adds
>  the ability for custom fields to inject themselves into this
>  sanity-check path. I don't think this is particularly controversial,
>  since it has no backwards-incompatibility impact and I'm happy to do the
>  reorg -- it's fairly straightforward.

I raised this issue[1] some time ago, and there was some brief
discussion with Adrian and Russell, among others. Of course, my
specific interest in the subject is no longer an issue, as post-#5361
upload_to will still use the same validation as exists today, but I
think it's still worth discussing. Having it raised by a core dev
should help move things along, I hope.

>  The question is, which of two approaches to take:

As evidenced in the other thread, my gut instinct was to go for option
(1), physically preventing invalid fields from being used. However,
Adrian raised a decent point that validating your code should really
only be necessary during development, rather than in production. I
still don't really understand just what he was getting at with memory
savings, or how important those savings are, or if they can be
preserved with any alternative approach.

Russell and I seemed to end up agreeing on something similar to option
(2), though Russell seemed to have some alternative thoughts on how it
could be handled. Personally, I'm less interested in asserting one of
the two methods as just making sure that it's more reasonable for
fields to define their own code validation rules, without having to
reach into core.

-Gul

[1] 
http://groups.google.com/group/django-developers/browse_thread/thread/9e1a01c1448c9e86/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Button on website to show version of trunk

[Marty Alchin]

While I understand the intent, and I agree that people need to realize
that trunk is quite stable, I don't think this is a good way to go
about it. It seems to me that it will only raise questions about the
Django release schedule: "Wait, if there have been 2297 commits since
0.96, why hasn't there been a 0.97 by now? Clearly it's because the
trunk version isn't stable enough for a release."

While that's not a fact, it's certainly a conclusion that could be
drawn by those numbers. Also, keep in mind that those revision numbers
include ALL commits that took place to the SVN repository that hosts
Django, including all the main branches and djangoproject.com. As an
example of why this is an important distinction, some basic research
on my end shows that of the over 7000 commits that have taken place,
well under 5000 of them were to Django trunk. I don't know how many of
them were post-0.96, but I expect it's well less than the 2297 the raw
numbers would suggest.

All in all, I agree that trunk should be promoted more heavily, but I
don't think revision numbers are the way to do it.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Future-proofing django.template (and elsewhere) against circular imports

[Marty Alchin]

On Feb 1, 2008 1:47 PM, Tom Tobin <[EMAIL PROTECTED]> wrote:
> That said, I'd like to propose that the majority of the contents of
> django.template.__init__ be preemptively moved to
> django.template.main, and imported back into __init__ as necessary.
> Furthermore, I'd like to see all use of __init__ modules for holding
> more than "moment-of-import" code to cease, for reasons of avoiding
> said nasty circular imports.

And in some cases, it can be valuable from a pure maintainability
standpoint. django.db.models.fields.__init__, for instance, covers
many varied cases, and I've already moved the file-related code out
into a files.py as part of my work with #5361, just so I could manage
it more easily. But then, when there was a ModelForm ticket was
solved, in involved a change to FileField, which took entirely too
much digging to discover.

All in all, I'm +1 on this, but I don't know if it needs to be an
absolute, across-the-board mandate. For instance, django.http.__init__
could be split into django.http.request and django.http.response, but
I'm not really sure it needs to be done for that case.

-Gul

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---