Hi Russell,
so, after our chat on IRC I've finally found the time to implement a
real proposal including unit tests. I've attached the patch to this
ticket:
http://code.djangoproject.com/ticket/13960
Now there is just one backend type with a single setting:
FILE_BACKENDS = (
'path.to.Backend',
'path.to.Backend2',
)
--------
The end-user will normally use the API via ModelForm like this when uploading:
form = MyModelForm()
form.prepare_upload(request)
By default, prepare_upload() will prepare an upload to the current URL
(and thus current view). Optionally, you can specify a different url
via form.prepare_upload(request, url='/other/url').
The prepare_upload function is necessary because we need to support
services which first send the file and form data to a different URL
(e.g., to S3 or '/_ah/upload/...' on GAE). Once the upload is
finished, the service or backend is responsible for sending a request
to the actual target URL (some Django view).
So, in the template you have to use the generated upload url:
<form action="{{ form.upload_url }}" ...>
<table>{{ form }}</table>
FYI, behind the scenes prepare_upload() might also add hidden input
fields to your form, depending on the service you're using.
------
Downloads are split into two functions which both exist on the file
object returned by FileField.
With file.serve(request) which returns an HttpResponse you can handle
the file download from a Django view. This also allows app developers
to check permissions in the view before calling file.serve(request).
Additionally, there's file.download_url() which deprecates file.url
(because that uses the storage backend which is the wrong place for
this feature). This function will always return an URL. If none of
your backends provides an URL we use "MEDIA_URL + file.name" as a
fallback.
In some cases file.download_url() will point to a Django view which
calls file.serve(request) (and which can do some permission checks if
necessary). Authors of reusable Django apps are responsible for
providing a default backend which generates the URL to their app's
built-in download view. End-users can overrides this.
In your templates you'd always use code like this:
<a href="{{ entity.file.download_url }}">Download</a>
--------
Backends
Every backend derives from BaseFileBackend. Every backend instance has
these members:
* self.model: the model that own the FileFields to upload to or download from
* self.fields: the list of FileField instances to upload to or
download from (in case of a download there is just one entry)
For convenience, there is also self.field which makes sure that there
is just one field in self.fields and returns that field.
Every backend can override the functions mentioned above
(prepare_upload, file.serve, file.download_url). You can take a look
at the source for more details. FYI, the prepare_upload() function is
pretty much the same as before, just without the private=True/False
parameter.
Additionally, backends can define get_storage_backend() which returns
a storage backend for the given model/field combination. As a fallback
DEFAULT_STORAGE_BACKEND is used.
The API is also similar to DB routers. If any of those functions
returns None the next backend is tried (as defined in
settings.FILE_BACKENDS).
Please provide some feedback. Does this solve all issues you had with the API?
Bye,
Waldemar Kornewald
On Mon, Jun 28, 2010 at 2:08 PM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> Apologies for the late reply - I was at a conference all weekend, so
> I'm still catching up on mail.
>
> On Thu, Jun 24, 2010 at 12:24 AM, Waldemar Kornewald
> <wkornew...@gmail.com> wrote:
>> On Wed, Jun 23, 2010 at 2:58 AM, Russell Keith-Magee
>> <russ...@keith-magee.com> wrote:
>>> On Wed, Jun 23, 2010 at 2:58 AM, Waldemar Kornewald
>>> <wkornew...@gmail.com> wrote:
>>>> On Tue, Jun 22, 2010 at 2:40 PM, Russell Keith-Magee
>>>> <russ...@keith-magee.com> wrote:
>>>>> On Tue, Jun 22, 2010 at 2:55 PM, Waldemar Kornewald
>>>>> <wkornew...@gmail.com> wrote:
>>>>> It also strikes me that a lot of this is being configured at the
>>>>> global level -- i.e., you have to nominate that your public upload
>>>>> backend will be S3, rather than nominating that a specific file field
>>>>> will be backed by S3.
>>>>
>>>> I'm repeating myself here, but anyway: The primary purpose of this API
>>>> is to allow for writing reusable Django apps that automatically work
>>>> with your project's file handling solution(s) without hard-coding
>>>> anything. This requires that you specify the upload/download backends
>>>> separately from FileField (otherwise it's hard-coded and not reusable)
>>>> and instead provide a mechanism for detecting which backend should be
>>>> chosen for the current request. Of course, this mechanism could take
>>>> the FileField into account.
>>>
>>> You may feel that you're repeating yourself, but this point certainly
>>> wasn't clear to me from your previous two posts.
>>>
>>> Making it possible to configure the file handling strategy of a
>>> reusable app is certainly a reasonable feature request. However:
>>>
>>> 1) Again, the file storage strategy isn't something that is constant
>>> across a deployment. I may want to use S3 for one type of file, but
>>> use local storage for another.
>>>
>>> 2) Handling this flexibility at the level of the request is
>>> completely the wrong approach. File storage doesn't change per request
>>> -- it's defined per model. Every usage of the Profile model has an
>>> 'avatar' ImageField; it doesn't matter how or where you access that
>>> field, it needs to be accessed and displayed the same way. This is a
>>> per-model setting, not a per-view or per-request setting.
>>
>> That's ok, but in addition to the model and field I'd still pass the
>> request object to the backend, so you can check if the user hasn't
>> used up his quote or maybe other things.
>
> You're missing my point. You aren't guaranteed to have a request when
> you're using the field. Consider the case of a standalone data
> processing script.
>
>>> There is a larger issue here of how we should treat the problem of
>>> configuring the internals of reusable applications. It's analogous to
>>> the reasons why we dropped the 'using' argument from Meta declarations
>>> on models -- it isn't a decision that a reusable-app-maker can make;
>>> it needs to be configured as a deployment decision.
>>>
>>> The solution for 'using' was to introduce the idea of a database
>>> Router; perhaps an analogous approach is required here.
>>
>> The "delegate" backend is basically a router. With the
>> django-filetransfers API you can just write another backend which does
>> the routing.
>
> They're not quite the same. Sure, the Delegate interface defines a way
> to configure behaviour, but it doesn't provide an interface to decide
> which behavior is appropriate at any given time.
>
> Again, this comes back to the concept that different reusable apps may
> have different file storage requirements, and you need to be able to
> define the strategy for those requirements at a project level. Sure,
> the simple case of "put everything on S3" needs to be simple, but you
> also need to be able to say "put all the Profile.avatar images on S3,
> but all the Document.upload files on a AppEngine filestore".
>
> You also need to take into account that a file field may not
> necessarily be backed by a model at all. It could just be an upload.
> Form.FileField doesn't require that it is backed by a model.FileField.
>
>>>> Don't worry about the current code. We haven't yet officially
>>>> announced that project and I'll hold it back until it's clear whether
>>>> this will be part of Django core or not. We can completely reinvent
>>>> the API from scratch if necessary.
>>>
>>> In which, case, you need to make a specific proposal. I'll admit that
>>> this is a problem that needs to be addressed, and I'm interested in
>>> seeing approaches that addressing this problem, I can't say it's a
>>> particularly high priority for me. I'm happy to give you feedback on a
>>> specific proposal, but I have many much higher priority items on my
>>> plate for 1.3.
>>
>> OK, so before I send a patch, here is how I'd like to do it:
>>
>> From the end-user perspective
>> -----------------------------------------------
>>
>> FileField gets a new method prepare_upload() which takes the following
>> arguments:
>> * request
>
> Again - this isn't always available.
>
>> * upload_url: the target URL of the upload view
>> * private: should this be only privately accessibly or also publicly?
>> (default: False; whether this actually works depends on the chosen
>> backend's capabilities and your hosting setup)
>
> As Robert and Luke both point out, public/private is neither a
> necessary nor sufficient basis on which to make this decision.
>
>> The function returns a tuple with a new target submission URL and a
>> dict with additional POST data.
>>
>> forms.Form also gets a prepare_upload() function which searches for a
>> FileField. If multiple FileFields exist and the backend doesn't
>> support multi-file uploads an exception is raised.
>
> Ok - no problems with this bit. I have no problems with the idea that
> if you've configured your file backend to use a particular strategy on
> your form, and your form won't support that strategy, an exception
> will need to be raised.
>
>> The function
>> doesn't return anything, but instead stores the results as
>> self.upload_url and self.some_file_field.post_data. When the form is
>> rendered the FileField automatically generates <input type="hidden" />
>> fields for self.post_data. Q: or should prepare_upload() better be
>> defined on forms.FileField?
>
> This sounds like a better place for it.
>
>> File (from FileField) gets a public_download_url() method which takes
>> no arguments. It returns the file's permanent public URL or None if no
>> such URL exists.
>
> Sorry - are you saying this is on the FileField, or on the File
> object? If it's on the FileField, how do you get the download URL when
> there isn't a model backing the form? And if it's on the File object,
> what's wrong with the url() method that is already there?
>
>> File (from FileField) gets a serve() method which takes the following
>> arguments:
>> * request
>> * save_as: if False (default) lets the browser decide whether to
>> display or download the file; if True forces browser to download as
>> "file.name"; if it's a string it forces a download with the given
>> string as the file name
>> * content_type: if None (default) automatically detects content type
>> based on file name; otherwise it overrides the default content type
>> This returns an HttpResponse.
>>
>> Example:
>>
>> Upload view:
>> form = FileForm()
>> form.prepare_upload(request, '/upload')
>
> I'm not wile that the prepare_upload step is required. It feels like
> something that should be handled internally, based on the
> configuration of handler for the field.
>
>> Upload template:
>> <form action="{{ form.upload_url }}" ...>
>> {% csrf_token %}
>> <table>{{ form }}</table>
>> </form>
>
> I'm not sure I follow what happends with the non-form data here. From
> what I can make out, upload_url won't be able to handle any non-file
> data. I'm also not sure what happens for the default case (i.e., the
> case the defines the current file upload behaviour), where multiple
> files can be uploaded to the normal form handling API.
>
>> Download template for private-only downloads (entity is a model instance):
>> <a href="{% url download_view pk=entity.pk %}">Download</a>
>
> What determines the view name "download_view"?
>
>> Download template for public downloads (entity is a model instance
>> with a FileField called "file"):
>> {% url download_view pk=entity.pk as fallback_url %}
>> <a href="{% firstof entity.file.public_download_url fallback_url
>> %}">Download</a>
>>
>> Download view for public and private downloads (public only as a
>> fall-back if there is no public_download_url):
>> entity.file.serve(request)
>>
>> The prepare_upload(), public_download_url(), and serve() functions
>> each have their own backend type. Each of the three backend types is
>> configured separately in settings.py:
>> PREPARE_UPLOAD_BACKEND = {
>> 'backend': 'some.backend',
>> 'setting': ...,
>> 'setting2': ...,
>> }
>> PUBLIC_DOWNLOAD_URL_BACKEND = {
>> 'backend': 'some.backend',
>> ...
>> }
>> SERVE_FILE_BACKEND = {
>> 'backend': 'some.backend',
>> ...
>> }
>>
>> Backend API
>> --------------------------------------------------------
>> I won't, yet talk too much about the backend API itself because we
>> should first talk about the end-user API. Each backend type gets all
>> of the parameters that the user passes to the respective public
>> function (prepare_upload(), etc.). Additionally, each function gets
>> the model and FileField. Also, the File methods (public_download_url()
>> and serve()) get the file instance:
>> def prepare_upload(model, field, request, upload_url, private, **kwargs):
>> ...
>>
>> def public_download_url(model, field, file, **kwargs):
>> ...
>>
>> def serve_file(model, field, file, request, save_as, content_type, **kwargs):
>> ...
>>
>> There is no special Router API. Instead, you can write your own
>> backend which routes/delegates to some other backend.
>
> Again, there will be a need for *some* sort of routing API, because
> this isn't a decision that can be configured on a project-wide basis
> with a single setting (or a pair of settings on an arbitrary
> public/private distinction).
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
