Re: help needed: non-relational DB support

[Russell Keith-Magee]

On Sun, Jan 17, 2010 at 9:37 AM, Waldemar Kornewald
 wrote:
> On Sat, Jan 16, 2010 at 10:35 PM, flo...@gmail.com  wrote:
>> I'm not really a developer on Django itself, but I am fairly
>> interested in non-relational databases, and some of the things being
>> said in this thread worry me a bit.
>
> OK, if you read my mail literally I sound like "all" nonrel DBs are
> the same. Of course they're not and you can find a counter-example for
> almost everything. There's nothing to worry about, though. The
> counter-examples simply need additional changes to get supported. It's
> not like my changes would prevent other backends.

No, but they don't allow other backends either. From my perspective,
the purpose of this effort is to make the modifications to core that
make *any* backend that stores data possible.

>> I think these efforts are great--a lot of people want to get up and
>> running on GAE with Django, and it's not so easy right now.  It just
>> worries me a bit that the description of the effort encompasses all of
>> non-relational databases when the implementation seems to primarily
>> reason around GAE.  It makes sense to concretely pick one database and
>> start there, otherwise no work would ever be finished, but I think
>> that the better thing to do is to call it GAE support instead of
>> nonrel support.
>
> We have people interested in adding MongoDB, CouchDB, and maybe
> SimpleDB support. The current code should be abstract enough for
> SimpleDB and probably also MongoDB (though, it would help to modify
> AutoField to also support string values). Other DBs might need
> additional changes, but that's what the "nonrel" project is for.
> Everyone can join and work on Django changes needed for their DB.

We need to be clear about your goals here.

Speaking for me personally, It's going to be very hard to get me to be
interested in this project unless you're trying to solve the whole
problem.

We have a backend interface that works well for SQL. I'm interested in
seeing the set of modifications that are required in break the
SQL-specific aspects of that interface.

I'm not at all interested in spending a bunch of time vetting
modifications that only solve part of the problem - especially when
those modifications may well prevent the elegant introduction of a
fully refactored backend interface.

I have no problems with the idea of tackling this problem in an
iterative fashion (i.e., get it working for GAE, then get it working
for GAE+MongoDB, and so on), but I'm not going to commit anything to
trunk until you've got enough backends (with enough breadth of
features) to demonstrate that the refactoring that you propose is
sufficient to solve the general problem.

Yours,
Russ Magee %-)
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: help needed: non-relational DB support

[Waldemar Kornewald]

On Sat, Jan 16, 2010 at 10:35 PM, flo...@gmail.com  wrote:
> I'm not really a developer on Django itself, but I am fairly
> interested in non-relational databases, and some of the things being
> said in this thread worry me a bit.

OK, if you read my mail literally I sound like "all" nonrel DBs are
the same. Of course they're not and you can find a counter-example for
almost everything. There's nothing to worry about, though. The
counter-examples simply need additional changes to get supported. It's
not like my changes would prevent other backends.

>> 1. Nonrel DBs don't distinguish between INSERT and UPDATE
>> On such DBs Model.save_base() shouldn't check if an entity already
>> exists. Instead, it should just always INSERT. For this I added
>> connection.features.distinguishes_insert_from_update.
>
> That's not entirely true.  Riak and CouchDB are two that come to mind
> instantly as explicitly distinguishing between insert and update (in
> Riak an update includes the vector clock information from a previous
> read, and in CouchDB they can be mapped to different HTTP verbs).

Yes, CouchDB is versioned, so an UPDATE operation is useful. OTOH,
many other nonrel DBs (SimpleDB, MongoDB, GAE, ...) wouldn't
distinguish between INSERT and UPDATE on a save().

>> 2. Deleting related objects can be too costly on non-relational DBs
>> For this I added
>> connection.features.supports_deleting_related_objects. If it's False
>> no related objects will be deleted. Ideally, the backend should be
>> able to delegate this job to a background task. This is planned.
>
> Again, this seems like a GAE-specific observation.  In Redis, for
> example, related objects can be stored in a list data structure, and
> the delete operation can be passed any number or keys, so it could be
> a very minimal set of operations to delete related objects (first read
> the list of related object keys, then delete them in bulk).  Cassandra
> also has support coming for batch deletion, at which point efficient
> related object deletion will be similarly trivial.

GAE and many other nonrel DBs have batch deletes, but I don't think
that we should delete several thousands of entities that way in a
single request. This will take too long - unless you want to make your
users wait a few seconds for the result.

> I think these efforts are great--a lot of people want to get up and
> running on GAE with Django, and it's not so easy right now.  It just
> worries me a bit that the description of the effort encompasses all of
> non-relational databases when the implementation seems to primarily
> reason around GAE.  It makes sense to concretely pick one database and
> start there, otherwise no work would ever be finished, but I think
> that the better thing to do is to call it GAE support instead of
> nonrel support.

We have people interested in adding MongoDB, CouchDB, and maybe
SimpleDB support. The current code should be abstract enough for
SimpleDB and probably also MongoDB (though, it would help to modify
AutoField to also support string values). Other DBs might need
additional changes, but that's what the "nonrel" project is for.
Everyone can join and work on Django changes needed for their DB.

Bye,
Waldemar Kornewald

--
http://twitter.com/wkornewald
http://bitbucket.org/wkornewald/
http://allbuttonspressed.blogspot.com/
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Suggestion: DictionaryField

[Михаил Лукин]

Well, the simplest way is making DictionaryField more like CharField, except
that it's values will be stored in separate table. For example (application
name is "hardware", engine is sqlite3), if I define

*class HDD(models.Model):
  form_factor = DictionaryField(max_length=10)
  capacity = models.FloatField()
*
there will be 2 tables generated:

*CREATE TABLE "hardware_hdd_form_factor_dict" (
"id" integer NOT NULL PRIMARY KEY,
"form_factor" varchar(10) NOT NULL UNIQUE
)
CREATE TABLE "hardware_hdd" (
"id" integer NOT NULL PRIMARY KEY,
"form_factor_id" integer NOT NULL REFERENCES
"hardware_hdd_form_factor_dict" ("id"),
"capacity" real NOT NULL
)

* I guess, field constructor should not accept "unique" argument as it make
no sense.
I see 2 ways of rendering such field on a form:
1) usual  with autocompletion;
2)  +  + (maybe) radio buttons
as it should be allowed to choose from existing and to create new dictionary
item.

2010/1/17 Łukasz Rekucki 

> Could you describe this more closely? I don't think I quite follow.
> What those auto-generated tables would contain? What would be
> acceptable values for this field ?
> --
> Łukasz Rekucki
>
>
-- 
regards,
Mihail
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Suggestion: DictionaryField

[Mihail Lukin]

Hi, Community

Sometimes an application model include a lot of fields which are just
dictionary values. Example:

class HDD(models.Model):
  form_factor = models.ForeignKey(HDDFF)
  capacity = models.ForeignKey(HDDCap)
  interface = models.ForeignKey(HDDIntf)
...

and so on, where HDDFF, HDDCap and HDDIntf are similar classes.
Abstract class with only one field "name" may be used, so that this
classes could be defined like this:

class HDDFF(DictModel): pass

but it is not as good as something like this:

class HDD(models.Model):
  form_factor = models.DictionaryField()
  capacity = models.DictionaryField()
  interface = models.DictionaryField()

Database tables "hdd_form_factor_dict", "hdd_capacity_dict" and so on
could be created automatically, just like it is done by
ManyToManyField.

What do you think about that?
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: help needed: non-relational DB support

[flo...@gmail.com]

I'm not really a developer on Django itself, but I am fairly
interested in non-relational databases, and some of the things being
said in this thread worry me a bit.

> 1. Nonrel DBs don't distinguish between INSERT and UPDATE
> On such DBs Model.save_base() shouldn't check if an entity already
> exists. Instead, it should just always INSERT. For this I added
> connection.features.distinguishes_insert_from_update.

That's not entirely true.  Riak and CouchDB are two that come to mind
instantly as explicitly distinguishing between insert and update (in
Riak an update includes the vector clock information from a previous
read, and in CouchDB they can be mapped to different HTTP verbs).

> 2. Deleting related objects can be too costly on non-relational DBs
> For this I added
> connection.features.supports_deleting_related_objects. If it's False
> no related objects will be deleted. Ideally, the backend should be
> able to delegate this job to a background task. This is planned.

Again, this seems like a GAE-specific observation.  In Redis, for
example, related objects can be stored in a list data structure, and
the delete operation can be passed any number or keys, so it could be
a very minimal set of operations to delete related objects (first read
the list of related object keys, then delete them in bulk).  Cassandra
also has support coming for batch deletion, at which point efficient
related object deletion will be similarly trivial.

> 5. We need transactions which lock rows
> I added a GAE-specific @commit_locked transaction decorator. This
> should be moved into the backend layer, of course. I'd just like to
> know if this is an option for 1.3 or not. On SQL this would be SELECT
> ... FOR UPDATE, but I don't know if all SQL DBs support it. Such a
> decorator would be important to make get_or_create() and a few other
> functions work 100% correctly, so it would also benefit the SQL layer.
> I couldn't provide the respective SQL implementations, though.

This is admittedly a GAE-specific thing.

I think these efforts are great--a lot of people want to get up and
running on GAE with Django, and it's not so easy right now.  It just
worries me a bit that the description of the effort encompasses all of
non-relational databases when the implementation seems to primarily
reason around GAE.  It makes sense to concretely pick one database and
start there, otherwise no work would ever be finished, but I think
that the better thing to do is to call it GAE support instead of
nonrel support.

Thanks,
Eric Florenzano
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Logging format decision

[Ivan Sagalaev]

Russell Keith-Magee wrote:

That said - I really do want to see logging in Django's trunk. If we
can sort out the details in the background and get a good
implementation ready, it could easily be the first feature committed
for v1.3.


Agreed!


I like Vinay's suggestion for exactly the reason you dislike it. The
definition of a handler is independent of the loggers that use that
handler. Simon's example in (4) hides a lot of details behind ellipses
- if you expand those ellipses, you discover that there is a lot of
repeated typing of '/tmp/django-sql.log' etc if you want all your logs
to appear in the same file. Under Vinay's proposal, you define a
single file log handler, and instruct each of the loggers to use that
handler.


Actually, I don't see this problem of repetition that you're describing 
so I thought that we could use some examples closer to real world to 
sort this out. May be I just don't see something obvious to you or have 
misunderstood Vinay's schema. I will refer to two formats we're 
discussing as "explicit handlers" (Vinay's) and "implicit handlers" 
(what Simon has implemented). Better suggestions for naming are welcome :-).


1. The first use-case is the development environment where all you want 
is everything logged in one file with DEBUG level.


"Implicit handlers":

LOGGING = {
'': {
'handler': 'django.logging.FileLogger',
'filename': '/home/user/myproject/all.log',
'level': 'debug',
},
}

"Explicit handlers":

LOGGING = {
'formatters': {
'all': '%(asctime)s %(name)-15s %(levelname)-8s %(message)s',
},
'handlers': {
'all': {
'class': 'django.logging.FileLogger',
'filename': '/home/user/myproject/all.log',
'formatter': 'all',
'level': 'debug',
},
},
'loggers': {
'': {
'handlers': ['all'],
},
},
}

2. A production environment. This actually what we have for one of our 
services:


- a file for uncaught server errors that are logged by a function 
listening to `got_request_exception` signal which uses 'exception' label 
to log
- a file for maintenance messages that come from cron scripts that all 
use 'maintenance.*' labels to log

- a file for everything else

"Implicit handlers":

_log_path = '/var/log/service/'

LOGGING = {
'exception': {
'handler': 'django.logging.FileLogger',
'filename': _log_path + 'exception.log',
'format': '%(asctime)s %(levelname)-8s %(message)s',
'level': 'warning',
'propagate': False,
},
'maintenance': {
'handler': 'django.logging.FileLogger',
'filename': _log_path + 'maintenance.log',
'format': '%(asctime)s %(name)-15s %(levelname)-8s 
%(message)s',

'level': 'info',
'propagate': False,
},
'': {
'handler': 'django.logging.FileLogger',
'filename': _log_path + 'info.log',
'format': '%(asctime)s %(name)-15s %(levelname)-8s 
%(message)s',

'level': 'info',
},
}

(Note: "propagate" is not implemented right now in Simon's code but is 
quite easy to add).


"Explicit handlers":

_log_path = '/var/log/service/'

LOGGING = {
'formatters': {
'exception': '%(asctime)s %(levelname)-8s %(message)s',
'default': '%(asctime)s %(name)-15s %(levelname)-8s 
%(message)s',

},
'handlers': {
'exception': {
'class': 'django.logging.FileLogger',
'filename': _log_path + 'exception.log',
'formatter': 'exception',
'level': 'warning',
},
'maintenance': {
'class': 'django.logging.FileLogger',
'filename': _log_path + 'maintenance.log',
'formatter': 'default',
'level': 'info',
},
'info': {
'class': 'django.logging.FileLogger',
'filename': _log_path + 'info.log',
'formatter': 'default',
'level': 'info',
}
},
'loggers': {
'exception': {'handlers': ['exception'], 'propagate': False},
'maintenance': {'handlers': ['maintenance'], 'propagate': 
False},

'': {'handlers': ['info']},
},
}

In both examples "implicit handlers" give shorter code and, to me, are 
more readable.


The main point that I'm trying to argue is that having handlers and 
loggers *separate* is not needed in real world scenarios. If you want 
your logging in several files you will have to define several handlers 
and several loggers anyway. "Implicit handlers" format accept this fact 
and couples both handler and logger definitions in one place.


The only thing that 

Re: Call for comment: #12624 Class based test runners

[Jeff Balogh]

On Sat, Jan 16, 2010 at 6:26 AM, Russell Keith-Magee
 wrote:
> Hi all,
>
> This is a quick call for comment on ticket #12624.
>
> This ticket proposes to make Django's test runner a class-based,
> rather than function based operation.

One thing: 
http://docs.djangoproject.com/en/dev/releases/1.2/#test-runner-exit-status-code
says that run_tests should return 0 for success or 1 for failure, but
your patch still returns the old `len(errors) + len(failures)`.

Other than that, this looks great.  I think it'll be easy to update
django-nose to the new system.  Thanks Russell!

Cheers,
jeff
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for comment: #12624 Class based test runners

[Jacob Kaplan-Moss]

On Sat, Jan 16, 2010 at 8:26 AM, Russell Keith-Magee
 wrote:
> Specifically, I want to make this change because I've had a couple of
> conversations in the recent past that have gone something like:
>
>  Them: How do you make Django do X before/during/after a test run?
>  Me: You write a custom test runner
>  Them: How do I do that?
>  Me: You copy this 50 line method and modify the 1 line that offends you.
>  Them: 1985 called - they want their code back.

Heh. I've had this exact conversation a few times myself.

IOW, +1 - this is a good idea.

Jacob
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for comment: #12624 Class based test runners

[Vitaly Babiy]

+1 for me, this will make my life a lot easier to implement a auto test
runner.

Is need to right up a proposal for change the code_change method a little,
but this is for another email.
Vitaly Babiy


On Sat, Jan 16, 2010 at 1:48 PM, Antoni Aloy  wrote:

> +1 for me
>
> El 16/01/2010 15:26, "Russell Keith-Magee" 
> escribió:
>
>
> Hi all,
>
> This is a quick call for comment on ticket #12624.
>
> This ticket proposes to make Django's test runner a class-based,
> rather than function based operation.
>
> This makes it much easier to modify the individual parts of the
> behavior of the Django test runner. For example if you want to:
>
>  - not destroy the databases at the end of the test run
>  - Use a TestRunner other than the TextTestRunner
>  - Not have DEBUG disabled during tests
>  - Do something else exotic before/during/after test execution
>
> you can subclass the basic test runner and override the method that
> implements the specific behavior that you want to change.
>
> Specifically, I want to make this change because I've had a couple of
> conversations in the recent past that have gone something like:
>
>  Them: How do you make Django do X before/during/after a test run?
>  Me: You write a custom test runner
>  Them: How do I do that?
>  Me: You copy this 50 line method and modify the 1 line that offends you.
>  Them: 1985 called - they want their code back.
>
> I don't expect this change to be especially controversial, but as a
> matter of good form I thought I raise it before committing. Comments
> welcome; barring feedback, I'll expect to commit this early next week.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Re: help needed: non-relational DB support

[Waldemar Kornewald]

On Fri, Jan 15, 2010 at 3:55 AM, Russell Keith-Magee
 wrote:
> And to be clear - a solid proposal isn't just "merge this branch". A
> patch/branch is one way to prove that you have thought about the
> problem in detail, but you also need to provide the discussion and
> description necessary to explain the nature of and reasoning behind
> the changes you want to make.

OK, I think I've found a better way to get very non-relational backend
support integrated with minimal changes to Django. The current patch
is very simple and it's not based on QueryData, but it already
supports the same feature set as our previous port. It's not yet
sufficient for emulating JOINs and maybe a few other "advanced"
features, but it looks like sql.Query and SQLCompiler would only need
minor modifications to be abstract enough for emulating advanced
features with nonrel DBs. Anyway, this can be added once we've
discussed whether the current solution is good enough.

The idea is to just use the SQLCompiler backend API and let the nonrel
backends interpret sql.Query.where and the other query attributes.
This required a few small changes to sql.Query and SQLCompiler.

The patch is this commit:
http://bitbucket.org/wkornewald/django-nonrel/changeset/71117e43ae33/

My reasoning behind the patch is the following:

1. Nonrel DBs don't distinguish between INSERT and UPDATE
On such DBs Model.save_base() shouldn't check if an entity already
exists. Instead, it should just always INSERT. For this I added
connection.features.distinguishes_insert_from_update.

2. Deleting related objects can be too costly on non-relational DBs
For this I added
connection.features.supports_deleting_related_objects. If it's False
no related objects will be deleted. Ideally, the backend should be
able to delegate this job to a background task. This is planned.

3. Multi-table inheritance needs JOINs
For this I added connection.features.supports_multi_table_inheritance.
If it's False Django will reject to save this model or run queries on
this model using that connection.
In the future, at least single inheritance could be emulated with a
ListField type. On App Engine multiple inheritance support would
require lots of custom datastore indexes and could lead to exploding
indexes. We can discuss this later.

4. sql.Query.has_results() used a trick that only works with SQL
I moved some of the code to SQLCompiler, so the backend can override it.

5. We need transactions which lock rows
I added a GAE-specific @commit_locked transaction decorator. This
should be moved into the backend layer, of course. I'd just like to
know if this is an option for 1.3 or not. On SQL this would be SELECT
... FOR UPDATE, but I don't know if all SQL DBs support it. Such a
decorator would be important to make get_or_create() and a few other
functions work 100% correctly, so it would also benefit the SQL layer.
I couldn't provide the respective SQL implementations, though.

Planned changes:

Delegate deletion of related objects to background task.

Maybe emulate multi-table inheritance with a ListField.

Nonrel DBs need special handling for primary key fields.
MongoDB and CouchDB store them in '_id' and App Engine uses a special
property which is not part of the field values dictionary. In order to
emulate JOINs we must store the column names of the primary keys used
in the sql.Query instance.

So, do you think this is a good path to take?

Bye,
Waldemar Kornewald

--
http://twitter.com/wkornewald
http://bitbucket.org/wkornewald/
http://allbuttonspressed.blogspot.com/
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Call for comment: #12624 Class based test runners

[Antoni Aloy]

+1 for me

El 16/01/2010 15:26, "Russell Keith-Magee" 
escribió:

Hi all,

This is a quick call for comment on ticket #12624.

This ticket proposes to make Django's test runner a class-based,
rather than function based operation.

This makes it much easier to modify the individual parts of the
behavior of the Django test runner. For example if you want to:

 - not destroy the databases at the end of the test run
 - Use a TestRunner other than the TextTestRunner
 - Not have DEBUG disabled during tests
 - Do something else exotic before/during/after test execution

you can subclass the basic test runner and override the method that
implements the specific behavior that you want to change.

Specifically, I want to make this change because I've had a couple of
conversations in the recent past that have gone something like:

 Them: How do you make Django do X before/during/after a test run?
 Me: You write a custom test runner
 Them: How do I do that?
 Me: You copy this 50 line method and modify the 1 line that offends you.
 Them: 1985 called - they want their code back.

I don't expect this change to be especially controversial, but as a
matter of good form I thought I raise it before committing. Comments
welcome; barring feedback, I'll expect to commit this early next week.

Yours,
Russ Magee %-)

--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Russell Keith-Magee]

On Sat, Jan 16, 2010 at 9:55 PM, Mat Clayton  wrote:
> Thanks, I've looked over both sets of code, seem's fine to me. The only
> complex stuff is that Indexes can only be created on certain field types, so
> additional checks may need doing. Also different index types are available
> on different storage engines, for MySQL. Would people want support for
> selecting the index type and direction?

I'd say don't bother. If you *really* want to fine tune your database,
you aren't going to use the ORM - you're going to use initial SQL
defintions, or leave it up to your handy neighborhood db admin. On top
of that, anything beyond a simple "CREATE INDEX foo ON bar (attr,
attr)" is going to start walking into areas that are specific to
individual databases.

db_index=True has been sufficient for indexes of all types so far; I'm
happy to continue the trend here.

> Finally char field index lengths need to be considered, I'm assuming the
> Meta field would have to take these into account as well.
> Also have reopened a ticket http://code.djangoproject.com/ticket/5805 to
> cover this

Cheers. I should have known this had been suggested before :-)

Yours,
Russ Magee %-)
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Mat Clayton]

Thanks, I've looked over both sets of code, seem's fine to me. The only
complex stuff is that Indexes can only be created on certain field types, so
additional checks may need doing. Also different index types are available
on different storage engines, for MySQL. Would people want support for
selecting the index type and direction?

Finally char field index lengths need to be considered, I'm assuming the
Meta field would have to take these into account as well.

Also have reopened a ticket http://code.djangoproject.com/ticket/5805 to
cover this

Mat

On Sat, Jan 16, 2010 at 1:34 PM, Russell Keith-Magee  wrote:

> On Sat, Jan 16, 2010 at 8:56 PM, Mat Clayton  wrote:
> > Great, have to confess I don't know the ORM internals at all, so not sure
> > how long this could take me, shall see what I can do. Also any
> indications
> > of where/what to start looking for would be appreciated.
>
> In the end, db_index_together will be a combination of
> unique_together's Meta handling, and db_index's index building. If you
> follow the threads that make those two features work, you should be
> able to cobble together something.
>
> > 'db_index_together' seems fair enough, will be trivial to change later
> > anyway.
> >
> > With respect to db support I assume all backend's need support in any
> patch?
>
> All backends will need to be supported before anything hits trunk.
>
> However, if you don't have the facilities to test a specific backends,
> it's ok to submit a patch that hasn't been fully tested. As long as
> you declare up front anything that you know to be untested, I'll
> probably accept a patch that makes allowances for the differences
> between backends and "looks right on visual inspection".
>
> Of course, if you *are* able to produce a patch that works for all
> backends, that would be excellent, and will dramatically improve the
> chances of this getting into trunk for 1.2. The more work I have to do
> to finish the patch, the less likely it is I will have the time to
> complete the work.
>
> Regardless of whether you attempt the patch or not, you should open a
> ticket so that the idea isn't forgotten.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>
>


-- 
-- 
Matthew Clayton | Founder/CEO
Wakari Limited

twitter http://www.twitter.com/matclayton

email m...@wakari.co.uk
mobile +44 7872007851

skype matclayton
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Russell Keith-Magee]

On Sat, Jan 16, 2010 at 8:56 PM, Mat Clayton  wrote:
> Great, have to confess I don't know the ORM internals at all, so not sure
> how long this could take me, shall see what I can do. Also any indications
> of where/what to start looking for would be appreciated.

In the end, db_index_together will be a combination of
unique_together's Meta handling, and db_index's index building. If you
follow the threads that make those two features work, you should be
able to cobble together something.

> 'db_index_together' seems fair enough, will be trivial to change later
> anyway.
>
> With respect to db support I assume all backend's need support in any patch?

All backends will need to be supported before anything hits trunk.

However, if you don't have the facilities to test a specific backends,
it's ok to submit a patch that hasn't been fully tested. As long as
you declare up front anything that you know to be untested, I'll
probably accept a patch that makes allowances for the differences
between backends and "looks right on visual inspection".

Of course, if you *are* able to produce a patch that works for all
backends, that would be excellent, and will dramatically improve the
chances of this getting into trunk for 1.2. The more work I have to do
to finish the patch, the less likely it is I will have the time to
complete the work.

Regardless of whether you attempt the patch or not, you should open a
ticket so that the idea isn't forgotten.

Yours,
Russ Magee %-)
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Mat Clayton]

Great, have to confess I don't know the ORM internals at all, so not sure
how long this could take me, shall see what I can do. Also any indications
of where/what to start looking for would be appreciated.

'db_index_together' seems fair enough, will be trivial to change later
anyway.

With respect to db support I assume all backend's need support in any patch?

Mat

On Sat, Jan 16, 2010 at 12:27 PM, Russell Keith-Magee <
freakboy3...@gmail.com> wrote:

> On Sat, Jan 16, 2010 at 7:52 PM, Mat Clayton  wrote:
> > I know this isn't really related, but figure I would bring it up here.
> When
> > working on optimizing our django app, it became very clear that one of
> the
> > biggest issues as usual is Indexes, particularly multi column indexes,
> which
> > at present django doesnt appear to support.
> > I know ticket 373 http://code.djangoproject.com/ticket/373 covers multi
> > column primary keys, but are there any intentions to cover multi column
> > indexes as a smaller issue prior to this. The reason I bring this up, is
> > after checking the mysql logs we discovered our most costly queries are
> > actually due to django comments and the lack of multicolumn indexes,
> after
> > adding the correct index query times went from 300mS to <10mS giving us a
> > nice speed boost.
> > If this is likely to be a nice addition to django, we would be willing to
> > commit some resources to trying to get it into core, any thoughts on
> this?
>
> Sounds like a reasonable suggestion to me. I've had similar thoughts
> over the last couple of weeks as I've been tuning the performance of
> the database at work.
>
> > Or does anyone else have a need for it? Or would we be standing on the
> toes
> > of the 373 ticket?
>
> I don't think it stands on the toes of #373 either. #373 will probably
> imply the need for multi-column indexes, but that doesn't remove the
> broader need for multi-column indexes.
>
> > I was thinking of cloning the unique_together style and adding an
> > index_together Meta option, thoughts?
>
> A Meta option sounds like the right way to define it. Getting out my
> bikeshed-painting brush, I'd probably call the Meta option
> 'db_index_together', so as to maintain the parity between
> 'unique->unique_together' and 'db_index->db_index_together'.
>
> We have about 10 days until the 1.2 beta freeze; if you can work up a
> patch quickly, I can see about getting this in for 1.2. Any
> suggestions on multi-column indexes that we should add to shipped
> Django models are also welcome (i.e., the indexes that are needed for
> comment models).
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>
>


-- 
-- 
Matthew Clayton | Founder/CEO
Wakari Limited

twitter http://www.twitter.com/matclayton

email m...@wakari.co.uk
mobile +44 7872007851

skype matclayton
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Russell Keith-Magee]

On Sat, Jan 16, 2010 at 7:52 PM, Mat Clayton  wrote:
> I know this isn't really related, but figure I would bring it up here. When
> working on optimizing our django app, it became very clear that one of the
> biggest issues as usual is Indexes, particularly multi column indexes, which
> at present django doesnt appear to support.
> I know ticket 373 http://code.djangoproject.com/ticket/373 covers multi
> column primary keys, but are there any intentions to cover multi column
> indexes as a smaller issue prior to this. The reason I bring this up, is
> after checking the mysql logs we discovered our most costly queries are
> actually due to django comments and the lack of multicolumn indexes, after
> adding the correct index query times went from 300mS to <10mS giving us a
> nice speed boost.
> If this is likely to be a nice addition to django, we would be willing to
> commit some resources to trying to get it into core, any thoughts on this?

Sounds like a reasonable suggestion to me. I've had similar thoughts
over the last couple of weeks as I've been tuning the performance of
the database at work.

> Or does anyone else have a need for it? Or would we be standing on the toes
> of the 373 ticket?

I don't think it stands on the toes of #373 either. #373 will probably
imply the need for multi-column indexes, but that doesn't remove the
broader need for multi-column indexes.

> I was thinking of cloning the unique_together style and adding an
> index_together Meta option, thoughts?

A Meta option sounds like the right way to define it. Getting out my
bikeshed-painting brush, I'd probably call the Meta option
'db_index_together', so as to maintain the parity between
'unique->unique_together' and 'db_index->db_index_together'.

We have about 10 days until the 1.2 beta freeze; if you can work up a
patch quickly, I can see about getting this in for 1.2. Any
suggestions on multi-column indexes that we should add to shipped
Django models are also welcome (i.e., the indexes that are needed for
comment models).

Yours,
Russ Magee %-)
-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: DB optimization docs

[Mat Clayton]

I know this isn't really related, but figure I would bring it up here. When
working on optimizing our django app, it became very clear that one of the
biggest issues as usual is Indexes, particularly multi column indexes, which
at present django doesnt appear to support.

I know ticket 373 http://code.djangoproject.com/ticket/373 covers multi
column primary keys, but are there any intentions to cover multi column
indexes as a smaller issue prior to this. The reason I bring this up, is
after checking the mysql logs we discovered our most costly queries are
actually due to django comments and the lack of multicolumn indexes, after
adding the correct index query times went from 300mS to <10mS giving us a
nice speed boost.

If this is likely to be a nice addition to django, we would be willing to
commit some resources to trying to get it into core, any thoughts on this?
Or does anyone else have a need for it? Or would we be standing on the toes
of the 373 ticket?

I was thinking of cloning the unique_together style and adding an
index_together Meta option, thoughts?

Mat

On Sat, Jan 16, 2010 at 3:32 AM, Luke Plant  wrote:

> I've added these docs now, or at least a good first stab at them.
> Suggestions for improvements are welcome, patches are more welcome, as
> always :-)
>
> I backported to 1.1.X, and tried to remove any anachronisms.
>
> Regards,
>
> Luke
>
> --
> "Outside of a dog, a book is a man's best friend... inside of a
> dog, it's too dark to read."
>
> Luke Plant || http://lukeplant.me.uk/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>
>


-- 
-- 
Matthew Clayton | Founder/CEO
Wakari Limited

twitter http://www.twitter.com/matclayton

email m...@wakari.co.uk
mobile +44 7872007851

skype matclayton
-- 

You received this message because you are subscribed to the Google Groups "Django developers" group.

To post to this group, send email to django-develop...@googlegroups.com.

To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.