Re: Denormalisation, magic, and is it really that useful?

2008-09-22 Thread Winsley von Spee

Hi,

this would be very useful for a lot of use cases. View counts that
originally came from a foo viewed bar table with timestamps for every
viewer, or storing some sort of sum calculation from a list of
related objects. Basically all that stuff you would display in an
"overview table" of some sort for which you would normally use
aggregation wich gets very slow very fast on large data sets.

I have written a generic denormalizer to store whole data sets
in denormalized tables to speed up searching.

It basically consists of a Metaclass that constructs a Model out of a
values queryset and a base class that adds method to generate sql to
keep the table up to date. This way most of the time you only have
to write the values query set and for special cases you can still add
"custom" Fields if you have to.

The tables get updated periodically, but it would be quite possible to
do this with signals instantly.

A previous version even generated triggers to keep the tables updated,
but this was dropped afterwards due to other reasons.

Queryset refactor did help a lot to make this sort of stuff easier.
Only quirk right now is that you have to use extra and "force"
queryset.query to use aggregation wich you'll need very often when
denormalizing.

I think a django.contrib.denormalization app would be a very nice
thingy.

+1 from me ;).

Regards


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---



Re: TimestampField

2007-11-12 Thread Winsley von Spee

Hello,

thanks for the documentation link, I didn't know that page so far ...
and you were right I should implement get_db_prep_lookup as well.

Right now I the conversion from datetime.datetime to unixtimstamp works
quite well, my only problem is that if access objects from the database
the unixtimestamp is not converted to datetime.datetime.

From my understanging to_python is responsible for that, but it doesn't
even get called if I access foo.timestampfield. Only if I access the
Model in the admin interface _get_val_from_obj gets called. 

Here's my current version, perhaps someone can give me a hint were to
convert the unixtimestamp in the database to datetime.datetime, so that
I don't get an int when I access foo.timestampfield.



from django.db.models.fields import DateTimeField, IntegerField, Field
from datetime import datetime
from time import mktime

class TimestampField(DateTimeField, IntegerField):
def get_db_type(self):
return super(IntegerField, self).get_db_type()

def get_internal_type(self):
return super(DateTimeField, self).get_internal_type()

def _get_val_from_obj(self, obj):
return self._from_unixtime(getattr(obj, self.attname))

def get_db_prep_save(self, value):
return super(IntegerField, self).get_db_prep_save(
self._to_unixtime(value))

def _to_unixtime(self, value):
if type(value) is datetime:
value = int(mktime(value.timetuple()))
elif type(value) is str:
value = datetime.strptime(value, "%Y-%m-%d
%H:%M:%S")
value = int(mktime(value.timetuple()))
else:
value = int(value)
return value

def _from_unixtime(self, value):
try:
return datetime.fromtimestamp(float(value))
except Exception, e:
raise e

def get_db_prep_lookup(self, lookup_type, value):
value = self._to_unixtime(value)
return super(IntegerField, self).get_db_prep_lookup(
lookup_type, value)

def to_python(self, value):
print "%s(%s)" % ("to_python", value,)
if value is None:
return value
try:
return datetime.fromtimestamp(value)
except (TypeError, ValueError):#Exception, e:
raise validators.ValidationError(
'%s couldn`t be converted to
datetime' % value)
 



On Fri, 09 Nov 2007 12:08:23 +1100
Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:

> 
> On Thu, 2007-11-08 at 16:40 +0100, Winsley von Spee wrote:
> > Hello,
> > 
> > I have implemeted a TimestampField that maps betweem unixtimestamps
> > and datetime.datetime objects, so i can use Datetime in Django
> > while not having to convert all timestamps to datetime fields yet.
> > I need that for downwards compatibility for an old application that
> > uses the same database. The Plan is to use it until I can
> > reimplement the old application in django and afterwards switch to
> > Datetime and DateFields entirely.
> > 
> [...]
> 
> > Can someone comment wether this is a "proper" approach or wether
> > I'll perhaps experience other problems with this solution?
> 
> Have you read [1] and omitted all the other methods you might need
> because you've checked they really aren't needed? For example, usually
> if you write get_db_prep_save(), you need get_db_prep_lookup() as
> well. And, in your case, I'd be tempted to write things so that if
> somebody assigned a datetime to the attribute, instead of a
> timestamp, you changed it in db_prep_save() and changed the attribute
> value as well, so that it's always consistent after a call to save().
> 
> A lot of this depends on what you're going to use it for, but have a
> read of the new documentation, if you haven't seen it before, and
> check that you know why you've only implemented those three methods.
> 
> [1]
> http://www.djangoproject.com/documentation/custom_model_fields/#useful-methods
> 
> Regards,
> Malcolm
> 

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---



TimestampField

2007-11-08 Thread Winsley von Spee

Hello,

I have implemeted a TimestampField that maps betweem unixtimestamps and
datetime.datetime objects, so i can use Datetime in Django while not
having to convert all timestamps to datetime fields yet. I need that for
downwards compatibility for an old application that uses the same
database. The Plan is to use it until I can reimplement the old
application in django and afterwards switch to Datetime and DateFields
entirely.


from django.db.models.fields import DateTimeField, IntegerField, Field
from datetime import datetime
from time import mktime

class TimestampField(DateTimeField):
def get_internal_type(self):
return "IntegerField"

def _get_val_from_obj(self, obj):
try:
return datetime.fromtimestamp((getattr(obj,
self.attname)))
except:
return datetime.fromtimestamp(0)

def get_db_prep_save(self, value):
if type(value) is datetime:
value = int(mktime(value.timetuple()))
else:
value = int(value)
return Field.get_db_prep_save(self, value)


Can someone comment wether this is a "proper" approach or wether I'll
perhaps experience other problems with this solution?


Regards


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---