>> The main slowdown is that messages depend on previous messages, so I >> need to match them together. This becomes slow if I have to search >> through too many messages. >> >> It is somewhat inefficient and I am working on optimising the >> algorithm as we speak, especially in these areas: >> 1) replacing lists with dictionaries, since lists get horribly slow >> as they grow. > > After certain size, operations on dict become horribly slow too [0]. I'd go > for bsddb if the dict size is too big, it offers the same interface than a > dict [1].
Yes - it's definitely worth looking at some kind of tree data structure more balanced than a linked-list if you're doing non-sequential record processing. bsddb is the obvious first choice. You might want also to have a look at skip lists (which theoretically have better performance). http://en.wikipedia.org/wiki/Skip_list http://dekorte.com/projects/opensource/SkipDB/ http://infohost.nmt.edu/tcc/help/lang/python/examples/pyskip/ -- steev http://www.daikaiju.org.uk/~steve/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Python Ireland" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.ie/group/pythonireland?hl=en -~----------~----~----~----~------~----~------~--~---
