On 10/06/14 05:41, Damian Johnson wrote: >>> let me make one remark about optimizing Postgres defaults: I wrote quite >>> a few database queries in the past, and some of them perform horribly >>> (relay search) whereas others perform really well (ExoneraTor). I >>> believe that the majority of performance gains can be achieved by >>> designing good tables, indexes, and queries. Only as a last resort we >>> should consider optimizing the Postgres defaults. >>> >>> You realize that a searchable descriptor archives focuses much more on >>> database optimization than the ExoneraTor rewrite from Java to Python >>> (which would leave the database untouched)? >> >> Are other datastore models such as splunk or MongoDB useful? >> [splunk has a free yet proprietary limited binary... those having >> historical woes and takebacks, mentioned just for example here.] > > Earlier I mentioned the idea of Dynamo. Unless I'm mistaken this lends > itself pretty naturally to addresses as a hash key, and descriptor > dates as the range key. Lookups would then be O(log(n)) where n is the > total number of descriptors an address has published (... that is to > say very, very quick). > > This would be a fun project to give Boto a try. *sigh*... there really > should be more hours in the day...
Quoting my reply to Damian to a similar question earlier in the thread: > I'm wary about moving to another database, especially NoSQL ones and/or > cloud-based ones. They don't magically make things faster, and Postgres is > something I understand quite well by now. [...] Not saying that DymanoDB > can't be the better choice, but switching the database is not a priority for > me. If somebody wants to give, say, MongoDB a try, I'd be interested in seeing the performance comparison to the current Postgres schema. When you do, please consider all three search_* functions that the current schema offers, including searches for other IPv4 addresses in the same /24 and other IPv6 addresses in the same /48. All the best, Karsten _______________________________________________ tor-dev mailing list [email protected] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
