Re: Scaling/Parallel patterns for a View executing complex database transaction

Exactly! We did prototype with Mongo and Riak to see if there were significant performance gains and the answer was no. Well tuned Postgres performed as well and as others. Add to this the complexities and risks associated with introducing a new component in your stack as against something

Re: Scaling/Parallel patterns for a View executing complex database transaction

On Sun, Jun 14, 2015 at 3:13 PM, Peter of the Norse wrote: > Almost all of them are faster at key/value returns than PostgreSQL. i seriously doubt that. most "fast" key-value databases are only fast if you either: keep all data in RAM, forget about durability, or

Re: Scaling/Parallel patterns for a View executing complex database transaction

On May 14, 2015, at 10:30 PM, Me Sulphur wrote: > > * It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5 > key and JSON for value. The look ups depend on the business logic but are > necessary. Nevertheless, there are no more than 10-12 queries

Re: Scaling/Parallel patterns for a View executing complex database transaction

Hi Stephen, We did try with Redis (instead of memcached) but the memory usage Redis is 2.5x-3x the volume of data, the economics of having entire data in memory then breaks down. Also because of the large degree of variation/random access patterns, there is little benefit in using an LRU

Re: Scaling/Parallel patterns for a View executing complex database transaction

If it's just key-value storage, adding a memcached layer sounds like a good thing to investigate. Do the tuples frequently change? On Thu, May 14, 2015 at 11:30 PM, Me Sulphur wrote: > Hi Russ, > > Thanks! While there are some pointers that we can pick up from your answer, >

Re: Scaling/Parallel patterns for a View executing complex database transaction

Hi Russ, Thanks! While there are some pointers that we can pick up from your answer, let me add some more details. * It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5 key and JSON for value. The look ups depend on the business logic but are necessary. Nevertheless,

Re: Scaling/Parallel patterns for a View executing complex database transaction

On Thu, May 14, 2015 at 6:03 PM, Me Sulphur wrote: > Stack: Django 1.7 + Postgres 9.3 + Linux (No caching) > > Our application has a view which is called/executed very frequently. The > view receives, parses and responds a JSON. > > In between request and response, there are

Scaling/Parallel patterns for a View executing complex database transaction

Stack: Django 1.7 + Postgres 9.3 + Linux (No caching) Our application has a view which is called/executed very frequently. The view receives, parses and responds a JSON. In between request and response, there are about 3-5 inserts and around 1200-5000 look ups depending upon some if..else