The cycle happened at least once a day. Several people need to do something to the data. (In sequence). Once the last one completes the assembly of the final deliverable blob is starting. Lots of data is read from the DB and it takes long time.
If I'll have this magic machine with 1TB RAM available and my entire DB was in RAM - the assembly of the deliverable would be 10 times faster. I currently have hibernate in front of the DB and the bottom line is big latency. I'm considering hibernate-memcached to get my DB cached. Thanks, -Michael On Sep 20, 11:02 pm, Dustin <[email protected]> wrote: > On Sep 20, 8:25 pm, "MikeG." <[email protected]> wrote: > > > The reason is that my transactions are reading large amount of data > > from the DB to generate > > a deliverable blob. > > All of them all the time? It may be possible to optimize your > source for whatever type of efficiency you're seeking (latency? > throughput?). > > > Now, to hold 1TB in memory I need a cluster and not a small one. > > It only takes 16 64GB nodes to give you 1 TB of RAM. Of course, not > all will be usable. It depends on how many objects you plan to have > loaded and other overhead. > > > Also, memcached (the C implementation) has a 1MB record size limit. 1) > > What's the reason for that? > > The limit is configurable, but defaults to 1MB. If you are caching > items larger than 1MB, latency is probably not your issue, so > memcached may not provide the most benefit. It'll do it, of course, > but as you get larger, it starts to be less beneficial than, say, > varnish. > > > 2) Can it be changed (with a hacked private version) > > -I > > > 3) Does Jmemcached has the same limit? > > You should try your design at a smaller scale and see what > components benefit you.
