On May 31, 2012, at May 31, 20121:21 PM, Paddy Byers wrote: > Hi, > > If a collection is of "indefinite size" then it should not be loaded entirely > in to memory in order to accomplish computation, it should be streamed. In > the case of streaming the collection you can iteratively perform the > computation without locking up the process indefinitely and without taking on > a large memory footprint of either the original collection or even what has > already been processed. > > Yes, you stream application data in and out of your application. But I'm > talking about application state, such as anything you have on a > per-connection or per-token basis, for example. It's in memory because it's > your application, not your application's data.
If that data is of "indefinite size" then it needs to be out of process. This is why Redis and memcache exist. I don't see any argument for computationally intensive tasks on application data of "indefinite size" that much be entirely in process memory at one time. > > Thanks - Paddy
