On Friday, May 1, 2015 at 11:48:21 AM UTC-4, Tim Holy wrote: > > On Friday, May 01, 2015 08:03:31 AM Scott Jones wrote: > > Still, same issue as I described above... probably better to increase by > 2x > > up to a point, and then by chunk sizes, where the chunk sizes might > slowly > > get larger... > > I see your point, but it will also break the O(nlogn) scaling. We couldn't > hard-code the cutoff, because some people run julia on machines with 4GB > of RAM > and others with 1TB of RAM. So, we could query the amount of RAM available > and > switch based on that result, but since all this would only make a > difference > for operations that consume between 0.5x and 1x the user's RAM (which to > me > seems like a very narrow window, on the log scale), is it really worth the > trouble? > > --Tim >
For what I was doing, yes, it was definitely worth the trouble, because you'd have systems with 10s of thousands of processes (the limit was 64K on a single node), and you had to be very careful about not using up too much memory, and ending up thrashing... Very different than when you maybe have a process for each core, and you have lots of memory for each one... Different usage... different performance issues...
