Igor wrote: >Now i hope at the end of the day, >the guys who doing data mining/statistical >analysis will finally shut up and happily >be able to work with more bloat without >need of learning a ways to properly >manage memory & resources, and >implement them finally.
The actual problem is of course having to work with all that data before you understand the structure. Or highly interconnected structures with unpredictable access patterns. Partial graphs are nice, once you understand how to partition. Needing to understand how to partition first is a dependency I'd rather avoid. >Because even if you can fit all data in >memory, consider how much time it takes >for GC to scan 4+ Gb of memory, That's often not what is happening. The large data is mostly static, so gets moved out of new space very quickly. Otherwise working with large data quickly becomes annoying indeed. I fully agree with you on that. Stephan
