2011/12/12 Falko Bräutigam <[email protected]>: > It takes ~30s to load 10.000 entities (consisting of ~100k ValueComposites > of different types). After loading all this eats up ~100MB of RAM. This is > to slow and to much.
1. Which store are you using? Neo4j *may* use less RAM than for instance JDBM. Loading speed is highly EntityStore dependent, and whether you are using indexing to retrieve those 10,000 entities. The "raw" speed of Qi4j is probably ~10k entities per second, slowing down as serialization and I/O become factors. 2. Not sure what you are saying. 100,000 values in total? So on average, each value taken 1kB is "too much"? Well, what is the composition of those values, otherwise it is hard to analyze. But I think that the general problem is deep inside Qi4j's data management, which relies on 4 HashMaps that probably occupy a few kB each, quickly eating away at the memory consumption. Perhaps some "hinting" system could improve the consumption size. Perhaps even more clever lookup mechanisms, especially when the number of properties per entity is low. > 10K entities are not much for a GIS application. Given the current memory > foodprint (and 1GB Java heap) not even 10 users can work with the > application concurrently. Where does the 1GB heap comes from in a multi-user application? It is hard to discuss in the abstract when your argument is based on concrete examples. > Usually a GIS application works in a pipelined mode when rendering features > (entities). Memory is never a problem with that architecture. Unfortunatelly > Qi4j holds all entities of an UoW in memory. We discussed this earlier on > this list. So I added a cache SPI to UnitOfWorkInstance. This works but it > does not actually cure the problem of memory consumption because of the time > needed to re-instantiate the entities. Interestingly enough, I was sketching on another architecture today, where I observed the "read-only" case being very separated (well, it is CQRS related) and UnitOfWork isolation not being an issue. If 2.x would move towards a "read-only"-mode of UoWs, could that solve "streaming", i.e. as soon as you are done "rendering" you simply drop the Entity and it is not held in memory? If so (to help you in 1.4), does that mean that your "render task" could be broken up to a series of smaller chunks rendered one at a time, or are there other constraints preventing this. Cheers -- Niclas Hedhman, Software Developer http://www.qi4j.org - New Energy for Java I live here; http://tinyurl.com/3xugrbk I work here; http://tinyurl.com/6a2pl4j I relax here; http://tinyurl.com/2cgsug _______________________________________________ qi4j-dev mailing list [email protected] http://lists.ops4j.org/mailman/listinfo/qi4j-dev
