I just added code in trunk so block size for string and array store can be configured when the store is created. This will be available in the 1.1 release but if you want to try it out now use 1.1-SNAPSHOT and create a new store like this:
Map<String,String> config = new HashMap<String, String>(); config.put( "string_block_size", "60" ); config.put( "array_block_size", "300" ); // create a new store with string block size 60 and array block size 300 new EmbeddedGraphDatabase( "path-to-db-that-do-not-exist", config ).shutdown(); The default value (120 bytes) was picked to fit common/avg size string/array properties in one block since it will be slower to load a property that is spread out on many blocks. Since datasets vary a lot in string size / array size and the like I think it is better to have it configurable at creation time. When tweaking these values remember that strings will consume twice the string length in bytes so a string block size of 60 will be able to fit a string of length 30 in a single block. Regarding scaling 1.0 and 1.1 releases have a limit of 4 billion records / store file so if you need to store 4 billion strings you have to make sure every string fits in a single block. This limit will be increased to 32 billion or more in the 1.2 release. -Johan On Mon, Jun 7, 2010 at 4:27 PM, Biren Gandhi <[email protected]> wrote: > Similar issue on my side as well. Test data is ok, but production data > (100 million+ objects, 200 relationships per object and 10 properties > per object, with multi-million queries per day about search and > traversal) would need clear disk sizing calculations due to iops and > other hardware limits in a monolithic storage model. > > Has anyone been able to use neo4j succeessfully in scaling needs > similar to mentioned avove? > > -b > > On Jun 7, 2010, at 4:45 AM, Craig Taverner <[email protected]> wrote: > >>> Is there a specific constrain on disk space? Normally disk space >>> isn't >>> a problem... it's cheap and there's usually loads of it. >>> >> >> Actually for most of my use cases the disk space has been fine. >> Except for >> one data source, that surprised me by expanding from less than a gig >> of >> original binary data, to over 20GB database. While this too can be >> managed, >> it was just a sample, and so I have yet to see what the customers >> 'real >> data' will do to the database (several hundred times larger, I'm >> expecting). >> When we get to that point we will need to decide how to deal with it. >> Currently we 'solve' the issue by allowing the user to filter out >> data on >> import, so we don't store everything. This will not satisfy all users, >> however. _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
