Ivan Peikov
Mon, 31 May 2010 06:44:03 -0700
Peter, hi again, I believe you hit a bug in the memory configuration calculation code. We're fixing this bug right now. As a temporary workaround you need to reduce the cache-memory/tuple-index-memory 4 times or to increase the JVM memory 4 times.
We're really sorry for the inconvenience caused. Cheers, Ivan On Monday 31 May 2010 15:47:30 Peter Kostelnik, PhD. wrote: > we'll .. for default config (+ build-pcsot=true), the run was: > ./import <opts> -Xmx1400m > > the memory consumption chart showed slow increase of resources which then > balanced between cca. 1200m-1400m until the end of the process .. > after cca. 8 hours, the data were inside .. > > for the configuration below and run: > ./import <opts> -Dcache-memory=7000m -Xmx9000m > > after half an hour, the memory consumtion chart showed increase near > 9000m, and then the memory consumption jumped between X-9000m, where X was > increasing up to 9000m and then came the heap-space :) .. > > what do you think? .. cheers, > Peter K. > > > So which one of the configurations seems to consume too much memory? > > When you specify -Dcache-memory=7000m this means that 7000m will be > > allocated > > for cache usage (different cache purposes). Do you see a lot more than > > that? > > > > On Monday 31 May 2010 15:24:20 Peter Kostelnik, PhD. wrote: > >> hi, > >> > >> thanks, ivan, the current configuration is as follows (right now we are > >> experimenting with various configs) :: > >> > >> "ruleset" -> "empty", > >> "build-pcsot" -> "true", > >> "build-ptsoc" -> "true", // we'll try to turn it off > >> "cache-memory" -> "384M", // the point for experiments > >> "predicate-memory" -> "56M", // queries don't use wildcard predicates > >> "enable-optimization" -> "true", > >> "entity-index-size" -> "20000000", > >> "fts-memory" -> "0M", // we are building our own lucene index > >> "storage-folder" -> "bigowlim-store", > >> "repository-type" -> "file-repository", > >> "console-thread" -> "false" > >> > >> last time, the run was something like: > >> ./import <options> -Dcache-memory=7000m -Xmx9000m -Xms256m > >> > >> all data are imported through conn.add(x, y, z, ctx1, ctx2) in a single > >> transaction commited when all data are inside .. > >> > >> (btw, on 3.2.a7 snapshot we were able to get them in with practically > >> default configuration) > >> > >> thanks in advance, > >> Peter K. > >> > >> > Hi Peter, > >> > > >> > The tripleset component is a 5-th dimension of the RDF space (subject, > >> > predicate, object and context being the first four). This component > >> > cannot be > >> > used through the Sesame API which is why you never knew it existed. In > >> > short, > >> > you don't need to enable this index, it won't help. > >> > > >> > You need to tune the memory parameters (e.g. tuple-index-memory, > >> > predicate-memory, cache-memory, etc.) if you experience memory issues. > >> > Can you post here your current configuration and also how much memory > >> > >> it > >> > >> > requires > >> > to load 100M statements? > >> > > >> > > >> > Cheers, > >> > Ivan > >> > > >> > On Monday 31 May 2010 15:03:32 Peter Kostelnik, PhD. wrote: > >> >> hi there .. > >> >> > >> >> I'd like to ask, what exactly means the tripleset, for which the > >> > >> ptsoc > >> > >> >> index can be build? .. we are wandering which indices to build when > >> >> loading the data (as we are often using context based querying, I > >> > >> know, > >> > >> >> that we have to build pcsot for sure .. ) > >> >> > >> >> btw, which configuration parameters can affect the load of data (e.g. > >> >> enable-optimization?)? .. we've faced the strange memory consuming > >> >> behaviour when loading cca. 100 milions of statements .. > >> >> > >> >> thanks in advance, > >> >> Peter K. > >> >> > >> >> _______________________________________________ > >> >> OWLIM-discussion mailing list > >> >> OWLIM-discussion@ontotext.com > >> >> http://ontotext.com/mailman/listinfo/owlim-discussion _______________________________________________ OWLIM-discussion mailing list OWLIM-discussion@ontotext.com http://ontotext.com/mailman/listinfo/owlim-discussion