Re: [qi4j-dev] OpenRDF performance problem...

Quoting Rickard Öberg : Hi, I did some tests, and found a number of ways to fix this issue. First of all, whenever you do benchmarking, make sure you run the test repeatedly. Running it once will not allow the JIT to kick in. Run tests about 10 times instead, in the same go, to get more st

Re: [qi4j-dev] OpenRDF performance problem...

On 2010-02-19 15.56, Arvid Huss wrote: Hi, org.qi4j.index.rdf.RDFPerformanceTest fails with "initializationError0" java.lang.Exception: No runnable methods Stupid JUnit. I added a dummy tes

Re: [qi4j-dev] OpenRDF performance problem...

Hi, org.qi4j.index.rdf.RDFPerformanceTest fails with "initializationError0" java.lang.Exception: No runnable methods /Arvid 2010/2/19 Rickard Öberg > Hi, > > I did some tests, and found a

Re: [qi4j-dev] OpenRDF performance problem...

Hi, I did some tests, and found a number of ways to fix this issue. First of all, whenever you do benchmarking, make sure you run the test repeatedly. Running it once will not allow the JIT to kick in. Run tests about 10 times instead, in the same go, to get more stable numbers. Second, I ch

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Georg Ragaller : Thanks for advice, will try that next. :) But, since this heap-dump is just a snapshot in time, it will show only 'leaky-like' problems. So if the underlying problem is, that there are a lot of very short living objects, it probably won't help you. Cheers, Georg Yea

Re: [qi4j-dev] OpenRDF performance problem...

> Thanks for advice, will try that next. :) But, since this heap-dump is just a snapshot in time, it will show only 'leaky-like' problems. So if the underlying problem is, that there are a lot of very short living objects, it probably won't help you. Cheers, Georg Stanislav Muhametsin wrote: Qu

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Georg Ragaller : For memory profiling I successfully used http://www.eclipse.org/mat/ . Simply run your test outside eclipse, make a heap-dump (I prefer JConsole) and analyze the heap-dump after the application has finished. MAT offers a quite intelligent mechanism to give you hints

Re: [qi4j-dev] OpenRDF performance problem...

For memory profiling I successfully used http://www.eclipse.org/mat/ . Simply run your test outside eclipse, make a heap-dump (I prefer JConsole) and analyze the heap-dump after the application has finished. MAT offers a quite intelligent mechanism to give you hints about the source of the evil.

Re: [qi4j-dev] OpenRDF performance problem...

I did a memory analysis on my 200-entity creating & removing test. Among many allocated objects, there were following: Class | Live instances | Active size | Total Instances | Total Size | Avg age --++-+-++ byte[]| 17 0

Re: [qi4j-dev] OpenRDF performance problem...

On 2010-02-18 11.50, Niclas Hedhman wrote: On Thu, Feb 18, 2010 at 11:47 AM, Stanislav Muhametsin wrote: It's 05:40 AM and I think it's time to sleep. ;) One thing the profiler could tell me certainly: the notifyChanges() method in RdfIndexingService.RdfEntityIndexerMixin is taking most time (

Re: [qi4j-dev] OpenRDF performance problem...

On Thu, Feb 18, 2010 at 11:47 AM, Stanislav Muhametsin wrote: > It's 05:40 AM and I think it's time to sleep. ;) One thing the profiler > could tell me certainly: the notifyChanges() method in > RdfIndexingService.RdfEntityIndexerMixin is taking most time (around 60-80%) > when performing this tes

Re: [qi4j-dev] OpenRDF performance problem...

It's 05:40 AM and I think it's time to sleep. ;) One thing the profiler could tell me certainly: the notifyChanges() method in RdfIndexingService.RdfEntityIndexerMixin is taking most time (around 60-80%) when performing this test. So, it's somewhere in OpenRDF... the question lies, where. :

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Niclas Hedhman : On Thu, Feb 18, 2010 at 11:19 AM, Stanislav Muhametsin wrote: Yeah that is what I currently am doing - a simple test with PreferenceEntityStore for service prefs, MemoryEntityStore (as per EntityTestAssembler) for storing entities, and RDF indexing service. I have one

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Stanislav Muhametsin : Quoting Stanislav Muhametsin : Quoting Rickard Öberg : Can you make a simple test: remove the RDF indexing from your app, so that only your UoW code and the EntityStore is running. Are you then seeing the same performance? Is the problem in Qi4j or are we l

Re: [qi4j-dev] OpenRDF performance problem...

On Thu, Feb 18, 2010 at 11:19 AM, Stanislav Muhametsin wrote: > Yeah that is what I currently am doing - a simple test with > PreferenceEntityStore for service prefs, MemoryEntityStore (as per > EntityTestAssembler) for storing entities, and RDF indexing service. I have > one entity type (Example

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Stanislav Muhametsin : Quoting Rickard Öberg : Can you make a simple test: remove the RDF indexing from your app, so that only your UoW code and the EntityStore is running. Are you then seeing the same performance? Is the problem in Qi4j or are we looking at Sesame being the culpri

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Rickard Öberg : On 2010-02-18 11.02, Stanislav Muhametsin wrote: I was creating I'd say around thousand entities in one uow, before I divided it to do same work in many smaller uow's. I managed to make the profiler work again (somewhat), it's still unstable as hell. So far it seems tha

Re: [qi4j-dev] OpenRDF performance problem...

On 2010-02-18 11.02, Stanislav Muhametsin wrote: I was creating I'd say around thousand entities in one uow, before I divided it to do same work in many smaller uow's. I managed to make the profiler work again (somewhat), it's still unstable as hell. So far it seems that even when doing 200 enti

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Rickard Öberg : On 2010-02-18 03.07, Stanislav Muhametsin wrote: I ran to this same issue myself a couple days ago. Well, my issue is regarding strictly to creating of entities. I'm not sure about deleting, as it seems that it gets stuck at 41188 [main] INFO org.openrdf.sail.nativerdf.

Re: [qi4j-dev] OpenRDF performance problem...

On 2010-02-18 03.07, Stanislav Muhametsin wrote: I ran to this same issue myself a couple days ago. Well, my issue is regarding strictly to creating of entities. I'm not sure about deleting, as it seems that it gets stuck at 41188 [main] INFO org.openrdf.sail.nativerdf.NativeStore - Waiting for

Re: [qi4j-dev] OpenRDF performance problem...

Quoting Niclas Hedhman : On Mon, Jan 4, 2010 at 5:30 PM, Niclas Hedhman wrote: But, no slow performance must be a general problem. I am not happy that deleting an entity in Qi4j will take 100s of milliseconds. Brr 2) are you sure the latest Sesame is used? The latest release has i

Re: [qi4j-dev] OpenRDF performance problem...

On Mon, Jan 4, 2010 at 5:30 PM, Niclas Hedhman wrote: > But, no slow performance must be a general problem. I am not happy > that deleting an entity in Qi4j will take 100s of milliseconds. Brr >> 2) are you sure the latest Sesame is used? The latest release has increased >> performance

Re: [qi4j-dev] OpenRDF performance problem...

On Mon, Jan 4, 2010 at 5:07 PM, Rickard Öberg wrote: > On 2010-01-04 06.38, Niclas Hedhman wrote: >> >> I have just noticed that Sesame/OpenRDF has a general performance >> problem when it comes to removing so called "connections", i.e. when >> we Remove an Entity from the entity store, the index

Re: [qi4j-dev] OpenRDF performance problem...

On 2010-01-04 06.38, Niclas Hedhman wrote: I have just noticed that Sesame/OpenRDF has a general performance problem when it comes to removing so called "connections", i.e. when we Remove an Entity from the entity store, the index needs to be updated. This takes on my machine 360ms per entity if

Re: [qi4j-dev] OpenRDF performance problem...

On Mon, Jan 4, 2010 at 2:19 PM, Niclas Hedhman wrote: > Correction; The number of entities were 100,000 and the removal of > 1000 entities varies from 3 min to 8 minutes... > > Either way, creating those 100,000 takes less than a minute. To remove > them takes 5-10 hours!!! > > I have tried with "

Re: [qi4j-dev] OpenRDF performance problem...

Correction; The number of entities were 100,000 and the removal of 1000 entities varies from 3 min to 8 minutes... Either way, creating those 100,000 takes less than a minute. To remove them takes 5-10 hours!!! I have tried with "spoc" and "ospc" tripleIndexes, but the result seems to be about th

[qi4j-dev] OpenRDF performance problem...

Gang, I have just noticed that Sesame/OpenRDF has a general performance problem when it comes to removing so called "connections", i.e. when we Remove an Entity from the entity store, the index needs to be updated. This takes on my machine 360ms per entity if I do a batch of 1000 removals per conn