Oops - it seems it might not be neo4j that's chewing up the CPU....by making my log4j calls disabled with 'if(_log.isDebugEnabled)' pre-conditions to avoid the expense of constructing the strings that might or might get logged, I was able to improve the times for the 'hot-cache' tests described below by about x10, bringing it within a factor of 1.5-2 of the C++ performance which is within acceptable limits.
I'm still interested to know whether others are using neo4j on Solaris, whether it's possible to use memory mapped i/o for buffers on Solaris, and indeed whether that would further help performance, or does it just reduce the heap demand? On 30 Jun 2011, at 21:02, Paul Bandler wrote: > Further to the problems reported below from running a test application on a > server class Solaris box (M4000) I re-ran the test but with the setting > 'use_memory_mapped_buffers' set to false. This had the +ve effect of > avoiding the warnings and nio exceptions mentioned before so clearly there's > some additional magic required to run under Solaris using memory mapped > buffers. I have enquired as to whether one needs setuid rights to use memory > mapped files on Solaris and am told not btw... > > The performance was woeful however - like walking thru' molasses in (a > Swedish) mid-winter! The test comprises a relatively complex navigation of a > domain model to derive some information. This is a re-implementation of a > function that already exists in a production C++ application that uses an > object database (based on a product called ObjectStore) - so I have > something to compare the results against. I also have the results of running > the same neo4j test on a low-end Windows PC - here are the comparative > timings. Each test contains 2 executions of exactly the same function with > the idea that the first execution will be the worst case scenario as the > caches are stone-cold, but then the second execution, being exactly the same > traversals, should be the best case scenario with the cache's red-hot... In > all cases I'm using an Embedded neo4j database approach. > > Windows (3.2Ghz PC with 1Gb of heap, neo4j database on local disk):- > - execution 1 ~ 41seconds > - execution 2 1.6 seconds > > Solaris (Sun SPARC Enterprise M4000 Server System clock frequency: 1012 MHz > Memory size: 32768 Megabytes with 3.5Gb heap; NEO4J database on nfs mounted > filesystem...) > - execution 1 ~ 27+ _minutes_ > - execution 2 ~ 5.5 seconds > > Re-run with neo4j database on a local file system > - execution 1 ~20 seconds > - execution 2 1.56 seconds > > > Existing C++/ObjectStore implementation on same Solaris box as above:- > - execution 1 ~ 3.2 seconds > - execution 2 ~ 0.1 seconds > > These results are disappointing. It appears that the first execution is > extremely I/O bound with a local enterprise server disk only improving on a > low-end Windows PC by a factor of 2, and nfs reducing performance by a factor > of about x80. Once warmed, the low-end Windows PC is almost on par withe > enterprise Solaris performance. The fastest neo4j execution 2 is 16x slower > than the existing C++/Objectstore implementation. > > Any suggestions on how it might be possible to narrow the gap between the > performance of C++/Objectstore implementation and the neo4j one would be most > welcome, as otherwise this will be the end of the line for this approach.... > I can accept the poor comparison of the first run if once the cache's are hot > the second run comes somewhere reasonably comparable to what might expect > from a comparison between a C++ and Java implementation of a CPU intensive > application operation (say 2-3 times slower in Java...)? > > cheers, > > Paul > On 30 Jun 2011, at 14:06, Paul Bandler wrote: > >> TEST:bandlerp@us2187$ uname -X >> System = SunOS >> Node = us2187 >> Release = 5.10 >> KernelID = Generic_138888-03 >> Machine = sun4u >> BusType = <unknown> >> Serial = <unknown> >> Users = <unknown> >> OEM# = 0 >> Origin# = 1 >> NumCPU = 16 >> >> Sent from my iPhone >> >> On 30 Jun 2011, at 12:53, Michael Hunger <michael.hun...@neotechnology.com> >> wrote: >> >>> Paul, >>> >>> what version of Solaris are you running that on? We don't have Solaris as >>> part of our build-qa workflow (yet). So I would try to see if there is an >>> ec2 instance that I could just use for that. >>> >>> Cheers >>> >>> Michael >>> >>> Am 30.06.2011 um 13:26 schrieb Paul Bandler: >>> >>>> A colleague has speculated that it maybe related to permissions. Im on a >>>> shared solaris box with no setuid access - can anyone elaborate on whether >>>> some specific access rights are required to use memory mapped IO? >>>> >>>> Sent from my iPhone >>>> >>>> On 30 Jun 2011, at 11:20, Paul Bandler <pband...@cseuk.co.uk> wrote: >>>> >>>>> Further to an earlier posting I also notice this warning from neo4j which >>>>> sounds relevant..? Note that the application does continue and eventually >>>>> complete successfully having taken orders of magnitude longer than when >>>>> run on windows. This is running on solaris - any suggestions about what >>>>> could be causing this behaviour most welcome . Are there issues with >>>>> using neo4j on Solaris? >>>>> >>>>> Jun 30, 2011 10:58:59 AM >>>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn >>>>> WARNING: >>>>> [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] >>>>> Unable to memory map >>>>> org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map >>>>> pos=28593 recordSize=33 totalSize=104841 >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.<init>(MappedPersistenceWindow.java:61) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:501) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:128) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:526) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:327) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114) >>>>> at >>>>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97) >>>>> at >>>>> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108) >>>>> at >>>>> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:604) >>>>> at >>>>> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:403) >>>>> at >>>>> org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:98) >>>>> at >>>>> com.nomura.smo.vcs.rdm.neo4j.AbstractBaseNeo4j.getSectorRelations(AbstractBaseNeo4j.java:42) >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 30 Jun 2011, at 07:40, Paul Bandler <pband...@cseuk.co.uk> wrote: >>>>> >>>>>> When running a test neo4j application on Solaris that I have previously >>>>>> run successfully on Windows I'm encountering the following exception: >>>>>> >>>>>> Caused by: java.io.IOException: Resource temporarily unavailable >>>>>> at sun.nio.ch.FileChannelImpl.map0(Native Method) >>>>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:747) >>>>>> at >>>>>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.<init>(MappedPersistenceWindow.java:53) >>>>>> >>>>>> Indeed this in turn is causing a huge knock-on effect but I've not >>>>>> included that stack trace for clarity. The program appears to attempt >>>>>> to continue, albeit at a snails pace. >>>>>> >>>>>> I'm running using the latest 1.4 milestone release, with a maximum heap >>>>>> of 2G, and defaulting all other store parameters. >>>>>> >>>>>> Any suggestions as to what is happening would be most welcome. >>>>>> _______________________________________________ >>>>>> Neo4j mailing list >>>>>> User@lists.neo4j.org >>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> User@lists.neo4j.org >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>> _______________________________________________ >>>> Neo4j mailing list >>>> User@lists.neo4j.org >>>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> _______________________________________________ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
