Re: [Neo] Nodes not found after restarting Neo4j via indexing service
Make sure that you're committing your transactions successfully, else the data will not be there when you ask for it later. Check out http://wiki.neo4j.org/content/Transactions#Controlling_success for more information. 2010/4/20 Lalit Kapoor lalitkap...@gmail.com Doh!, I jumped the gun on that one, thanks for pointing that out, I must be doing something else wrong in my app. -Lalit On Mon, Apr 19, 2010 at 7:00 PM, Tobias Ivarsson tobias.ivars...@neotechnology.com wrote: On lines 52-64 you are removing all the data that was created, that is why you are unable to find it again. On Tue, Apr 20, 2010 at 12:46 AM, Lalit Kapoor lalitkap...@gmail.com wrote: If I insert data into neo4j and index it I am able to use the indexing service to find specific nodes. However, if I shutdown my neo4j app and start it up again I am not able to use the indexing service to find nodes. As an example I am using the EmbeddedNeo4jWithIndexing java code provided. I run this code and it is able to find the node associated with the user who has an id of 45. Next I comment out the for loop that inserts data. Therefore I no longer insert data and skip right to doing the index lookup. Unfortunately this is resulting in 0 nodes being found. I have provided copies of the original code as well as the insertion commented out version: original: http://pastebin.com/iC5TgzxD insertion commented out: http://pastebin.com/pL98gB0c I would appreciate it if someone could take a look. Thanks. Sincerely, Lalit ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] force preloading into memory
On Tue, Apr 20, 2010 at 10:42 AM, Erik Ask erik...@maths.lth.se wrote: Tobias Ivarsson wrote: The speedup you are seeing is because of caching. Items that are used are loaded into an in-memory structure, that does not need to go through any filesystem API, memory-mapped or not. The best way to load things into cache is to run the query once to touch everything that needs to be loaded. Pre-adapting the memory-maps as you suggest would give some speedup to the actual process of the first query, but that time would be spent in startup instead, meaning that the time from cold start to completed first query would be exactly the same. Cheers, Tobias On Mon, Apr 19, 2010 at 6:31 PM, Erik Ask ask.e...@gmail.com wrote: Hello I'm getting really slow performance when working against the HD. A given set of queries can take up to 10 minutes when performed the first time. Repeating the same set of queries a second time is executed in seconds (2-5). As far as I can tell from watching in jconsole, the heap behaves in almost the exact same maner (slowly rising slope) for both transactions (each set of queries has it own transaction) so it seems the speedup is due to memory mapping. I've tinkered with the settings, but is there a way of explicitly forcing the IO mapper to preload all or part of the node store and relationship store? Am I right to assume that initially nothing is IO mapped and these buffers builds up during runtime as requests are made? Is there any way of tuning access to the HD? greetz Then i don't understand the purpose of loading files in to memory. I thought it was used to make a copy of as much of a file as possible into memory, then do all subsequent lookups there, and if needed replace parts if nonloaded parts of the file are more frequently requested than loaded. This would result in one hd-read per node/rel (assuming it fit into memory and no replacing was needed), as opposed to searching for entries in file that would require lots of reads and comparisons. The amount of data that needs to be loaded into memory just doesn't seem to warrant that much time being spent. I could easily copy files several times the size of my complete DB in less time than it takes to run my query sets. Hi, Tobias is right about the caching part but there are issues with memory mapped I/O in play here too. If you turn off memory mapped I/O and use normal buffers (use_memory_mapped_buffers=false) you will probably see a speedup in initial query time. This is because using memory mapped I/O will result in lots of seeks since most OS/configurations implement them in such a (maybe not ideal) way. Non memory mapped buffers will do sequential reads to fill the entire buffer and (depending on how much of the graph the first search touches) it will likely be faster. To explain further, if you request to map a region of a file into memory it will do so and return almost instantly. The contents of the file is however not loaded into memory, instead it will do lazy loads when you start to read bytes from the buffer resulting in more random I/O and seeks. This in turn results in slow searches and long warmup time on mechanical disks. (Note, behavior I described here may vary depending on OS, JVM implementation and so on.) You are right about the purpose of loading regions of files into memory so we don't have to do lookups on disk (and dynamically change those regions depending on access patterns). The problem is that initial access patterns when nothing has been loaded yet will look random. Then to further kill performance memory mapped regions will not do a sequential read of the data (this is very bad in your scenario but is better when the server is warm). A work around for this is to pre-fill the OS file-system caches before you start searching. Write a script that sequentially reads the node, relationship (and property store file if your searches access properties). That will cause the memory mapped regions to map against the file-system cache and then the contents of the file will already be in memory. Regards, Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Exception when adding and deleting in a single transaction
Hi, There should be no problem to do multiple modifying operations in the same transaction. Since you are talking about statements I take it you are using the rdf component? What happens if you move the delete statement before the call to Tx.success()? Regards, Johan On Wed, Apr 14, 2010 at 5:59 PM, Subbacharya, Madhu madhu.subbacha...@corp.aol.com wrote: Hi, When I add a statement and then delete a different statement within a single transaction begin/end block, Tomcat is complaining about a dangling thread. Here is what I am doing: 1. Tx begin 2. Add a statement 3. If add succeeded, Tx.success() 4. Delete a statement 5. If delete failed, Tx.failure() 6. In the finally block of the try statement, I call Tx.finish(), shutdown the store, etc. I have a workaround by creating 2 separate transactions (one for the add and the other for the delete) and managing them independently with associated status codes, which works fine. However, what I would really like is to be able to do multiple DB modifying operations within a single transaction block. Assuming this is doable, would be nice to have a Tx.state() method that returns whether Tx.success() or Tx.failure() was last called. Thanks madhu Tomcat log: Apr 13, 2010 9:51:27 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [null] (value [null]) and a value of type [org.neo4j.index.Isolation] (value [SAME_TX]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Apr 13, 2010 9:51:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadLocals WARNING: Failed to clear ThreadLocal references java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja va:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso rImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.loader.WebappClassLoader.clearThreadLocalMap(Webapp ClassLoader.java:2102) at org.apache.catalina.loader.WebappClassLoader.clearReferencesThreadLocal s(WebappClassLoader.java:2027) at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClas sLoader.java:1710) at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.jav a:1622) at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:710) at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4649 ) at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:9 24) at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1 174) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1342) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:3 03) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleS upport.java:119) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase. java:1337) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pro cessChildren(ContainerBase.java:1601) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pro cessChildren(ContainerBase.java:1610) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run (ContainerBase.java:1590) at java.lang.Thread.run(Thread.java:637) Caused by: java.lang.NullPointerException at java.lang.ThreadLocal.access$400(ThreadLocal.java:53) at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:436) ... 20 more ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
