Re: [Neo4j] Unable to being transaction
Any thoughts on the matter ? It scares me that the database can stop processing requests like that, unless I am doing something very illegal and wrong. Thanks in advance. Regards, Dima Gutzeit. On Thu, Sep 1, 2011 at 9:09 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Here is the story. I had some problem with my index, some inconsistency, which cause lots of those: org.neo4j.graphdb.NotFoundException: phone property not found for NodeImpl#115. at org.neo4j.kernel.impl.core.Primitive.newPropertyNotFoundException(Primitive.java:172) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:167) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:145) ~[neo4j-kernel-1.4.jar:1.4] at com.foo.module.graph.actors.BaseGraphServiceActor.nodeTo(BaseGraphServiceActor.java:89) ~[module-neo4j-1.0-SNAPSHOT.jar:na] at com.foo.module.graph.actors.NodeManagerActor.updates(NodeManagerActor.java:118) ~[module-neo4j-1.0-SNAPSHOT.jar:na] at com.foo.module.graph.actors.NodeManager$$ProxiedByAWDelegation$$ 1315857921056.aw$original$_AW_$updates$_AW_$com_foo_module_graph_actors_NodeManager$$ProxiedByAWDelegation$$1315857921056(Unknown Source) [aspectwerkz-2.2.3.jar:na] at com.foo.module.graph.actors.NodeManager$$ProxiedByAWDelegation$$1315857921056_1_ 2012754604_1561793389___AW_JoinPoint.proceed(Unknown Source) [aspectwerkz-2.2.3.jar:na] at akka.actor.TypedActor$$anonfun$receive$1$$anonfun$apply$1$$anonfun$apply$2.apply(TypedActor.scala:227) [akka-typed-actor-1.2-RC5.jar:na] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [scala-library-2.9.0-1.jar:na] at akka.actor.TypedActor$$anonfun$receive$1$$anonfun$apply$1.apply(TypedActor.scala:216) [akka-typed-actor-1.2-RC5.jar:na] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [scala-library-2.9.0-1.jar:na] at akka.actor.TypedActor$$anonfun$receive$1.apply(TypedActor.scala:215) [akka-typed-actor-1.2-RC5.jar:na] at akka.actor.TypedActor$$anonfun$receive$1.apply(TypedActor.scala:213) [akka-typed-actor-1.2-RC5.jar:na] at akka.actor.Actor$class.apply(Actor.scala:551) [akka-actor-1.2-RC5.jar:na] at akka.actor.TypedActor.apply(TypedActor.scala:114) [akka-typed-actor-1.2-RC5.jar:na] at akka.actor.LocalActorRef.invoke(ActorRef.scala:905) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.MessageInvocation.invoke(MessageHandling.scala:25) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutableMailbox$class.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:216) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:122) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutableMailbox$class.run(ExecutorBasedEventDrivenDispatcher.scala:188) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.run(ExecutorBasedEventDrivenDispatcher.scala:122) [akka-actor-1.2-RC5.jar:na] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_25] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_25] at akka.dispatch.MonitorableThread.run(ThreadPoolBuilder.scala:184) [akka-actor-1.2-RC5.jar:na] it happened only one some of the operations. Over time all operations started to fail with: org.neo4j.graphdb.TransactionFailureException: Unable to begin transaction at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:393) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.EmbeddedGraphDatabase.beginTx(EmbeddedGraphDatabase.java:138) ~[neo4j-kernel-1.4.jar:1.4] at com.foo.module.graph.actors.NodeManagerActor.promoteNode(NodeManagerActor.java:228) ~[module-neo4j-1.0-SNAPSHOT.jar:na] at com.foo.module.graph.actors.NodeManagerActor.createNode(NodeManagerActor.java:55) ~[module-neo4j-1.0-SNAPSHOT.jar:na] at com.foo.module.graph.actors.NodeManager$$ProxiedByAWDelegation$$ 1316234529667.aw$original$_AW_$createNode$_AW_$com_foo_module_graph_actors_NodeManager$$ProxiedByAWDelegation$$1316234529667(Unknown Source) ~[aspectwerkz-2.2.3.jar:na] at com.foo.module.graph.actors.NodeManager$$ProxiedByAWDelegation$$1316234529667_1_1717022187_920046141___AW_JoinPoint.proceed(Unknown Source) ~[aspectwerkz-2.2.3.jar:na] at akka.actor.TypedActor$$anonfun$receive$1$$anonfun$apply$1$$anonfun$apply$2.apply(TypedActor.scala:227) ~[akka-typed-actor-1.2-RC5.jar:na] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) [scala-library-2.9.0-1.jar:na] at akka.actor.TypedActor
[Neo4j] Lucene index recovery
Dear list members, Each time I restart my server based on Neo4J I can see this in the logs: Sep 1, 2011 7:23:17 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [/opt/data/nioneo_logical.log.2]. Recovery started ... Sep 1, 2011 7:23:18 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [/opt/data/index/lucene.log.1]. Recovery started ... Operation which takes time ... lots of time. What is the correct way of preventing that when restarting ? Thanks in advance. Regards, Dima Gutzeit. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Unable to being transaction
(TypedActor.scala:213) [akka-typed-actor-1.2-RC5.jar:na] at akka.actor.Actor$class.apply(Actor.scala:551) [akka-actor-1.2-RC5.jar:na] at akka.actor.TypedActor.apply(TypedActor.scala:114) [akka-typed-actor-1.2-RC5.jar:na] at akka.actor.LocalActorRef.invoke(ActorRef.scala:905) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.MessageInvocation.invoke(MessageHandling.scala:25) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutableMailbox$class.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:216) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:122) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutableMailbox$class.run(ExecutorBasedEventDrivenDispatcher.scala:188) [akka-actor-1.2-RC5.jar:na] at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.run(ExecutorBasedEventDrivenDispatcher.scala:122) [akka-actor-1.2-RC5.jar:na] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_25] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_25] at akka.dispatch.MonitorableThread.run(ThreadPoolBuilder.scala:184) [akka-actor-1.2-RC5.jar:na] Caused by: javax.transaction.SystemException: TM has encountered some problem, please perform neccesary action (tx recovery/restart) at org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:554) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:388) ~[neo4j-kernel-1.4.jar:1.4] ... 23 common frames omitted Maybe its related and maybe not, not sure. Regards , Dima Gutzeit. On Thu, Sep 1, 2011 at 6:20 PM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Dima, got the rest of the stacktrace? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/- Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Sep 1, 2011 at 1:44 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: Any idea what can cause this: org.neo4j.graphdb.TransactionFailureException: Unable to begin transaction at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:393) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.EmbeddedGraphDatabase.beginTx(EmbeddedGraphDatabase.java:138) ~[neo4j-kernel-1.4.jar:1.4] Caused by: javax.transaction.SystemException: TM has encountered some problem, please perform neccesary action (tx recovery/restart) at org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:554) ~[neo4j-kernel-1.4.jar:1.4] at org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:388) ~[neo4j-kernel-1.4.jar:1.4] ... 22 common frames omitted ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Remote database contents inspection
Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Remote database contents inspection
Cool, which dependency is needed to use WrappingNeoServerBootstrapper ? Regards , Dima Gutzeit. On Sun, Jul 31, 2011 at 7:57 PM, Michael Hunger michael.hun...@neotechnology.com wrote: neo4j-shell or you could try to use WrappingNeoServerBootstrapper which runs a neo4j-server with an existing, embedded GraphDatabaseservice. See this thread for context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-download-neo4j-1-3-for-using-jo4Neo-tp3191002p3206212.html Cheers Michael Am 31.07.2011 um 18:52 schrieb Dima Gutzeit: Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Remote database contents inspection
I am sorry, missed it, its all there in the original post: org.neo4j.app:server:1.4 On Sun, Jul 31, 2011 at 8:46 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Cool, which dependency is needed to use WrappingNeoServerBootstrapper ? Regards , Dima Gutzeit. On Sun, Jul 31, 2011 at 7:57 PM, Michael Hunger michael.hun...@neotechnology.com wrote: neo4j-shell or you could try to use WrappingNeoServerBootstrapper which runs a neo4j-server with an existing, embedded GraphDatabaseservice. See this thread for context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-download-neo4j-1-3-for-using-jo4Neo-tp3191002p3206212.html Cheers Michael Am 31.07.2011 um 18:52 schrieb Dima Gutzeit: Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Remote database contents inspection
To be exact its org.neo4j.app:neo4j-server:1.4, and its not posted on central repository (why ?) but should be fetched from: http://m2.neo4j.org/http://m2.neo4j.org/releases/org/neo4j/app/neo4j-server/ On Sun, Jul 31, 2011 at 9:03 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: I am sorry, missed it, its all there in the original post: org.neo4j.app:server:1.4 On Sun, Jul 31, 2011 at 8:46 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: Cool, which dependency is needed to use WrappingNeoServerBootstrapper ? Regards , Dima Gutzeit. On Sun, Jul 31, 2011 at 7:57 PM, Michael Hunger michael.hun...@neotechnology.com wrote: neo4j-shell or you could try to use WrappingNeoServerBootstrapper which runs a neo4j-server with an existing, embedded GraphDatabaseservice. See this thread for context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-download-neo4j-1-3-for-using-jo4Neo-tp3191002p3206212.html Cheers Michael Am 31.07.2011 um 18:52 schrieb Dima Gutzeit: Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Remote database contents inspection
Update and a question. I was able to get the server JARs and run the WrappingNeoServerBootstrapper. The problem is that after getting everything from maven it misses the neo4j-server-static-web.jar. Its not a part of any POM and is not listed in dependencies, nevertheless its required when starting the server (runtime dependency). Can this be fixed somehow, on the POM of neo4j-server ? Thanks in advance. On Sun, Jul 31, 2011 at 9:11 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: To be exact its org.neo4j.app:neo4j-server:1.4, and its not posted on central repository (why ?) but should be fetched from: http://m2.neo4j.org/http://m2.neo4j.org/releases/org/neo4j/app/neo4j-server/ On Sun, Jul 31, 2011 at 9:03 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: I am sorry, missed it, its all there in the original post: org.neo4j.app:server:1.4 On Sun, Jul 31, 2011 at 8:46 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: Cool, which dependency is needed to use WrappingNeoServerBootstrapper ? Regards , Dima Gutzeit. On Sun, Jul 31, 2011 at 7:57 PM, Michael Hunger michael.hun...@neotechnology.com wrote: neo4j-shell or you could try to use WrappingNeoServerBootstrapper which runs a neo4j-server with an existing, embedded GraphDatabaseservice. See this thread for context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-download-neo4j-1-3-for-using-jo4Neo-tp3191002p3206212.html Cheers Michael Am 31.07.2011 um 18:52 schrieb Dima Gutzeit: Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Remote database contents inspection
Runtime dependency is required for classifier 'static-web'. It should be documented somewhere so that others will not find it out the hard way :) On Sun, Jul 31, 2011 at 10:45 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Update and a question. I was able to get the server JARs and run the WrappingNeoServerBootstrapper. The problem is that after getting everything from maven it misses the neo4j-server-static-web.jar. Its not a part of any POM and is not listed in dependencies, nevertheless its required when starting the server (runtime dependency). Can this be fixed somehow, on the POM of neo4j-server ? Thanks in advance. On Sun, Jul 31, 2011 at 9:11 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: To be exact its org.neo4j.app:neo4j-server:1.4, and its not posted on central repository (why ?) but should be fetched from: http://m2.neo4j.org/http://m2.neo4j.org/releases/org/neo4j/app/neo4j-server/ On Sun, Jul 31, 2011 at 9:03 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: I am sorry, missed it, its all there in the original post: org.neo4j.app:server:1.4 On Sun, Jul 31, 2011 at 8:46 PM, Dima Gutzeit dima.gutz...@mailvision.com wrote: Cool, which dependency is needed to use WrappingNeoServerBootstrapper ? Regards , Dima Gutzeit. On Sun, Jul 31, 2011 at 7:57 PM, Michael Hunger michael.hun...@neotechnology.com wrote: neo4j-shell or you could try to use WrappingNeoServerBootstrapper which runs a neo4j-server with an existing, embedded GraphDatabaseservice. See this thread for context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-download-neo4j-1-3-for-using-jo4Neo-tp3191002p3206212.html Cheers Michael Am 31.07.2011 um 18:52 schrieb Dima Gutzeit: Has anyone tried running Neoclipse with X11 Forwarding on a remote linux server? Any pointers ? It does not seem to work for me :( Any other tips for inspecting remote database contents of embedded installation, without the web admin ? Regards , Dima Gutzeit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Announcing: Neo4j 1.4 Kiruna Stol GA
Is it only me or the artifacts of 1.4 from maven are not alright ... (size and etc. it does not compile anymore) ? neo4j-kernel: Version TypeDownload 1.4http://mvnrepository.com/artifact/org.neo4j/neo4j-kernel/1.4 release Binaryhttp://repo1.maven.org/maven2/org/neo4j/neo4j-kernel/1.4/neo4j-kernel-1.4-docs.jar (14 KB) 1.4.M06http://mvnrepository.com/artifact/org.neo4j/neo4j-kernel/1.4.M06 milestone Binaryhttp://repo1.maven.org/maven2/org/neo4j/neo4j-kernel/1.4.M06/neo4j-kernel-1.4.M06.jar (802 KB) Regards , Dima Gutzeit. On Mon, Jul 11, 2011 at 12:02 PM, Jim Webber j...@neotechnology.com wrote: Hello graphistas, It’s been six milestones since we started development of our 1.4 release. Today we present to you Neo4j 1.4 “Kiruna Stol”, our finest release yet, with a host of feature additions and performance and usability improvements. *Cypher Query Language* In addition to Gremlin, you can now use our new declarative query language named Cypher. It has a readable, simple yet expressive syntax that DBAs will feel comfortable with. For example, the usual friend of a friend query can be concisely expressed in Cyhper in just three lines: START user = (mail-index, name, John) MATCH user-[:SENT_MAIL_TO]- () -[:SENT_MAIL_TO]- fof RETURN fof *Auto Indexing* Indexes are essential for long-lived names in a graph. But Neo4j’s indexes have, to-date, made you work hard. The 1.4 GA release allows properties to be automatically indexed by the database through the new Auto Indexing API. This API allows users to define names of properties and have them automatically added, removed and updated in your index consistently with the underlying graph. *Indexing improvements* But Auto Indexing is not the end of improvements in the index front. Our implementation of index provider has been upgraded to Lucene 3.1 which, thanks to the hard work from the Lucene community, brings significant performance boosts across the whole range of operations. This changes comes with an upgrade to the index on-disk format though, so keep that in mind when upgrading and don’t forget to backup important data. *Self Relationships* One of the most requested features by the community has been the ability to create relationships from a node to itself, commonly referred to as self relationships. As of 1.4 GA you can create self relationships using the existing createRelationshipTo() method. Of course when you’re traversing a graph, try not to get stuck in those loops! *Performance Improvements* One of our favorite passtimes is finding new ways to cut down time spent in the database code paths, giving you more time to manage your data. In the kernel you will find changes that have made small transactions significantly faster and also improvements in the cache that take in consideration the direction of relationships, enabling faster in-memory lookups. *REST Improvements* We’ve devoted a lot of time bringing some much needed usability improvements to those folks who use Neo4j server. The index management operations that were exclusive to the embedded API are now exposed over REST also, so now you can create and delete indexes, as well as perform arbitrary queries. The traversers are now capable of being paged, reducing the network overhead and allowing for more flexibility in your code. Finally, the concept of batch commands has been introduced, giving a substantial performance boost for larger, predictable sequences of operations like bulk uploads. *Webadmin Improvements* The Webadmin tool has received a facelift and some useful new features that. A new Index Management tab gives DBAs full control over indexes with a simple point-and-click interface. We’ve added two new command line interfaces to the Consoles tab: one for our Cypher query language, and one for those that prefer working over HTTP with a curl-like tool. All that along with many bug fixes and improvements for better cross platform support are just a browser window away! *New server management scripts* A major pain, especially for OSX users, has been the server management scripts. “Kiruna Stol” comes with our own, home grown scripts for *nix systems and batch scripts for Windows machines that actually work and are far more pleasant than the previous third-party wrapper scripts. *Download!* Now the code’s ready, we’d love for you to download and tryout “Kiruna Stol” and tell us what you think. Feedback from our community is what drives us, and we are eager to hear what you think. For more details on this GA, see the release blog post at http://blog.neo4j.org/2011/07/announcing-neo4j-14-kiruna-stol-ga.html and, as always, get the latest release at http://neo4j.org/download/ Happy hacking! -- Jim Webber ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing
Re: [Neo4j] Neo4J sizing calculations
Johan, Thanks for your answer. The graph represents a social network, where users are friends of other users (followers and following). Each node has a name and is connected to other users (his friends), so the relationship is user-is-a-friend-of-another-user. Each relationship has a nickname, by which name one user knows another. Most of the graph access queries are to find who user follows or who follow a user. Each node holds user name and maybe few other parameters and a relationship holds a nickname and 1-2 other properties. Thanks in advance. Regards, Dima Gutzeit. On Mon, May 2, 2011 at 11:45 AM, Johan Svensson jo...@neotechnology.comwrote: Hi, This will depend on types of queries, access patterns and what the data look like. Could you provide some more information on what the data look like, specifically relationships traversed and properties loaded for a query? Regarding adding another machine to an already active cluster it is easy. Just configure it (assign an id and point it to the cluster) then startup it up. This will trigger replication of data to the new machine and once that is done it will be available. Regards, Johan On Sat, Apr 30, 2011 at 10:02 AM, Dima Gutzeit dima.gutz...@mailvision.com wrote: Dear list members, I am building a Neo4J cluster that should hold around 2 billion of nodes with ~5 billion properties. Data will be mostly accessed for read, about 90/10. Around 200,000 concurrent users will require mostly/read access to the database. To translate it to number of queries is up to 10,000 per second. I need to calculate the sizing of such cluster, number of machines, required RAM, CPU and disk space. Any suggestions ? Another questions is how complicated it to add a new machine to an active cluster, while system is running, its it achievable ? Thanks in advance. Regards, Dima Gutzeit. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Architecturing Neo4J database
On Mon, Apr 18, 2011 at 11:40 AM, Mattias Persson matt...@neotechnology.com wrote: 2011/4/17 Dima Gutzeit dima.gutz...@mailvision.com: Tobias, thank you very much for your quick response. Hi Dima, Estimated number of relationships is ten times number of nodes, which makes it 20 billion. Number of properties for each node is ~10 and for each relationship is ~5. Currently neo4j supports 64 billion properties, your dataset wouldn't fit property-wise... are all the properties for each node necessary? Could you outline some/all of them what they are and what meaning they have? Sometimes you can infer things by looking at a node/relationship instead of having a property explicitly. Ok, I can live with 64 billion properties, thanks. I have another question that troubles me, and its what the correct way to go with, using standalone server or using embedded and implementing some kind of Google ProtoBuf towards other elements in the network. If using standalone server with REST then the transaction logic should be implemented on the REST client side, and not using already available transaction support of Neo4J. (as well as batch operation). What will be the downsides of using embedded vs standalone approach ? You can write server-side plugins which instead exposes groups of operations suitable for your domain/clients - reducing the overhead and giving you control over transactions, on the server-side though. Where can I read about server-side plugins ? You mean that I can run a standalone server with my plugins, without the need to wrap around an embedded server ? What is a common practice for Neo4J users in this case ? On Sun, Apr 17, 2011 at 8:44 PM, Tobias Ivarsson tobias.ivars...@neotechnology.com wrote: What is the required number of relationships? I'll answer your questions inline below. On Sun, Apr 17, 2011 at 7:27 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Oops, small correction. The required number of nodes is 2 billions and not 100 millions as originally stated. On Sun, Apr 17, 2011 at 8:16 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Dear list members, I have a need for storage engine (graph based) for a server I am building. Here are the requirements: 1. Support for High Availability You can tick that box with Neo4j :) 2. Ability to store around 100 millions of nodes, estimated data size of around 5 terabytes. Measuring data in bytes is blunt and difficult. Statistically, all users trying to do that before using Neo4j for at least a while has gotten it wrong in one way or another. Is it ok if I take this figure with a grain of salt? Your statement of 2 bilion nodes is more helpful, add to that the estimated number of relationships (either per node or in total) and the estimated number of properties for each node and relationship and we'll have a better model for estimating both on-disk size and how to best configure your Neo4j installation for optimal cache utilization. I've been reading about different options that Neo4J provides and I need to make a final decision about the layout, few questions: - I expect mostly-read-access to the database and I've been reading about the HA approach, by replicating the data across several nodes. Is there an option to use a centralized storage server (NAS/SAN) and working with several nodes (for scaling) but without data replication ? No. Since Neo4j requires exclusive file locks for the storage files to guarantee write consistency. It also doesn't work well even if you have a single write-instance and let all other instances be read-only instances used for only scaling reads. This is because Neo4j relies heavily on caching to get good read performance. If the underlying storage data is changed by the write-master the read slaves will not know about it and thus suffer from stale caches which can lead to an inconsistent view of the graph. The Neo4j HA solution provides the appropriate replication and cache invalidation to make this work consistently across multiple instances. - The documentation describes HA installation with embedded neo4j instance, can the same be configured for a standalone server ? Yes. - If using an embedded version of the database, can still the webadmin be used ? Not yet. This is being considered for future versions. - What is supported request rate / throughput of a single Neo4j server, is there benchmarking data I can consult with including the hardware config of a testbed ? The figures for an embedded Neo4j setup is between 1000 to 2000 relationships traversed per millisecond for fully cached data. With approximately 10% of that for data only in the low level (file
[Neo4j] Architecturing Neo4J database
Dear list members, I have a need for storage engine (graph based) for a server I am building. Here are the requirements: 1. Support for High Availability 2. Ability to store around 100 millions of nodes, estimated data size of around 5 terabytes. I've been reading about different options that Neo4J provides and I need to make a final decision about the layout, few questions: - I expect mostly-read-access to the database and I've been reading about the HA approach, by replicating the data across several nodes. Is there an option to use a centralized storage server (NAS/SAN) and working with several nodes (for scaling) but without data replication ? - The documentation describes HA installation with embedded neo4j instance, can the same be configured for a standalone server ? - If using an embedded version of the database, can still the webadmin be used ? - What is supported request rate / throughput of a single Neo4j server, is there benchmarking data I can consult with including the hardware config of a testbed ? Thanks in advance for your response. Regards, Dima Gutzeit. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Architecturing Neo4J database
Oops, small correction. The required number of nodes is 2 billions and not 100 millions as originally stated. On Sun, Apr 17, 2011 at 8:16 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Dear list members, I have a need for storage engine (graph based) for a server I am building. Here are the requirements: 1. Support for High Availability 2. Ability to store around 100 millions of nodes, estimated data size of around 5 terabytes. I've been reading about different options that Neo4J provides and I need to make a final decision about the layout, few questions: - I expect mostly-read-access to the database and I've been reading about the HA approach, by replicating the data across several nodes. Is there an option to use a centralized storage server (NAS/SAN) and working with several nodes (for scaling) but without data replication ? - The documentation describes HA installation with embedded neo4j instance, can the same be configured for a standalone server ? - If using an embedded version of the database, can still the webadmin be used ? - What is supported request rate / throughput of a single Neo4j server, is there benchmarking data I can consult with including the hardware config of a testbed ? Thanks in advance for your response. Regards, Dima Gutzeit. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Architecturing Neo4J database
Tobias, thank you very much for your quick response. Estimated number of relationships is ten times number of nodes, which makes it 20 billion. Number of properties for each node is ~10 and for each relationship is ~5. I have another question that troubles me, and its what the correct way to go with, using standalone server or using embedded and implementing some kind of Google ProtoBuf towards other elements in the network. If using standalone server with REST then the transaction logic should be implemented on the REST client side, and not using already available transaction support of Neo4J. (as well as batch operation). What will be the downsides of using embedded vs standalone approach ? What is a common practice for Neo4J users in this case ? On Sun, Apr 17, 2011 at 8:44 PM, Tobias Ivarsson tobias.ivars...@neotechnology.com wrote: What is the required number of relationships? I'll answer your questions inline below. On Sun, Apr 17, 2011 at 7:27 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Oops, small correction. The required number of nodes is 2 billions and not 100 millions as originally stated. On Sun, Apr 17, 2011 at 8:16 PM, Dima Gutzeit dima.gutz...@mailvision.comwrote: Dear list members, I have a need for storage engine (graph based) for a server I am building. Here are the requirements: 1. Support for High Availability You can tick that box with Neo4j :) 2. Ability to store around 100 millions of nodes, estimated data size of around 5 terabytes. Measuring data in bytes is blunt and difficult. Statistically, all users trying to do that before using Neo4j for at least a while has gotten it wrong in one way or another. Is it ok if I take this figure with a grain of salt? Your statement of 2 bilion nodes is more helpful, add to that the estimated number of relationships (either per node or in total) and the estimated number of properties for each node and relationship and we'll have a better model for estimating both on-disk size and how to best configure your Neo4j installation for optimal cache utilization. I've been reading about different options that Neo4J provides and I need to make a final decision about the layout, few questions: - I expect mostly-read-access to the database and I've been reading about the HA approach, by replicating the data across several nodes. Is there an option to use a centralized storage server (NAS/SAN) and working with several nodes (for scaling) but without data replication ? No. Since Neo4j requires exclusive file locks for the storage files to guarantee write consistency. It also doesn't work well even if you have a single write-instance and let all other instances be read-only instances used for only scaling reads. This is because Neo4j relies heavily on caching to get good read performance. If the underlying storage data is changed by the write-master the read slaves will not know about it and thus suffer from stale caches which can lead to an inconsistent view of the graph. The Neo4j HA solution provides the appropriate replication and cache invalidation to make this work consistently across multiple instances. - The documentation describes HA installation with embedded neo4j instance, can the same be configured for a standalone server ? Yes. - If using an embedded version of the database, can still the webadmin be used ? Not yet. This is being considered for future versions. - What is supported request rate / throughput of a single Neo4j server, is there benchmarking data I can consult with including the hardware config of a testbed ? The figures for an embedded Neo4j setup is between 1000 to 2000 relationships traversed per millisecond for fully cached data. With approximately 10% of that for data only in the low level (file system) cache. The server setup used the same graph database internally, but adds some per-request network overhead to that. I don't have any benchmark figures. Could someone fill in? Cheers, -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
