[Neo4j] Neil Ellis wants to stay in touch on LinkedIn
LinkedIn Neil Ellis requested to add you as a connection on LinkedIn: -- Craig, I'd like to add you to my professional network on LinkedIn. - Neil Ellis Accept invitation from Neil Ellis http://www.linkedin.com/e/5gyj7a-gsb1bodu-4e/h9LPQ_TdyUOQHKzIpND15vYO56OQOUsn/blk/I199375840_9/pmpxnSRJrSdvj4R5fnhv9ClRsDgZp6lQs6lzoQ5AomZIpn8_elYMd3wRdPcVej59bQhHmDtQizdNbPsQdz8OdPkOcjgLrCBxbOYWrSlI/EML_comm_afe/?hs=falsetok=1phqMRJD3wy4U1 View invitation from Neil Ellis http://www.linkedin.com/e/5gyj7a-gsb1bodu-4e/h9LPQ_TdyUOQHKzIpND15vYO56OQOUsn/blk/I199375840_9/0VnP0Qe3kTcPAVckALqnpPbOYWrSlI/svi/?hs=falsetok=27Z93xxdLwy4U1 -- (c) 2011, LinkedIn Corporation ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neil Ellis wants to stay in touch on LinkedIn
Apologies, a classic address book import mistake :-) Sorry folk! Neil Ellis Sign up for the beta of Hashbo, because life is for sharing: http://hashbo.com Read more about what we're doing at: http://hashbo.posterous.com On 8 Sep 2011, at 02:04, Neil Ellis via LinkedIn wrote: LinkedIn Neil Ellis requested to add you as a connection on LinkedIn: -- Craig, I'd like to add you to my professional network on LinkedIn. - Neil Ellis Accept invitation from Neil Ellis http://www.linkedin.com/e/5gyj7a-gsb1bodu-4e/h9LPQ_TdyUOQHKzIpND15vYO56OQOUsn/blk/I199375840_9/pmpxnSRJrSdvj4R5fnhv9ClRsDgZp6lQs6lzoQ5AomZIpn8_elYMd3wRdPcVej59bQhHmDtQizdNbPsQdz8OdPkOcjgLrCBxbOYWrSlI/EML_comm_afe/?hs=falsetok=1phqMRJD3wy4U1 View invitation from Neil Ellis http://www.linkedin.com/e/5gyj7a-gsb1bodu-4e/h9LPQ_TdyUOQHKzIpND15vYO56OQOUsn/blk/I199375840_9/0VnP0Qe3kTcPAVckALqnpPbOYWrSlI/svi/?hs=falsetok=27Z93xxdLwy4U1 -- (c) 2011, LinkedIn Corporation ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j-utils
I tend to write xxxTX() and xxxNoTX methods so the user can see instantly if the methods contain a TX. All the best Neil Ellis (Sent from my iPhone) On 7 Jun 2010, at 11:14, Wouter De Borger w.debor...@gmail.com wrote: Hi, I'm using it! I agree that the transactions shouldn't be there. But it could be excused by the fact that neo4j ignores nested transactions (I think?) So it doesn't really break any existing transactions and yields the expected behavior when not in a transaction. Furthermore, if you remove them, a lot of software might break. so I would advise changing it at the next major version? Wouter On Wed, Jun 2, 2010 at 5:08 PM, Mattias Persson matt...@neotechnology.com wrote: Is there someone out there using neo4j-utils component, http://components.neo4j.org/neo4j-utils/ ? I'm the one responsible for creating the (somewhat messy) utilities in there. Something just hit me when looking at it: most of the public methods in the code (although not all) which does some write operation to the graph wraps the code in its own transaction. I find that to be a little off, since it's good to be explicit about the scopes of your transactions. So I was planning to remove all such transaction wrappings and also remove a lot of GraphDatabaseService references from constructors, since you now can reach the graph database via http://components.neo4j.org/neo4j-kernel/apidocs/org/neo4j/graphdb/PropertyContainer.html#getGraphDatabase () , making that extra reference unnecessary. Does anyone have an opinion about all this? -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Dispell the myth? FlockDB vs. Neo4j
Bless them, that was nice of them; free advertising too. On 14 Apr 2010, at 15:58, Laurent Laborde wrote: The FlockDB README changed :) old : -This is a distributed graph database. we use it to store social graphs (who follows whom, who blocks whom) and secondary indices at twitter. new : This is a distributed graph database. we use it to store social graphs (who follows whom, who blocks whom) and secondary indices at twitter. it is much simpler than other graph databases such as neo4j. it scales horizontally and is designed for on-line, low-latency, high throughput environments such as web-sites. yay! for neo4j \o/ -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user All the best Neil Ellis Consultant Mangala Solutions http://www.mangala.co.uk Tel/Fax: +44 (0) 20 7183 1318 | Skype: neilellis | AIM: neilel...@mac.com | Blog: http://web.mac.com/neilellis/ UK Company Reg No: 4557538-- Vat Reg No: 815793111 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] newbie questions - Neo4j client? PHP module?
Sounds like REST is your best friend here :) On 13 Apr 2010, at 13:18, paul.b...@zodiacaerospace.com wrote: Hello all, I am excited about what I have heard/learned about Neo4j. I am working on building a content management system / intranet at work to replace a lot of seperate systems that we currently use. I believe that Neo4j will be the database that fits the data the best. I was raised using procedural languages like pascal and C and have not embraced OOP. So I am struggling to get started with Neo4j since you mostly interact with it through Java code. Lately I have been writing web apps using LAMP. Specifically, MySQL and PHP. as I try to figure out Neo4j, I remember how I started using MySQL from MySQL's text based client. Is there an interface like that with Neo4j? Maybe something where you can write and execute the java code a line at a time, or transaction at a time? I really like PHP. I am currently embracing its socket and stream functions as I write code to send announcement emails from my CMS, and interact with our ERP system through a SSH2 interface. MySQL has a nice module with the MySQLi functions. Will there ever be such a module for Neo4j? How should I go about working with Neo4j from PHP? Sincerely Paul Bock ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user Kind regards Neil Ellis See it, Share it. Interweb: http://cazcade.com Twitter: http://twitter.com/cazcade_com Tel/Fax: +44 (0) 20 7183 1318 | Skype: neilellis | AIM: neilel...@mac.com | Twitter: neilellis ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Dispell the myth? FlockDB vs. Neo4j
Hmm.. Yep, that sounds about right. I wonder how long it would have taken to write a sharding abstraction for their use case compared to writing FlockDB, but hey I'm not trying to second guess the problems they have at hand. I myself am pretty ruthless about what technologies I use, it just doesn't take a genius to shard many graphs by hand and how long would it take to use a hash based sharding for example on the client and then do something like: node.setProperty(node.ref, node://shard-name/nodeId) on a node acting as a proxy when creating a relationship to a node outside of the current shard. Sure, the logic is in the application, but you're going to write an abstraction around Neo to translate into the appropriate domain anyway (as we always do with persistence)? Sorry going tangental . On 13 Apr 2010, at 14:45, Alastair James wrote: (not sure why they didn't use a proven high speed graph DB myself - but then I am biased!). I am guessing its because they only need 1st order relations (there are little friend of a friend or higher operations on twitter) and were very worried about scalability and sharding (as far as I am aware there is no way for a Neo Db to be bigger than one machine at the moment). Not to mention that they probably have considerable mysql infrastructure and expertise I think its pretty obvious that FlockDB will be very slow for 'deep' traversals (network protocol speed alone will ensure this as FlockDb will need to talk to mysql on each node visited), but for order 1 traversals (e.g. what boils down to a join across a mapping table) mysql is probably a lot faster (no transactional overhead for read ops). Al ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Dispell the myth? FlockDB vs. Neo4j
Long time no post from me ;-) But it seemed worth commenting on. I think it would be good to quickly leap to illustrate the differences and twitter this ASAP before the morning twitter storm that will occur about this. I find it hard to believe they're solution is going to perform anything like Neo4J (from which we tend to get pretty impressive results); so now will be the time to blog and Twitter why Neo4J kicks ass :-) Personally I'm extremely skeptical of this and _very_ comfortable with Neo (like pipe and slippers comfortable). Also for other Neo4J users/fans, why not twitter something like this, or RT me: http://twitter.com/neilellis/status/12070188997 ATB Neil On 12 Apr 2010, at 22:16, Alastair James wrote: Yeah, I choked on my coffee when I read FlockDB described as a 'graph db'. Its a key / value store with the ability to create relationships between keys. I hope projects like this misusing the 'graph db' term will not tarnish the reputation of 'proper' graph dbs! Al On 12 April 2010 22:06, Marko Rodriguez okramma...@gmail.com wrote: Hi guys, I saw this on Twitter: dviner @wbelk haven't looked at neo4j. seems hard to look at a non-distributed graph db when a distributed graph db is also available. Seems like people think graphdb means graphdb FlockDB is not traversal ready (at least how I see it from their writing and their implementation)... Neo4j will now have to explain itself ... Bummers, Marko. http://markorodriguez.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Dr Alastair James CTO James Publishing Ltd. http://www.linkedin.com/pub/3/914/163 www.worldreviewer.com WINNER Travolution Awards Best Travel Information Website 2009 WINNER IRHAS Awards, Los Angeles, Best Travel Website 2008 WINNER Travolution Awards Best New Online Travel Company 2008 WINNER Travel Weekly Magellan Award 2008 WINNER Yahoo! Finds of the Year 2007 Noli nothis permittere te terere! ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user All the best Neil Ellis Consultant Mangala Solutions http://www.mangala.co.uk Tel/Fax: +44 (0) 20 7183 1318 | Skype: neilellis | AIM: neilel...@mac.com | Blog: http://web.mac.com/neilellis/ UK Company Reg No: 4557538-- Vat Reg No: 815793111 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] getReferenceNode
Hey Todd Take a look on the wiki for the IndexService stuff . that's what you're looking for. All the best Neil On 5 Sep 2009, at 02:09, Todd Stavish wrote: Cool. Thanks guys, I have it know. Have you given thought to just having a global persistent hash map to find root objects? That way, you could have several root objects that were easily retrievable, and then you could start your navigation / traversal from there. Thanks again, Todd On Fri, Sep 4, 2009 at 6:25 PM, Anders Nawroth and...@neotechnology.com wrote: Hi! Just one small addition to this: don't assume that the reference node will always have an id of 0. To check if a node is the reference node, use the equals() method to compare the nodes! /anders Dan Heaver: Todd, the reference node is node 0 It's there even if you have an empty graph. You don't set it it's just there from the start... Dan On 4 Sep 2009, at 23:00, Todd Stavish toddstav...@gmail.com wrote: How do you set a reference node? Or create a root node? Is the examples.NeoclipseExample package published somewhere? Thanks, Todd ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] getReferenceNode
http://components.neo4j.org/index-util/ On 5 Sep 2009, at 02:09, Todd Stavish wrote: Cool. Thanks guys, I have it know. Have you given thought to just having a global persistent hash map to find root objects? That way, you could have several root objects that were easily retrievable, and then you could start your navigation / traversal from there. Thanks again, Todd On Fri, Sep 4, 2009 at 6:25 PM, Anders Nawroth and...@neotechnology.com wrote: Hi! Just one small addition to this: don't assume that the reference node will always have an id of 0. To check if a node is the reference node, use the equals() method to compare the nodes! /anders Dan Heaver: Todd, the reference node is node 0 It's there even if you have an empty graph. You don't set it it's just there from the start... Dan On 4 Sep 2009, at 23:00, Todd Stavish toddstav...@gmail.com wrote: How do you set a reference node? Or create a root node? Is the examples.NeoclipseExample package published somewhere? Thanks, Todd ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] New features in trunk
Fantastic Johan Once our September alpha is done we'll run against the b10-SNAPSHOT, those features really make a difference to us. All the best Neil On 2 Sep 2009, at 14:16, Johan Svensson wrote: Hello, Now that the 1.0-b9 release has been out for a while we have had some time to work on some important new features that we want to be part of the final 1.0 release. The 1.0-b10-SNAPSHOT (trunk) now includes: o New algorithm to rebuild id generators during recovery after a crash. Cuts recovery time on large stores to a fraction of the time it used to take. o Read-only mode. You can now startup multiple read-only NeoServices to the same store (using EmbbededReadOnlyNeo) and have a single writer to that store. o Improved depth first traversal speed on nodes with many relationships by loading relationships on demand. This will also reduce the stress on GC when traversing over a node with many relationships since much fewer objects will be created. If you have some suitable project please help us test and try out these new features. Regards, -Johan -- Johan Svensson [jo...@neotechnology.com] Chief Technology Officer, Neo Technology www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Load hugh graph
Hi Vanessa You might want to look at the bulk load feature, it's described on the wiki. All the best Neil Ellis (Sent from my iPhone) On 21 Aug 2009, at 10:26, Vanessa Junquero Trabado vanessitajunqu...@hotmail.com wrote: I'm trying to load a hugh database with 17 million nodes and 180 million edges. I tried it with the new release of Neo4j, but it is very very slow. I put in the configuration java file the following parameters: neostore.nodestore.db.mapped_memory=153M 2.neostore.relationshipstore.db.mapped_memory=6G 3.neostore.propertystore.db.mapped_memory=50M 4.neostore.propertystore.db.strings.mapped_memory=300M 5.neostore.propertystore.db.arrays.mapped_memory=0M Any solution? Thank you. _ Toda la información meteorológica. Consulta en MSN el tiempo que va a hacer en cualquier lugar de España o del Mundo. http://eltiempo.es.msn.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] InMemoryNeoService
So that kind of ties in with the Terracotta questions. An InMemory service could be distributed and persisted by Terracotta. Though I'm not sure I'd actually suggest anyone doing that as Terracotta's persistence is more fragile than the Neo4J storage. ATB Neil On 18 Aug 2009, at 15:16, Tobias Ivarsson wrote: Hi Gary, I'm forwarding this to the list, since it might be of general interest. I have made some experiments towards an in-memory version of NeoService implemented as a transport layer for RemoteNeo. But I didn't have time to finalize it. If anyone wants to give it a shot, this approach is probably the easiest. We have some vague plans of doing an in-memory implementation of NeoService as part of the work on NeoHA, but that would not be earlier than late Q4 this year. Cheers, Tobias On Mon, Aug 17, 2009 at 8:33 PM, Moh, Gary gary@merrillcorp.com wrote: Hi Tobias, I came upon your message regarding an in-memory version of NeoService. Just wondering if there is any update on this feature. Thanks, gary -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] neo4j Beginner Question
And then use the index-utils to find the node (it's a good simple and efficient integration with Lucene). On 14 Aug 2009, at 16:10, Dan Heaver wrote: Onur, add a property to each node to hold the surragate key from your RDBMS... Dan On 14 Aug 2009, at 15:30, Onur AKTAS onur.ak...@live.com wrote: Thank you for your response. I'm trying to understand the main concept, please correct me if I am wrong. For example, there are many users of an application and each represented as Node. And each User has a UserId in RDBMS. I have checked Node class and saw that there is a method called as getId() but I could not see setId() for mapping the User in the RDBMS to neo4j. Sample data: (UserId / Friends Ids) 1 - [2, 4] 2 - [1, 3] 3 - [10] 4 - [1, 3, 5] 5 - [3] And I need to query What are the all shortest paths from 1 to 10 for max length 4 And results should be, 1 - 2 - 3 - 10 1 - 4 - 3 - 10 * 1 - 4 - 5 - 3 - 10 (It will not be shown because it exceeds the length 4). You mean that it can be handled with neo4j, but how can I map each User in RDBMS, to Nodes by their original Ids? Thanks. From: neubauer.pe...@gmail.com Date: Wed, 5 Aug 2009 11:16:15 +0200 To: user@lists.neo4j.org Subject: Re: [Neo] neo4j Beginner Question Hi Onur 2009/8/4 Onur AKTAS onur.ak...@live.com: Calculating,friends of friends, 3rd degree contacts or 4th degree etc. is not a good idea with RDBMS. In short words, neo4j fits perfectly for these kinds of jobs? yes, that is the kind of problems Neo4j and Graph Databases are trying to solve in general. Semi-structured data, havy use of relational information between information entities and complex analysis of high depth fast are some of the common use cases for using a graph database. Do we have a chance to scale neo4j by adding nodes, does it support distributed processing? Also, can we delete any node in the graph when we need to delete user or we must create the entire graph again when we need any updates? Do graph objects have id's that helps us to find the node quickly? Neo4j atm does not support distributed processing or partitioning the graph. There is high-availability and online-backup being worked on which feels more acute since Neo4j on a single instance scales over 1 Billion primitives, which should be enough for most immediate scenarios. But there are of course ways to achieve a custom partitioning using targeted techniques - but that depends on your use case ... HTH /peter ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user _ Anılarınızı istediğiniz herkesle çevrimiçi paylaşın. http://www.microsoft.com/turkiye/windows/windowslive/products/photos-share.aspx?tab=1 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] neo4j Beginner Question
Ain't that cool :-) I speak from experience it works and works well. Lucene is an efficient index, the integration is seemless (from my experience). On 14 Aug 2009, at 17:13, Mattias Persson wrote: // One instance per NeoService is enough IndexService indexService = new LuceneIndexService( neoService ); and that will be your index, fully transactional and participates in your neo transactions. 2009/8/14 Onur AKTAS onur.ak...@live.com: So I must build it on the top of Lucene? Hmm, shouldn't it be a native property of neo4j? Do we have a chance to get Node's id after we insert it? So I can use an additional key - value based database for keeping the relation between Node and User like below: (rdbms_user_id, neo4j_node_id) Is it a good idea? From: neil.el...@mangala.co.uk To: user@lists.neo4j.org Date: Fri, 14 Aug 2009 16:21:38 +0100 Subject: Re: [Neo] neo4j Beginner Question And then use the index-utils to find the node (it's a good simple and efficient integration with Lucene). On 14 Aug 2009, at 16:10, Dan Heaver wrote: Onur, add a property to each node to hold the surragate key from your RDBMS... Dan On 14 Aug 2009, at 15:30, Onur AKTAS onur.ak...@live.com wrote: Thank you for your response. I'm trying to understand the main concept, please correct me if I am wrong. For example, there are many users of an application and each represented as Node. And each User has a UserId in RDBMS. I have checked Node class and saw that there is a method called as getId() but I could not see setId() for mapping the User in the RDBMS to neo4j. Sample data: (UserId / Friends Ids) 1 - [2, 4] 2 - [1, 3] 3 - [10] 4 - [1, 3, 5] 5 - [3] And I need to query What are the all shortest paths from 1 to 10 for max length 4 And results should be, 1 - 2 - 3 - 10 1 - 4 - 3 - 10 * 1 - 4 - 5 - 3 - 10 (It will not be shown because it exceeds the length 4). You mean that it can be handled with neo4j, but how can I map each User in RDBMS, to Nodes by their original Ids? Thanks. From: neubauer.pe...@gmail.com Date: Wed, 5 Aug 2009 11:16:15 +0200 To: user@lists.neo4j.org Subject: Re: [Neo] neo4j Beginner Question Hi Onur 2009/8/4 Onur AKTAS onur.ak...@live.com: Calculating,friends of friends, 3rd degree contacts or 4th degree etc. is not a good idea with RDBMS. In short words, neo4j fits perfectly for these kinds of jobs? yes, that is the kind of problems Neo4j and Graph Databases are trying to solve in general. Semi-structured data, havy use of relational information between information entities and complex analysis of high depth fast are some of the common use cases for using a graph database. Do we have a chance to scale neo4j by adding nodes, does it support distributed processing? Also, can we delete any node in the graph when we need to delete user or we must create the entire graph again when we need any updates? Do graph objects have id's that helps us to find the node quickly? Neo4j atm does not support distributed processing or partitioning the graph. There is high-availability and online-backup being worked on which feels more acute since Neo4j on a single instance scales over 1 Billion primitives, which should be enough for most immediate scenarios. But there are of course ways to achieve a custom partitioning using targeted techniques - but that depends on your use case ... HTH /peter ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user _ Anılarınızı istediğiniz herkesle çevrimiçi paylaşın. http://www.microsoft.com/turkiye/windows/windowslive/products/photos-share.aspx?tab=1 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user _ Windows Live tüm arkadaşlarınızla tek bir yerden iletişim kurmanıza yardımcı olur. http://www.microsoft.com/turkiye/windows/windowslive/products/social-network-connector.aspx ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Interrupts
Hi Johan I meant the application is being shutdown (not neo.shutdown()) :-) - though I understand the confusion. So as part of our shutdown process we will tell all active threads to stop by interrupting them. When they've all stopped I'll shutdown Neo. So the thread this code runs in is receiving an interrupt, however Neo is converting the exception into a generic RuntimeException. Which makes it very hard to catch, with the code being like this it makes it very hard for me to request a thread interruption and then deal with the consequences (i.e. a RuntimeException does not tell me that the thread has been interrupted, so I don't know whether to report an error and continue - or to end the loop it is in). Thanks as always Neil On 30 Jul 2009, at 19:45, Johan Svensson wrote: Hi Neil, Yes, there are a lot of error messages that needs to be improved. On the other hand the documentation should state that calling any Neo4j API operation after (or concurrently with) NeoService.shutdown() is not valid. Adding a flag to check if running is not an option because of performance issues. Could be that there are certain places we could throw a better exception with a better message to help the user narrow down what is wrong. We are planing to do a refactoring of the low level exceptions before 1.0 final and we will keep this in mind then. Regards, Johan On Thu, Jul 23, 2009 at 2:14 AM, Neil Ellisneil.el...@mangala.co.uk wrote: Hi guys, if a thread gets interrupted (i.e. during a shutdown), I end up with a RuntimeException, I think it would be better to bubble something meaningful. What do you think? java.lang.RuntimeException: org.neo4j.impl.nioneo.store.StoreFailureException: Unable to load position[24058922] @[601473050] at org.neo4j.impl.core.NodeManager.loadProperties(NodeManager.java:622) at org.neo4j.impl.core.NodeImpl.loadProperties(NodeImpl.java: 76) at org .neo4j.impl.core.NeoPrimitive.ensureFullProperties(NeoPrimitive.java: 573) at org.neo4j.impl.core.NeoPrimitive.hasProperty(NeoPrimitive.java:282) at org.neo4j.impl.core.NodeProxy.hasProperty(NodeProxy.java: 133) at com .peepwl .sociagraph .figs .impl .index .FigsIndexNeoService.iterateIdentities(FigsIndexNeoService.java: 438) at com .peepwl .sociagraph .figs .impl .index.FigsIndexQueryImpl.iterateIdentities(FigsIndexQueryImpl.java: 94) at com .peepwl .sociagraph.figs.job.FigsPrankReportJob.run(FigsPrankReportJob.java: 58) at com.peepwl.sociagraph.common.jobs.JobExecutorImpl $2.run(JobExecutorImpl.java:71) at java.util.concurrent.Executors $RunnableAdapter.call(Executors.java: 417) at java.util.concurrent.FutureTask $Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ScheduledThreadPoolExecutor $ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65) at java.util.concurrent.ScheduledThreadPoolExecutor $ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:172) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:613) Caused by: org.neo4j.impl.nioneo.store.StoreFailureException: Unable to load position[24058922] @[601473050] at org .neo4j .impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java: 101) at org .neo4j .impl .nioneo .store.PersistenceWindowPool.acquire(PersistenceWindowPool.java: 150) at org .neo4j .impl .nioneo .store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java: 406) at org .neo4j .impl.nioneo.store.PropertyStore.getLightRecord(PropertyStore.java: 251) at org .neo4j .impl .nioneo .xa.NeoReadTransaction.nodeGetProperties(NeoReadTransaction.java:143) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ ReadOnlyResourceConnection .nodeLoadProperties(NioNeoDbPersistenceSource.java:232) at org .neo4j .impl .persistence .PersistenceManager.loadNodeProperties(PersistenceManager.java:84) at org.neo4j.impl.core.NodeManager.loadProperties(NodeManager.java:616) ... 16 more Caused by: java.nio.channels.ClosedByInterruptException at java .nio .channels .spi .AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java: 184) at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315) at org .neo4j .impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java: 80) ... 23 more java.lang.RuntimeException: javax.transaction.RollbackException: Failed to commit, transaction rolledback at org.neo4j.api.core.EmbeddedNeo $TransactionImpl.finish(EmbeddedNeo.java:383) at com .peepwl .sociagraph
Re: [Neo] Interrupts
Cool, Thanks as ever Johan. ATB Neil On 30 Jul 2009, at 20:03, Johan Svensson wrote: You are right, we should change that generic one to a specific runtime exception. Will look into that during our exception refactoring. -Johan On Thu, Jul 30, 2009 at 8:51 PM, Neil Ellisneil.el...@peepwl.com wrote: Hi Johan I meant the application is being shutdown (not neo.shutdown()) :-) - though I understand the confusion. So as part of our shutdown process we will tell all active threads to stop by interrupting them. When they've all stopped I'll shutdown Neo. So the thread this code runs in is receiving an interrupt, however Neo is converting the exception into a generic RuntimeException. Which makes it very hard to catch, with the code being like this it makes it very hard for me to request a thread interruption and then deal with the consequences (i.e. a RuntimeException does not tell me that the thread has been interrupted, so I don't know whether to report an error and continue - or to end the loop it is in). Thanks as always Neil On 30 Jul 2009, at 19:45, Johan Svensson wrote: Hi Neil, Yes, there are a lot of error messages that needs to be improved. On the other hand the documentation should state that calling any Neo4j API operation after (or concurrently with) NeoService.shutdown() is not valid. Adding a flag to check if running is not an option because of performance issues. Could be that there are certain places we could throw a better exception with a better message to help the user narrow down what is wrong. We are planing to do a refactoring of the low level exceptions before 1.0 final and we will keep this in mind then. Regards, Johan On Thu, Jul 23, 2009 at 2:14 AM, Neil Ellisneil.el...@mangala.co.uk wrote: Hi guys, if a thread gets interrupted (i.e. during a shutdown), I end up with a RuntimeException, I think it would be better to bubble something meaningful. What do you think? java.lang.RuntimeException: org.neo4j.impl.nioneo.store.StoreFailureException: Unable to load position[24058922] @[601473050]... ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Side effect
Hi Mattias Cool, you guys do get stuff fixed fast. All the best Neil On 28 Jul 2009, at 12:10, Mattias Persson wrote: I just committed a fix for it (rev. 3017) ... it should do the trick 2009/7/28 Mattias Persson matt...@neotechnology.com: You're absolutely right... I see the problem in the iterator implementation of that method. I or Johan will fix that soon. 2009/7/27 Neil Ellis neil.el...@peepwl.com: Hi guys I noticed a side-effect in the iterator for all nodes. If you do not do a hasNext() before a next() you get a no such element exception. So even if I have a million nodes .next() on the first node throws an exception, unless I do a hasNext() check first. I believe a hasXXX() method should be without side effect, is that fair comment? All the best Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Read Only
Hi Johan That would be great, those are the things that will cause us problems when we get to production. All the best Neil On 26 Jul 2009, at 13:07, Johan Svensson wrote: Hi Neil, It is possible to make Neo4j act as a read-only database resulting in no recovery after a non clean shutdown. We are planing to have a look at this after the b9 release. Also speeding up the recovery process is something we have to do before going 1.0 final (should be possible to drop the rebuild of id generator which is taking all the time during recovery). Regards, -Johan On Sun, Jul 26, 2009 at 2:35 AM, Neil Ellisneil.el...@peepwl.com wrote: Hi Guys So thanks for the plug for peepwl :-) I saw the slides from OSCON :-) Okay so we have a strong use case where we want to access Neo but don't want to trigger a complete transaction recovery if Neo fails. We're using Neo as read-only at this point. Recovery currently takes about half an hour or an hour which is a lot of downtime. Is it possible to make it so that Neo doesn't do this check and acts as a read-only database? All the best Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Read Only
Hi Guys So thanks for the plug for peepwl :-) I saw the slides from OSCON :-) Okay so we have a strong use case where we want to access Neo but don't want to trigger a complete transaction recovery if Neo fails. We're using Neo as read-only at this point. Recovery currently takes about half an hour or an hour which is a lot of downtime. Is it possible to make it so that Neo doesn't do this check and acts as a read-only database? All the best Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Merging
Oh and one more question, is there a utility written already to do rapid merging of graphs. We're going to build several graphs on several machines and then merge them later into a single graph (using a surrogate key), wondered if anyone had already done anything in this space already? ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Sorry full trace.
SEVERE: Failed loading relationships for node[5906099] java.lang.RuntimeException: java.nio.BufferUnderflowException at org.neo4j.impl.core.NodeManager.loadRelationships(NodeManager.java: 608) at org.neo4j.impl.core.NodeImpl.ensureFullRelationships(NodeImpl.java: 299) at org.neo4j.impl.core.NodeImpl.getAllRelationshipsOfType(NodeImpl.java: 118) at org.neo4j.impl.core.NodeImpl.getRelationships(NodeImpl.java:154) at org.neo4j.impl.core.NodeProxy.getRelationships(NodeProxy.java:76) at org .neo4j .impl .traversal.AbstractTraverser.addEndNodesToList(AbstractTraverser.java: 252) at org .neo4j .impl .traversal.AbstractTraverser.traverseToNextNode(AbstractTraverser.java: 201) at org .neo4j.impl.traversal.AbstractTraverser.hasNext(AbstractTraverser.java: 134) at org .neo4j .impl.traversal.AbstractTraverser.getAllNodes(AbstractTraverser.java: 303) at com .peepwl .sociagraph .figs .impl .index .FigsIndexNeoService .getAggregatedIdentityForNodeInternal(FigsIndexNeoService.java:250) at com .peepwl .sociagraph .figs .impl.index.FigsIndexNeoService.luceneIndex(FigsIndexNeoService.java: 361) at com.peepwl.sociagraph.figs.impl.index.FigsIndexNeoService.access $200(FigsIndexNeoService.java:45) at com.peepwl.sociagraph.figs.impl.index.FigsIndexNeoService $5.run(FigsIndexNeoService.java:323) at com.peepwl.sociagraph.figs.impl.exec.FigsExecutorServiceImpl $1.run(FigsExecutorServiceImpl.java:118) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:474) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:117) at org.neo4j.impl.nioneo.store.Buffer.get(Buffer.java:151) at org .neo4j .impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 180) at org .neo4j .impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 92) at org .neo4j .impl .nioneo .xa.NeoReadTransaction.nodeGetRelationships(NeoReadTransaction.java:89) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ ReadOnlyResourceConnection .nodeLoadRelationships(NioNeoDbPersistenceSource.java:237) at org .neo4j .impl .persistence .PersistenceManager.loadRelationships(PersistenceManager.java:79) at org.neo4j.impl.core.NodeManager.loadRelationships(NodeManager.java: 572) ... 16 more ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Sorry full trace.
Hey Johan Unit test is not so likely , it's an occasional error that appears during a long running process - I have seen it on our 64bit CentOS Linux boxes but not on my OS X development machine (don't read too much into that, the linux boxes run for hours doing their processing). I'll try and see if I can come up with a test, in the meantime I might be able to send some code snippets to yourself directly. Do you guys have a public JIRA or other bug tracker? All the best Neil On 20 Jul 2009, at 15:34, Johan Svensson wrote: Can you reproduce this? If you could create a test that triggers this would be great cause it looks like a bug. Also are you running this on windows? -Johan On Mon, Jul 20, 2009 at 1:45 PM, Neil Ellisneil.el...@mangala.co.uk wrote: SEVERE: Failed loading relationships for node[5906099] ... Caused by: java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:474) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:117) at org.neo4j.impl.nioneo.store.Buffer.get(Buffer.java:151) at org .neo4j .impl .nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 180) at org .neo4j .impl .nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 92) at org .neo4j .impl .nioneo .xa.NeoReadTransaction.nodeGetRelationships(NeoReadTransaction.java: 89) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ ReadOnlyResourceConnection .nodeLoadRelationships(NioNeoDbPersistenceSource.java:237) at org .neo4j .impl .persistence .PersistenceManager.loadRelationships(PersistenceManager.java:79) at org.neo4j.impl.core.NodeManager.loadRelationships(NodeManager.java: 572) ... 16 more ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Sorry full trace.
Hmmm I was using a traverser while iterating through all nodes I wonder any connection, code on it's way to you. On 20 Jul 2009, at 15:34, Johan Svensson wrote: Can you reproduce this? If you could create a test that triggers this would be great cause it looks like a bug. Also are you running this on windows? -Johan On Mon, Jul 20, 2009 at 1:45 PM, Neil Ellisneil.el...@mangala.co.uk wrote: SEVERE: Failed loading relationships for node[5906099] ... Caused by: java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:474) at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:117) at org.neo4j.impl.nioneo.store.Buffer.get(Buffer.java:151) at org .neo4j .impl .nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 180) at org .neo4j .impl .nioneo.store.RelationshipStore.getRecord(RelationshipStore.java: 92) at org .neo4j .impl .nioneo .xa.NeoReadTransaction.nodeGetRelationships(NeoReadTransaction.java: 89) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ ReadOnlyResourceConnection .nodeLoadRelationships(NioNeoDbPersistenceSource.java:237) at org .neo4j .impl .persistence .PersistenceManager.loadRelationships(PersistenceManager.java:79) at org.neo4j.impl.core.NodeManager.loadRelationships(NodeManager.java: 572) ... 16 more ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Write performance? Contention? How's this design?
We've been doing an easy Gig per minute using the batch updater. All the best Neil Ellis (Sent from my iPhone) On 16 Jun 2009, at 15:50, Dennis Peterson dennisbpeter...@gmail.com wrote: Hi everybody, I just came across Neo4J, and while the read performance sounds great, I haven't seen anything about write performance. I suspect it's good, but wanted to ask. I'm considering using Neo for a threaded discussion forum. It definitely sounds like it would retrieve a discussion tree very quickly. Would it also be fast in posting new messages? Root comments would be attached to a topic, so by starting at the topic's node you can get a list of the latest posts. It seems the way to do this would essentially be to make a linked list of posts, and with each new one, splice the new node in between the topic node and the previous head of the list. Would there likely be undue contention at that point? I'll also be rating comments, and would like to get a list of highest-rated comments. For that, I would likely keep a list of them. Keep the current low score somewhere, refresh it periodically with a scan of the list, anything above the low score just add it to the list, and periodically purge the excess. In a relational database of course it's easy, just sort by the scores...but that score index is going to be constantly re-sorting itself, so there's plenty going on, you just don't see it. I suspect Neo would actually be more performant since you don't have to keep *all* the comments sorted by score. Am I on the right track? ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Read only
Hi Guys Is there a simple way to create a readonly NeoService instance? It would be useful for us to not have any file locks or attempts at transaction recovery etc. It might also speed up reads if you know that there are no writes? I imagine it might have been done before just couldn't see any links. All the best Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] NPE
I'm fairly darn certain I've messed something up, but I'm sure you'd
prefer not to throw an NPE at this point.
java.lang.NullPointerException
at org.neo4j.impl.util.ArrayMap.remove(ArrayMap.java:221)
at
org
.neo4j
.util
.index
.LuceneIndexBatchInserterImpl
.getNodes(LuceneIndexBatchInserterImpl.java:143)
at
org
.neo4j
.util
.index
.LuceneIndexBatchInserterImpl
.getSingleNode(LuceneIndexBatchInserterImpl.java:222)
From ArrayMap:
..
if ( arrayCount == -1 )
{
V value = propertyMap.remove( key );
if ( switchBackToArray propertyMap.size()
toMapThreshold )
{
.
All the best
Neil
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
[Neo] NPE :-)
That NPE is probably due to multi-threaded access btw. so maybe not a priority to fix ;-) ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] NPE :-)
Hi Tobias On 16 Jun 2009, at 20:13, Tobias Ivarsson wrote: Are you doing multi threaded access to the batch inserter API?! In that case you are on uncertain waters. Yeah I know, it was my bad :-) Would be nice to have a safety net on this class for when we do silly things, either synchronized or multiple access checks anyway thanks Tobias, loving Neo4J so far. Neil. The batch insertion mode is for bulk loading of initial data, not something you use in the actual running application. There really isn't much to gain from feeding the batch inserter from multiple threads, it's better to have multiple threads doing computation, if you need to do that in your batch processing, and then have one writer thread reading from a queue and feeding the batch inserter. I can see why the problem occurs. Since the batch insertion API is intended for single threaded use it uses the unsynchronized implementation of ArrayMap for speed. When your application is accessing it through multiple threads there is a chance that you access it in an inconsistent state, and therefore end up with the exception you reported. Still, it is at least not a very descriptive exception, and we might consider doing something to improve this. So thank you for reporting the problem. Cheers, Tobias On Tue, Jun 16, 2009 at 8:59 PM, Neil Ellis neil.el...@mangala.co.ukwrote: That NPE is probably due to multi-threaded access btw. so maybe not a priority to fix ;-) ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Read only
Hi Johan If there are the use cases to support such a change that would be fantastic. All the best Neil On 16 Jun 2009, at 22:13, Johan Svensson wrote: Hi Neil, There is currently no simple way to do this, it would require some changes to the code. However, I am starting to see more and more use-cases for a read-only solution so after b9 has been released I will have a look at this. Regards, Johan On Tue, Jun 16, 2009 at 6:40 PM, Neil Ellisneil.el...@mangala.co.uk wrote: Hi Guys Is there a simple way to create a readonly NeoService instance? It would be useful for us to not have any file locks or attempts at transaction recovery etc. It might also speed up reads if you know that there are no writes? I imagine it might have been done before just couldn't see any links. All the best Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Performance
Some feedback: Using the batch update code in trunk, some anecdotal observations I've had an even 1Gb per minute stored in Neo, now that's a mixture of data and includes some text blocks but there is also 2.8 million nodes in 4 minutes. Disk writes during this time have been even also at about 75 Mb/s (bytes/second) across RAID 5 storage. This includes adding an entry in a Lucene index for each node. Of course my application may well be adding delays itself and this figure is not including adding relationships (at the mo). But considering I am very new to Neo I'm very happy with what has been achieved so far. Are there any actual benchmarks comparing Neo4J with competition? Neil ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Some exceptions
So after a weekend away from the problem it occured to me that I had neglected to nohup the java process. Therefore I am suspect that was the problem, running now with nohup ;-) It also occured to me that on receipt of ClosedByInterruptException Neo4J should probably trigger a shutdown (i.e. neo.shutdown()) and log that it was interrupted since an interrupt has been triggered on the write thread. The later stack traces show that Neo doesn't recover from such an interrupt, so one would suspect that this is currently the best course of action and is logically what should happen when an interrupt is triggered. Let me know what you think please Johan, meanwhile I'm running with nohup and will let you know if that was indeed the cause. All the best Neil On 5 Jun 2009, at 19:05, Johan Svensson wrote: These problems are hard to find. I do not like the way java.nio behaves on interrupts since I don't know how much data got written/read and the underlying file channel just gets closed. At the moment the only thing we can do is throw an exception and do a full recovery process... Also I think Thread.interrupt idiom/usage is broken. Only time you can use it is when you have total control and own the full stack knowing exactly where the specific thread is executing. I have seen some web servers/containers use it to try control threads that should timeout and that doesn't work very well. How many CPUs do you have on the machine running this (could be a spurious wakeup somewhere)? Also do you run concurrent transactions or not? I have been trying to reproduce the other problem with nested transactions but nothing so far. -Johan On Fri, Jun 5, 2009 at 7:42 PM, Neil Ellis neil.el...@mangala.co.uk wrote: Nope, don't think it's a user exception, digging deeper. On 5 Jun 2009, at 14:55, Neil Ellis wrote: Hi Johan Took a little longer to fail (circa 31Gb) with java.nio.channels.ClosedByInterruptException Received http://www.myspace.com/nuski at java .nio .channels .spi .AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java: 184) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) at org.neo4j.impl.transaction.TxLog.txDone(TxLog.java:221) at org.neo4j.impl.transaction.TxManager.rollback(TxManager.java:732) at org .neo4j .impl.transaction.TransactionImpl.rollback(TransactionImpl.java:108) at org.neo4j.api.core.EmbeddedNeo $TransactionImpl.finish(EmbeddedNeo.java:377) So I'm going to have to check through my code to see how the thread got interrupted. That smells like a user error, however it would be good of course if we can track the mistake down and allow Neo4j to realise it is a user error and fail more gracefully. Anyway, my turn to investigate a little more :) ATB Neil On 4 Jun 2009, at 19:19, Johan Svensson wrote: Thanks, I'll have a look at this and run some tests with nested transactions. -Johan On Thu, Jun 4, 2009 at 7:25 PM, Neil Ellis neil.el...@mangala.co.uk wrote: Hi these are from centos and 1.0-b8 I have changed to avoid nested transactions and now I'm not getting this, so (at the moment) looks like a nested transaction quirk ... and would only occur after a large number of successful writes (gigs). Thx Neil On 4 Jun 2009, at 18:18, Johan Svensson wrote: Hi Niel, What version of Neo4j are you running (b8,b9-SNAPSHOT) and on what OS/file system? -Johan On Thu, Jun 4, 2009 at 4:45 PM, Neil Ellis neil.el...@mangala.co.uk wrote: I took a little more of a look in the logs and found: java.lang.NullPointerException at org.neo4j.impl.nioneo.store.Buffer.getInt(Buffer.java: 161) at org .neo4j .impl .nioneo .store .AbstractDynamicStore.getLightRecord(AbstractDynamicStore.java: 409) at org .neo4j .impl .nioneo .store .AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java: 364) at org .neo4j .impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java: 306) at org .neo4j .impl .nioneo .xa.NeoTransaction.nodeChangeProperty(NeoTransaction.java: 1107) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection $NodeEventConsumerImpl.changeProperty(NeoStoreXaConnection.java: 217) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ NioNeoDbResourceConnection .nodeChangeProperty(NioNeoDbPersistenceSource.java:140) at org .neo4j .impl .persistence .PersistenceManager.nodeChangeProperty(PersistenceManager.java: 114) at org.neo4j.impl.core.NodeManag ive.java:407) at org.neo4j.impl.core.NodeProxy.setProperty(NodeProxy.java: 103) at com .peepwl .sociagraph .figs.impl.triple.FigsIndexNeo.index(FigsIndexNeo.java:52) at com.peepwl.sociagraph.figs.impl.triple.FigsTripleIndex $ FigsReindexReadCallback .acceptFinalIdentities(FigsTripleIndex.java: 159) at com .peepwl .sociagraph .figs
Re: [Neo] Some exceptions
So I've now checked all my code for correct dealing of InterruptedException also :) It occurred to me it would be wise for you to log the interrupted exception before clearing the status. Otherwise later on you'll be wondering why a thread just quitted in the middle of a method, no? I've just made sure that I clear and log for each thread, double check for isInterrupted() during long loops and everywhere else just allow the exception to bubble upwards so hopefully I'm behaving okay now, can't speak for dependent libraries alas. I am currently running patched b8 will let you know how it goes. Thanks again Neil On 8 Jun 2009, at 14:24, Johan Svensson wrote: On Mon, Jun 8, 2009 at 1:42 PM, Neil Ellis neil.el...@mangala.co.uk wrote: On 8 Jun 2009, at 12:21, Johan Svensson wrote: On Mon, Jun 8, 2009 at 12:04 PM, Neil Ellis neil.el...@mangala.co.uk wrote: So after a weekend away from the problem it occured to me that I had neglected to nohup the java process. Therefore I am suspect that was the problem, running now with nohup ;-) Lets hope this works. I am not 100% sure about this but normally a kill signal sent to a Java process shouldn't affect any of the Java thread's interrupt status. If thats the case (when for example using java.nio stuff) behavior is new to me. A way to achieve this would be to register a signal handler (via sun.misc.Signal) and manually go through each thread and interrupt them (so check if there is any code like that running). What JVM are you running? Okay so I was clutching a little ;-) I bit of reading and I now understand that was a naive guess. Although the fact that Java actually can receive a HUP even if nohup'ed depresses me a little :). I tested this on Sun's 1.6 JVM and could not get any threads to interrupt sending various signals to the Java process. What JVM are you running? Another thing, I just now went through all the catching of InterruptedException we do in Neo4j and actually found one place that calls java.nio code that didn't reset the interrupt properly after a wait. I fixed this in trunk (b9-SNAPSHOT) and if you want you could try to apply the same patch to b8 (see https://trac.neo4j.org/changeset/2890) or I could provide a jar for you. If you want to try out b9-SNAPSHOT just remember you can't switch back to b8 since the store layout as changed. That's fine, we recreate the database from scratch each time, I'll checkout and build today and let you know - how stable is b9-SNAPSHOT btw. am I likely to shoot myself in the foot using it? That depends :) I would say that trunk is not stable now because we haven't done any extensive testing on it yet. The plan is to release b9-SNAPSHOT at the end of this month. Right now we are working on some (minor) known issues and next week we will get some API changes in place. After that there are a few improvements I hope to get in (or they will be pushed to next release). This is followed by two weeks of testing then hopefully a release. Regards, Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Some exceptions
I took a little more of a look in the logs and found: java.lang.NullPointerException at org.neo4j.impl.nioneo.store.Buffer.getInt(Buffer.java:161) at org .neo4j .impl .nioneo .store.AbstractDynamicStore.getLightRecord(AbstractDynamicStore.java: 409) at org .neo4j .impl .nioneo .store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java: 364) at org.neo4j.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java: 306) at org .neo4j .impl.nioneo.xa.NeoTransaction.nodeChangeProperty(NeoTransaction.java: 1107) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection $NodeEventConsumerImpl.changeProperty(NeoStoreXaConnection.java:217) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ NioNeoDbResourceConnection .nodeChangeProperty(NioNeoDbPersistenceSource.java:140) at org .neo4j .impl .persistence .PersistenceManager.nodeChangeProperty(PersistenceManager.java:114) at org.neo4j.impl.core.NodeManag ive.java:407) at org.neo4j.impl.core.NodeProxy.setProperty(NodeProxy.java: 103) at com .peepwl .sociagraph.figs.impl.triple.FigsIndexNeo.index(FigsIndexNeo.java:52) at com.peepwl.sociagraph.figs.impl.triple.FigsTripleIndex $FigsReindexReadCallback.acceptFinalIdentities(FigsTripleIndex.java:159) at com .peepwl .sociagraph .figs.impl.serialization.FigsJackson.readInternal(FigsJackson.java:321) at com .peepwl .sociagraph.figs.impl.serialization.FigsJackson.read(FigsJackson.java: 254) at com.peepwl.sociagraph.figs.impl.FigsStore.read(FigsStore.java:100) at com.peepwl.sociagraph.figs.impl.FigsBuilderQueryImpl $1.run(FigsBuilderQueryImpl.java:77) at com.peepwl.sociagraph.figs.impl.exec.FigsExecutorImpl $1.run(FigsExecutorImpl.java:55) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Then later on: org.neo4j.impl.nioneo.store.StoreFailureException: Unable to load position[10631186] @[1413947738] at org .neo4j .impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:86) at org .neo4j .impl .nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java: 147) at org .neo4j .impl .nioneo .store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:385) at org .neo4j .impl .nioneo .store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java: 360) at org.neo4j.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java: 306) at org .neo4j .impl.nioneo.xa.NeoTransaction.nodeChangeProperty(NeoTransaction.java: 1107) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection $NodeEventConsumerImpl.changeProperty(NeoStoreXaConnection.java:217) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ NioNeoDbResourceConnection .nodeChangeProperty(NioNeoDbPersistenceSource.java:140) at org .neo4j .impl .persistence .PersistenceManager.nodeChangeProperty(PersistenceManager.java:114) at org.neo4j.impl.core.NodeManager.nodeChangeProperty(NodeManager.java:821) at org.neo4j.impl.core.NodeImpl.changeProperty(NodeImpl.java: 61) at org.neo4j.impl.core.NeoPrimitive.setProperty(NeoPrimitive.java:407) at org.neo4j.impl.core.NodeProxy.setProperty(NodeProxy.java: 103) at com .peepwl .sociagraph.figs.impl.triple.FigsIndexNeo.index(FigsIndexNeo.java:48) at com.peepwl.sociagraph.figs.impl.triple.FigsTripleIndex $FigsReindexReadCallback.acceptFinalIdentities(FigsTripleIndex.java:159) at com .peepwl .sociagraph .figs.impl.serialization.FigsJackson.readInternal(FigsJackson.java:321) at com .peepwl .sociagraph.figs.impl.serialization.FigsJackson.read(FigsJackson.java: 254) at com.peepwl.sociagraph.figs.impl.FigsStore.read(FigsStore.java:100) at com.peepwl.sociagraph.figs.impl.FigsBuilderQueryImpl $1.run(FigsBuilderQueryImpl.java:77) at com.peepwl.sociagraph.figs.impl.exec.FigsExecutorImpl $1.run(FigsExecutorImpl.java:55) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java: 91) at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:302) at org .neo4j .impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:65) ... 22 more followed by more ClosedChannelExceptions I'm digging around to see if I'm doing anything stupid right now.
Re: [Neo] Some exceptions
Fantastic thanks for having a look. We'd like to build a Neo graph across multiple machines due to the volume of data being processed (and to keep our costs down) how easy (or not) is this to do with Neo4j? Btw we're up to 15Gb Neo database now so I'm pretty sure the previous errors were only occurring because of nested transactions, I didn't make any other code changes. (Bare in mind I could have done something fundamentally wrong that trigger the failures, but still it does need to be graceful hey ;-)) All the best Neil On 4 Jun 2009, at 19:19, Johan Svensson wrote: Thanks, I'll have a look at this and run some tests with nested transactions. -Johan On Thu, Jun 4, 2009 at 7:25 PM, Neil Ellis neil.el...@mangala.co.uk wrote: Hi these are from centos and 1.0-b8 I have changed to avoid nested transactions and now I'm not getting this, so (at the moment) looks like a nested transaction quirk ... and would only occur after a large number of successful writes (gigs). Thx Neil On 4 Jun 2009, at 18:18, Johan Svensson wrote: Hi Niel, What version of Neo4j are you running (b8,b9-SNAPSHOT) and on what OS/file system? -Johan On Thu, Jun 4, 2009 at 4:45 PM, Neil Ellis neil.el...@mangala.co.uk wrote: I took a little more of a look in the logs and found: java.lang.NullPointerException at org.neo4j.impl.nioneo.store.Buffer.getInt(Buffer.java: 161) at org .neo4j .impl .nioneo .store .AbstractDynamicStore.getLightRecord(AbstractDynamicStore.java: 409) at org .neo4j .impl .nioneo .store .AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java: 364) at org .neo4j .impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java: 306) at org .neo4j .impl .nioneo.xa.NeoTransaction.nodeChangeProperty(NeoTransaction.java: 1107) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection $NodeEventConsumerImpl.changeProperty(NeoStoreXaConnection.java: 217) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ NioNeoDbResourceConnection .nodeChangeProperty(NioNeoDbPersistenceSource.java:140) at org .neo4j .impl .persistence .PersistenceManager.nodeChangeProperty(PersistenceManager.java:114) at org.neo4j.impl.core.NodeManag ive.java:407) at org.neo4j.impl.core.NodeProxy.setProperty(NodeProxy.java: 103) at com .peepwl .sociagraph.figs.impl.triple.FigsIndexNeo.index(FigsIndexNeo.java: 52) at com.peepwl.sociagraph.figs.impl.triple.FigsTripleIndex $ FigsReindexReadCallback.acceptFinalIdentities(FigsTripleIndex.java: 159) at com .peepwl .sociagraph .figs.impl.serialization.FigsJackson.readInternal(FigsJackson.java: 321) at com .peepwl .sociagraph .figs.impl.serialization.FigsJackson.read(FigsJackson.java: 254) at com.peepwl.sociagraph.figs.impl.FigsStore.read(FigsStore.java:100) at com.peepwl.sociagraph.figs.impl.FigsBuilderQueryImpl $1.run(FigsBuilderQueryImpl.java:77) at com.peepwl.sociagraph.figs.impl.exec.FigsExecutorImpl $1.run(FigsExecutorImpl.java:55) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Then later on: org.neo4j.impl.nioneo.store.StoreFailureException: Unable to load position[10631186] @[1413947738] at org .neo4j .impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java: 86) at org .neo4j .impl .nioneo .store.PersistenceWindowPool.acquire(PersistenceWindowPool.java: 147) at org .neo4j .impl .nioneo .store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java: 385) at org .neo4j .impl .nioneo .store .AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java: 360) at org .neo4j .impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java: 306) at org .neo4j .impl .nioneo.xa.NeoTransaction.nodeChangeProperty(NeoTransaction.java: 1107) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection $NodeEventConsumerImpl.changeProperty(NeoStoreXaConnection.java: 217) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource $ NioNeoDbResourceConnection .nodeChangeProperty(NioNeoDbPersistenceSource.java:140) at org .neo4j .impl .persistence .PersistenceManager.nodeChangeProperty(PersistenceManager.java:114) at org .neo4j.impl.core.NodeManager.nodeChangeProperty(NodeManager.java: 821) at org.neo4j.impl.core.NodeImpl.changeProperty(NodeImpl.java: 61) at org.neo4j.impl.core.NeoPrimitive.setProperty(NeoPrimitive.java:407) at org.neo4j.impl.core.NodeProxy.setProperty(NodeProxy.java: 103) at com .peepwl .sociagraph.figs.impl.triple.FigsIndexNeo.index(FigsIndexNeo.java: 48
